Computation

Visualization
Programming
For Use with MATLAB

Users Guide
Version 2
Thomas Coleman
Mary Ann Branch
AndrewGrace
Optimization
Toolbox
How to Contact The MathWorks:
508-647-7000 Phone
508-647-7001 Fax
The MathWorks, I nc. Mai l
24 Pri me Park Way
Nati ck, MA 01760-1500
http://www.mathworks.com Web
ftp.mathworks.com Anonymous FTP server
comp.soft-sys.matlab Newsgroup
[email protected] Techni cal support
[email protected] Product enhancement suggesti ons
[email protected] Bug reports
[email protected] Documentati on error reports
[email protected] Subscri bi ng user regi strati on
[email protected] Order status, l i cense renewal s, passcodes
[email protected] Sal es, pri ci ng, and general i nformati on
Optimization Toolbox Users Guide
COPYRI GHT 1990 - 1999 by The MathWor ks, I nc.
The softwar e descr i bed i n thi s document i s fur ni shed under a l i cense agreement. The software may be used
or copi ed onl y under the terms of the l i cense agr eement. No par t of thi s manual may be photocopi ed or r epr o-
duced i n any form wi thout pr i or wr i tten consent from The MathWor ks, I nc.
U.S. GOVERNMENT: I f Li censee i s acqui r i ng the Progr ams on behal f of any uni t or agency of the U.S.
Gover nment, the fol l owi ng shal l appl y: (a) For uni ts of the Depar tment of Defense: the Gover nment shal l
have onl y the ri ghts speci fi ed i n the l i cense under whi ch the commerci al computer softwar e or commer ci al
softwar e documentati on was obtai ned, as set for th i n subpar agr aph (a) of the Ri ghts i n Commer ci al
Computer Softwar e or Commer ci al Softwar e Documentati on Cl ause at DFARS 227.7202-3, ther efor e the
ri ghts set for th her ei n shal l appl y; and (b) For any other uni t or agency: NOTI CE: Notwi thstandi ng any
other l ease or l i cense agr eement that may per tai n to, or accompany the del i ver y of, the computer software
and accompanyi ng documentati on, the ri ghts of the Gover nment regar di ng i ts use, r epr oducti on, and di scl o-
sure are as set for th i n Cl ause 52.227-19 (c)(2) of the FAR.
MATLAB, Si mul i nk, Statefl ow, Handl e Gr aphi cs, and Real -Ti me Workshop ar e r egi stered trademar ks, and
Tar get Language Compi l er i s a trademar k of The MathWor ks, I nc.
Other pr oduct or br and names are tr ademar ks or r egi stered tr ademar ks of thei r respecti ve hol der s.
Pri nti ng Hi story: November 1990 Fi r st pr i nti ng
December 1996 Second pri nti ng for MATLAB 5
Januar y 1999 Thi rd pri nti ng Revi sed for Ver si on 2 (Rel ease 11)
i
Contents
BeforeYou Begin
What Is theOptimization Toolbox? . . . . . . . . . . . . . . . . . . . . . viii
NewFeatures in Version 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Large-Scal e Al gori thms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Functi on Names and Cal l i ng Syntax . . . . . . . . . . . . . . . . . . . . . . ix
Howto Usethis Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
InstallingtheToolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Typographical Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Matri x, Vector, and Scal ar Notati on . . . . . . . . . . . . . . . . . . . . . xiii
1
Tutorial
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Overvi ew . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-4
Medium-ScaleExamples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-6
Unconstrai ned Exampl e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-6
Nonl i near I nequal i ty Constrai ned Exampl e . . . . . . . . . . . . . . . 1-8
Constrai ned Exampl e wi th Bounds . . . . . . . . . . . . . . . . . . . . . . 1-9
Constrai ned Exampl e wi th Gradi ents . . . . . . . . . . . . . . . . . . . 1-11
Gradi ent Check: Anal yti c Versus Numeri c . . . . . . . . . . . . . . . 1-13
Equal i ty Constrai ned Exampl e . . . . . . . . . . . . . . . . . . . . . . . . 1-14
Maxi mi zati on . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-15
Greater than Zero Constrai nts . . . . . . . . . . . . . . . . . . . . . . . . . 1-15
Addi ti onal Arguments: Avoi di ng Gl obal Vari abl es . . . . . . . . . 1-15
Mul ti objecti ve Exampl es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-16
ii Contents
Large-ScaleExamples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-29
Nonl i near Equati ons wi th Jacobi an . . . . . . . . . . . . . . . . . . . . . 1-33
Nonl i near Equati ons wi th Jacobi an Sparsi ty Pattern . . . . . . . 1-35
Nonl i near Least-Squares wi th
Ful l Jacobi an Sparsi ty Pattern . . . . . . . . . . . . . . . . . . . . . . . 1-37
Nonl i near Mi ni mi zati on wi th Gradi ent and Hessi an . . . . . . . 1-39
Nonl i near Mi ni mi zati on wi th
Gradi ent and Hessi an Sparsi ty
Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-40
Nonl i near Mi ni mi zati on wi th Bound Constrai nts and
Banded
Precondi ti oner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-43
Nonl i near Mi ni mi zati on wi th Equal i ty Constrai nts . . . . . . . . 1-47
Quadrati c Mi ni mi zati on wi th Bound Constrai nts . . . . . . . . . . 1-49
Li near Least-Squares wi th Bound Constrai nts . . . . . . . . . . . . 1-50
Li near Programmi ng wi th Equal i ti es and I nequal i ti es . . . . . . 1-52
Li near Programmi ng wi th Dense Col umns i n the Equal i ti es . 1-53
Default Parameter Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-56
Changi ng the Defaul t Setti ngs . . . . . . . . . . . . . . . . . . . . . . . . . 1-56
DisplayingIterativeOutput . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-58
Output Headi ngs: Medi um-Scal e Al gori thms . . . . . . . . . . . . . 1-58
Output Headi ngs: Large-Scal e Al gori thms . . . . . . . . . . . . . . . 1-60
Optimization of InlineObjects Instead of M-Files . . . . . . . 1-63
Practicalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-65
ConvertingYour Codeto Version 2.0Syntax . . . . . . . . . . . . 1-69
Usi ng opti mset and opti mget . . . . . . . . . . . . . . . . . . . . . . . . . . 1-70
New Cal l i ng Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-70
Exampl e of Converti ng from constr to fmi ncon . . . . . . . . . . . . 1-80
iii
2
Introduction to Algorithms
Parametric Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-2
Unconstrained Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-4
Quasi -Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5
Li ne Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
Quasi-Newton Implementation . . . . . . . . . . . . . . . . . . . . . . . . 2-11
Hessi an Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-11
Li ne Search Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-11
Least Squares Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17
Gauss-Newton Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-18
Levenberg-Marquardt Method . . . . . . . . . . . . . . . . . . . . . . . . . 2-19
Nonlinear Least Squares Implementation . . . . . . . . . . . . . . 2-21
Gauss-Newton I mpl ementati on . . . . . . . . . . . . . . . . . . . . . . . . 2-21
Levenberg-Marquardt I mpl ementati on . . . . . . . . . . . . . . . . . . 2-21
Constrained Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-23
Sequenti al Quadrati c Programmi ng (SQP) . . . . . . . . . . . . . . . 2-24
QP Subprobl em . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-24
SQP Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-27
Updati ng the Hessi an Matri x . . . . . . . . . . . . . . . . . . . . . . . . . . 2-27
Quadrati c Programmi ng Sol uti on . . . . . . . . . . . . . . . . . . . . . . . 2-28
Li ne Search and Meri t Functi on . . . . . . . . . . . . . . . . . . . . . . . . 2-31
MultiobjectiveOptimization . . . . . . . . . . . . . . . . . . . . . . . . . . 2-33
I ntroducti on . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-33
Goal Attai nment Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-39
Al gori thm I mprovements for Goal Attai nment Method . . . . . 2-40
iv Contents
Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-43
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-44
3
Large-ScaleAlgorithms
Large-ScaleOptimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2
Trust Region Methods for Nonlinear Minimization . . . . . . . 3-3
Preconditioned ConjugateGradients . . . . . . . . . . . . . . . . . . . 3-6
Linearly Constrained Problems . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Li near Equal i ty Constrai nts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Box Constrai nts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Nonlinear Least-Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-11
Quadratic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-12
Linear Least-Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-13
Large-ScaleLinear Programming . . . . . . . . . . . . . . . . . . . . . 3-14
Mai n Al gori thm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-14
Preprocessi ng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-18
v
4
Reference
Optimization Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Mi ni mi zati on . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Equati on Sol vi ng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Least-Squares (Curve Fi tti ng) . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4
Uti l i ty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4
Demonstrati ons of Large-Scal e Methods . . . . . . . . . . . . . . . . . . 4-4
Demonstrati ons of Medi um-Scal e Methods . . . . . . . . . . . . . . . . 4-5
Function Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-6
vi Contents
Before You Begi n
Before You Begin
viii
What Is the Optimization Toolbox?
The Opti mi zati on Tool box i s a col l ecti on of functi ons that extend the capabi l i ty
of the MATLAB

numeri c computi ng envi ronment. The tool box i ncl udes

routi nes for many types of opti mi zati on i ncl udi ng:
Unconstrai ned nonl i near mi ni mi zati on
Constrai ned nonl i near mi ni mi zati on, i ncl udi ng goal attai nment probl ems,
mi ni max probl ems, and semi -i nfi ni te mi ni mi zati on probl ems
Quadrati c and l i near programmi ng
Nonl i near l east squares and curve-fi tti ng
Nonl i near system of equati on sol vi ng
Constrai ned l i near l east squares
Speci al i zed al gori thms for l arge-scal e (sparse) probl ems are al so avai l abl e (see
Large-Scal e Al gori thms i n the next secti on New Features i n Versi on 2).
Al l of the tool box functi ons are MATLAB M-fi l es, made up of MATLAB
statements that i mpl ement speci al i zed opti mi zati on al gor i thms. You can vi ew
the MATLAB code for these functi ons usi ng the statement:
type function_name
You can extend the capabi l i ti es of the Opti mi zati on Tool box by wri ti ng your
own M-fi l es, or by usi ng the tool box i n combi nati on wi th other tool boxes, or
wi th MATLAB, or Si mul i nk

.
New Features in Version 2
ix
New Features in Version 2
Versi on 2 of the Opti mi zati on Tool box offers a number of advances over
previ ous versi ons.
Large-Scale Algorithms
The focus of thi s versi on i s new al gori thms for sol vi ng l arge-scal e probl ems,
i ncl udi ng
Li near programmi ng
Nonl i near l east squares wi th bound constrai nts
Nonl i near system of equati on sol vi ng
Unconstrai ned nonl i near mi ni mi zati on
Nonl i near mi ni mi zati on wi th bound constrai nts
Nonl i near mi ni mi zati on wi th l i near equal i ti es
Quadrati c probl ems wi th bound constrai nts
Quadrati c probl ems wi th l i near equal i ti es
Li near l east squares wi th bound constrai nts
The new l arge-scal e al gori thms have been i ncorporated i nto the tool box
functi ons. The new functi onal i ty i mproves the abi l i ty of the tool box to sol ve
l arge sparse probl ems.
Function Names and Calling Syntax
To accommodate thi s new functi onal i ty, many of the functi on names and
cal l i ng sequences have changed. Some of the i mprovements i ncl ude
Command l i ne syntax has changed:
- Equal i ty constrai nts and i nequal i ty constrai nts are now suppl i ed as
separate i nput arguments.
- Li near constrai nts are suppl i ed as separate arguments from the nonl i near
constrai nt functi on.
- The gradi ent of the objecti ve i s computed i n the same functi on as the
objecti ve, rather than i n a separate functi on, i n order to provi de more
effi ci ent computati on (because the gradi ent and objecti ve often share
Before You Begin
x
si mi l ar computati ons). Si mi l arl y, the gradi ent of the nonl i near constrai nts
i s computed by the (now separate) nonl i near constrai nt functi on.
- The Hessi an matri x can be provi ded by the objecti ve functi on when usi ng
the l arge-scal e al gori thms.
Opti mi zati on parameters are now contai ned i n a structure, wi th functi ons to
create, change, and retri eve val ues.
Each functi on returns an exi t fl ag that denotes the termi nati on state.
For more i nformati on on how to convert your ol d syntax to the new functi on
cal l i ng sequences, see Converti ng Your Code to Versi on 2.0 Syntax i n
Chapter 1.
How to Use this Manual
xi
How to Use this Manual
Thi s manual has three mai n parts:
Chapter 1 provi des a tutori al for sol vi ng di fferent opti mi zati on probl ems,
i ncl udi ng a speci al secti on that hi ghl i ghts l arge-scal e probl ems. Thi s chapter
al so provi des i nformati on on how to use the tool box functi ons i n conjuncti on
wi th Si mul i nk usi ng mul ti objecti ve opti mi zati on. Other secti ons i ncl ude
i nformati on about changi ng defaul t parameters and usi ng i nl i ne objects.
Chapters 2 and 3 descri be the al gori thms i n the opti mi zati on functi ons.
Chapter 2 descri bes the probl em formul ati ons and the al gori thms for the
medi um-scal e al gori thms. Chapter 3 focuses on the l arge-scal e al gori thms.
Chapter 4 provi des a detai l ed reference descri pti on of each tool box functi on.
Reference descri pti ons i ncl ude the functi ons syntax, a descri pti on of the
di fferent cal l i ng sequences avai l abl e, and detai l ed i nformati on about
arguments to the functi on, i ncl udi ng rel evant opti mi zati on opti ons
parameters. Reference descri pti ons may al so i ncl ude exampl es, a summary
of the functi ons al gori thms, and references to addi ti onal readi ng materi al .
Before You Begin
xii
Installing the Toolbox
To determi ne i f the Opti mi zati on Tool box i s i nstal l ed on your system, type thi s
command at the MATLAB prompt:
ver
When you enter thi s command, MATLAB di spl ays i nformati on about the
versi on of MATLAB you are runni ng, i ncl udi ng a l i st of al l tool boxes i nstal l ed
on your system and thei r versi on numbers.
I f the Opti mi zati on Tool box i s not i nstal l ed, check the I nstallation Guidefor
i nstructi ons on how to i nstal l i t.
Typographical Conventions
xiii
Typographical Conventions
Matrix, Vector, and Scalar Notation
Upper-case l etters such as are used to denote matri ces. Lower-case l etters
such as are used to denote vectors, except where noted that i t i s a scal ar. For
functi ons, the notati on di ffers sl i ghtl y to fol l ow the usual conventi ons i n
opti mi zati on. For vector functi ons, we use an upper-case l etter such as i n
. A functi on that returns a scal ar val ue i s denoted wi th a l ower-case l etter
such as i n .
To Indicate This Guide Uses Example
Exampl e code Monospace type To assi gn the val ue 5 to A,
enter:
A = 5
MATLAB
output
Monospace type MATLAB responds wi th
A =
5
Functi on
names
Monospace type The cos functi on fi nds the
cosi ne of each array
el ement.
Functi on
syntax
Monospace type for text
that must appear as
shown.
The fminbnd functi on
uses the syntax
x= fminbnd('sin',3,4)
Mathemati cal
expressi ons
Vari abl es i n italics.
Functi ons, operator s, and
constants i n standard
type.
Thi s vector represents the
pol ynomi al
p = x
2
+ 2x+ 3
New terms I talics An array i s an ordered
col l ecti on of i nformati on.
A
x
F
F x ( )
f f x ( )
Before You Begin
xiv
1
Tutori al
Introduction . . . . . . . . . . . . . . . . . . . . 1-2
Medium-ScaleExamples . . . . . . . . . . . . . . 1-6
Large-ScaleExamples . . . . . . . . . . . . . . . 1-29
Default Parameter Settings . . . . . . . . . . . . . 1-56
DisplayingIterativeOutput . . . . . . . . . . . . . 1-58
Optimization of InlineObjects Instead of M-Files . . . 1-63
Practicalities . . . . . . . . . . . . . . . . . . . 1-65
ConvertingYour Codeto Version 2.0Syntax . . . . . 1-69
1 Tutorial
1-2
Introduction
Opti mi zati on concerns the mi ni mi zati on or maxi mi zati on of functi ons. The
Opti mi zati on Tool box consi sts of functi ons that perform mi ni mi zati on (or
maxi mi zati on) on general nonl i near functi ons. Functi ons for nonl i near
equati on sol vi ng and l east-squares (data-fi tti ng) probl ems are al so provi ded.
The tabl es bel ow show the functi ons avai l abl e for mi ni mi zati on, equati on
sol vi ng, and sol vi ng l east squares or data fi tti ng probl ems.
Table 1-1: Minimization
Type Notation Function
Scal ar Mi ni mi zati on such that fminbnd
Unconstrai ned
Mi ni mi zati on
fminunc,
fminsearch
Li near Programmi ng linprog
Quadrati c Programmi ng quadprog
Constrai ned
Mi ni mi zati on
fmincon
Goal Attai nment fgoalattain
f a ( )
a
min a
1
a a
2
< <
f x ( )
x
min
f
T
x
x
min such that
A x b Aeq x beq l x u , = ,
1
2
---x
T
Hx f
T
x +
x
min such that
A x b Aeq x beq l x u , = ,
f x ( )
x
min such that c x ( ) 0 ceq x ( ) , 0 =
A x b Aeq x beq l x u , = ,

x ,
min such that F x ( ) w goal
c x ( ) 0 ceq x ( ) , 0, =
A x b Aeq x beq l x u , = ,
Introduction
1-3
Mi ni max fminimax
Semi -i nfi ni te
Mi ni mi zati on
fseminf
Table 1-1: Minimization (Continued)
Type Notation Function
F
i
x ( ) { } such that
F
i
{ }
max
x
min
c x ( ) 0 ceq x ( ) , 0, =
A x b Aeq x beq l x u , = ,
f x ( ) such that
x
min K x w , ( ) 0 for al l w,
c x ( ) 0 ceq x ( ) , 0, =
A x b Aeq x beq l x u , = ,
Table 1-2: Equation Solving
Type Notation Function
Li near Equati ons , n equati ons, n vari abl es \ (slash)
Nonl i near Equati on of One
Vari abl e
fzero
Nonl i near Equati ons , n equati ons, n vari abl es fsolve
C x d =
f a ( ) 0 =
F x ( ) 0 =
Table 1-3: Least-Squares (Curve Fitting)
Type Notation Function
Li near Least Squares , mequati ons, n vari abl es \ (slash)
Nonnegati ve Li near Least
Squares
such that lsqnonneg
C x d
2
x
min
2
C x d
2
x
min
2
x 0
1 Tutorial
1-4
Overview
Most of these opti mi zati on routi nes requi re the defi ni ti on of an M-fi l e
contai ni ng the functi on to be mi ni mi zed. Al ternati vel y, an i nl i ne object created
from a MATLAB expressi on can be used. Maxi mi zati on i s achi eved by
suppl yi ng the routi nes wi th f, where f i s the functi on bei ng opti mi zed.
Opti mi zati on opti ons passed to the routi nes change opti mi zati on parameters.
Defaul t opti mi zati on parameters are used extensi vel y but can be changed
through an options structure.
Gradi ents are cal cul ated usi ng an adapti ve fi ni te-di fference method unl ess
they are suppl i ed i n a functi on. Parameters can be passed di rectl y to functi ons,
avoi di ng the need for gl obal vari abl es.
Thi s Users Guideseparates medi um-scal e al gori thms from l arge-scal e
al gori thms. Medi um-scal e i s not a standard term and i s used here onl y to
di fferenti ate these al gori thms from the l arge-scal e al gori thms, whi ch are
desi gned to handl e l arge-scal e probl ems effi ci entl y.
Medium-Scale Algorithms
The Opti mi zati on Tool box routi nes offer a choi ce of al gori thms and l i ne search
strategi es. The pri nci pal al gori thms for unconstrai ned mi ni mi zati on are the
Nel der-Mead si mpl ex search method and the BFGS quasi -Newton method. For
constrai ned mi ni mi zati on, mi ni max, goal attai nment, and semi -i nfi ni te
opti mi zati on, vari ati ons of Sequenti al Quadrati c Programmi ng are used.
Constrai ned Li near Least
Squares
such that lsqlin
Nonl i near Least Squares
such that
lsqnonlin
Nonl i near Curve Fi tti ng
such that
lsqcurvefit
Table 1-3: Least-Squares (Curve Fitting) (Continued)
Type Notation Function
C x d
2
x
min
2
A x b Aeq x beq l x u , = ,
1
2
--- F x ( )
2
2
x
min
1
2
--- F
i
x ( )
2
i

=
l x u
1
2
--- F x xdata , ( ) ydata
2
2
x
min
l x u
Introduction
1-5
Nonl i near l east squares probl ems use the Gauss-Newton and
Levenberg-Marquardt methods.
A choi ce of l i ne search strategy i s gi ven for unconstrai ned mi ni mi zati on and
nonl i near l east squares probl ems. The l i ne search strategi es use safeguarded
cubi c and quadrati c i nterpol ati on and extrapol ati on methods.
Large-Scale Algorithms
Al l the l arge-scal e al gori thms, except l i near programmi ng, are trust-regi on
methods. Bound constrai ned probl ems are sol ved usi ng refl ecti ve Newton
methods. Equal i ty constrai ned probl ems are sol ved usi ng a projecti ve
precondi ti oned conjugate gradi ent i terati on. You can use sparse i terati ve
sol vers or sparse di rect sol vers i n sol vi ng the l i near systems to determi ne the
current step. Some choi ce of precondi ti oni ng i n the i terati ve sol vers i s al so
avai l abl e.
The l i near programmi ng method i s a vari ant of Mehrotras predi ctor-corrector
al gori thm, a pri mal -dual i nteri or-poi nt method.
1 Tutorial
1-6
Medium-Scale Examples
The Opti mi zati on Tool box presents medium-scaleal gori thms through a
tutori al . The fi rst part of thi s tutori al (through the Equal i ty Constrai ned
Exampl e) fol l ows the fi rst demonstrati on Tutorial Walk Through i n the M-fi l e
optdemo. The exampl es i n the manual di ffer i n that M-fi l e functi ons were
wri tten for the objecti ve functi ons, whereas i n the onl i ne demonstrati on, i nl i ne
objects were used for some functi ons.
The tutori al di scusses the functi ons fminunc and fmincon i n detai l . The other
opti mi zati on routi nes fgoalattain, fminimax, lsqnonlin, fsolve, and
fseminf are used i n a nearl y i denti cal manner, wi th di fferences onl y i n the
probl em formul ati on and the termi nati on cri teri a. The next secti on di scusses
mul ti objecti ve opti mi zati on and gi ves several exampl es usi ng lsqnonlin,
fminimax, and fgoalattain i ncl udi ng how Si mul i nk can be used i n conjuncti on
wi th the tool box.
Unconstrained Example
Consi der the probl em of fi ndi ng a set of val ues [x
1
, x
2
] that sol ves
(1-1)
To sol ve thi s two-di mensi onal probl em, wri te an M-fi l e that returns the
functi on val ue. Then, i nvoke the unconstrai ned mi ni mi zati on routi ne fminunc.
Step 1: Write an M-file objfun.m
function f = objfun(x)
f = exp(x(1))*(4*x(1)^2+2*x(2)^2+4*x(1)*x(2)+2*x(2)+1);
Step 2: Invoke one of the unconstrained optimization routines
x0 = [1,1]; % Starting guess
options = optimset('LargeScale','off');
[x,fval,exitflag,output] = fminunc('objfun',x0,options);
After 40 functi on eval uati ons, thi s produces the sol uti on:
x =
0.5000 1.0000
f x ( )
x
minimize e
x
1
4x
1
2
2x
2
2
4x
1
x
2
2x
2
1 + + + + ( ) =
Medium-Scale Examples
1-7
The functi on at the sol uti on x i s returned i n fval:
fval =
1.3030e10
The exitflag tel l s i f the al gori thm converged. An exitflag > 0 means a l ocal
mi ni mum was found:
exitflag =
1
The output structure gi ves more detai l s about the opti mi zati on. For fminunc,
i t i ncl udes the number of i terati ons i n iterations, the number of functi on
eval uati ons i n funcCount, the fi nal step-si ze i n stepsize, a measure of
fi rst-order opti mal i ty (whi ch i n thi s unconstrai ned case i s the i nfi ni ty norm of
the gradi ent at the sol uti on) i n firstorderopt, and the type of al gori thm used
i n algorithm:
output =
iterations: 7
funcCount: 40
stepsize: 1
firstorderopt: 9.2801e-004
algorithm: 'medium-scale: Quasi-Newton line search'
When there exi sts more than one l ocal mi ni mum, the i ni ti al guess for the
vector [x
1
, x
2
] affects both the number of functi on eval uati ons and the val ue of
the sol uti on poi nt. I n the exampl e above, x0 i s i ni ti al i zed to [1,1].
The vari abl e options can be passed to fminunc to change characteri sti cs of the
opti mi zati on al gori thm, as i n
x = fminunc('objfun',x0,options);
options i s a structure that contai ns val ues for termi nati on tol erances and
al gori thm choi ces. An options structure can be created usi ng the optimset
functi on
options = optimset('LargeScale','off');
I n thi s exampl e we have turned off the defaul t sel ecti on of the l arge-scal e
al gori thm and so the medi um-scal e al gori thm i s used. Other opti ons i ncl ude
control l i ng the amount of command l i ne di spl ay duri ng the opti mi zati on
i terati on, the tol erances for the termi nati on cri teri a, i f a user-suppl i ed gradi ent
1 Tutorial
1-8
or Jacobi an i s to be used, and the maxi mum number of i terati ons or functi on
eval uati ons. See the References chapter pages for optimset and the i ndi vi dual
opti mi zati on functi ons, and Tabl e 4-3 for more opti ons and i nformati on.
Nonlinear Inequality Constrained Example
I f i nequal i ty constrai nts are added to Eq. 1-1, the resul ti ng probl em may be
sol ved by the fmincon functi on. For exampl e, fi nd x that sol ves
(1-2)
Si nce nei ther of the constr ai nts i s l i near , you cannot pass the constr ai nts to
fmincon at the command l i ne; i nstead you can create a second M-fi l e confun.m
that returns the val ue at both constrai nts at the current x i n a vector c. The
constrai ned opti mi zer, fmincon, i s then i nvoked. Because fmincon expects the
constrai nts to be wri tten i n the form , you must rewri te your
constrai nts i n the form
(1-3)
Step 1: Write an M-file confun.m for the constraints
function [c, ceq] = confun(x)
% nonlinear inequality constraints
c = [1.5 + x(1)*x(2) x(1) x(2);
x(1)*x(2) 10];
% nonlinear equality constraints
ceq = [];
f x ( )
x
minimize e
x
1
4x
1
2
2x
2
2
4x
1
x
2
2x
2
1 + + + + ( ) =
subject to the constrai nts: x
1
x
2
x
1
x
2
1.5
x
1
x
2
10
c x ( ) 0
x
1
x
2
x
1
x
2
1.5 + 0
x
1
x
2
10 0
Medium-Scale Examples
1-9
Step 2: Invoke constrained optimization routine:
x0 = [1,1]; % Make a starting guess at the solution
options = optimset('LargeScale','off');
[x, fval] = ...
fmincon('objfun',x0,[],[],[],[],[],[],'confun',options)
After 38 functi on cal l s, the sol uti on x produced wi th functi on val ue fval i s
x =
9.5474 1.0474
fval =
0.0236
We can eval uate the constrai nts at the sol uti on
[c,ceq] = confun(x)
c=
1.0e15 *
0.8882
0
ceq =
[]
Note that both constrai nt val ues are l ess than or equal to zero, that i s, x
sati sfi es .
Constrained Example with Bounds
The vari abl es i n x can be restri cted to certai n l i mi ts by speci fyi ng si mpl e bound
constrai nts to the constrai ned opti mi zer functi on. For fmincon, the command
x = fmincon('objfun',x0,[],[],[],[],lb,ub,'confun',options);
l i mi ts x to be wi thi n the range lb <= x <= ub.
c x ( ) 0
1 Tutorial
1-10
To restri ct x i n Eq. 1-2 to be gr eater than zero (i .e., ), use the
commands:
x0 = [1,1]; % Make a starting guess at the solution
lb = [0,0]; % Set lower bounds
ub = [ ]; % No upper bounds
options = optimset('LargeScale','off');
[x,fval = ...
fmincon('objfun',x0,[],[],[],[],lb,ub,'confun',options)
[c, ceq] = confun(x)
Note that to pass i n the l ower bounds as the seventh argument to fmincon, you
must speci fy val ues for the thi rd through si xth arguments. I n thi s exampl e, we
speci fi ed [] for these arguments si nce we have no l i near i nequal i ti es or l i near
equal i ti es.
After 13 functi on eval uati ons, the sol uti on produced i s
x =
0 1.5000
fval =
8.5000
c =
0
10
ceq =
[]
When lb or ub contai ns fewer el ements than x, onl y the fi rst correspondi ng
el ements i n x are bounded. Al ternati vel y, i f onl y some of the vari abl es are
bounded, then use inf i n lb for unbounded bel ow vari abl es and inf i n ub for
unbounded above vari abl es. For exampl e,
lb = [inf 0];
ub = [10 inf];
bounds ( has no l ower bound and has no upper bound).
Usi ng inf and inf gi ve better numeri cal resul ts than usi ng a very l arge
posi ti ve number or a very l arge negati ve number to i mpl y l ack of bounds.
Note that the number of functi on eval uati ons to fi nd the sol uti on i s reduced
si nce we further restri cted the search space. Fewer functi on eval uati ons are
usual l y taken when a probl em has more constrai nts and bound l i mi tati ons
x
1
0 , x
2
0
x
1
10 , 0 x
2
x
1
x
2
Medium-Scale Examples
1-11
because the opti mi zati on makes better deci si ons regardi ng step-si ze and
regi ons of feasi bi l i ty than i n the unconstrai ned case. I t i s, therefore, good
practi ce to bound and constrai n probl ems, where possi bl e, to promote fast
convergence to a sol uti on.
Constrained Example with Gradients
Ordi nari l y the medi um-scal e mi ni mi zati on routi nes use numeri cal gradi ents
cal cul ated by fi ni te-di fference approxi mati on. Thi s procedure systemati cal l y
perturbs each of the vari abl es i n order to cal cul ate functi on and constrai nt
parti al deri vati ves. Al ternati vel y, you can provi de a functi on to compute
parti al deri vati ves anal yti cal l y. Typi cal l y, the probl em i s sol ved more
accuratel y and effi ci entl y i f such a functi on i s provi ded.
To sol ve the Eq. 1-2 usi ng anal yti cal l y determi ned gradi ents, do the fol l owi ng:
Step 1: Write an M-file for the objective function and gradient
function [f,G] = objfungrad(x)
f = exp(x(1))*(4*x(1)^2+2*x(2)^2+4*x(1)*x(2)+2*x(2)+1);
% Gradient of the objective function
t = exp(x(1))*(4*x(1)^2+2*x(2)^2+4*x(1)*x(2)+2*x(2)+1);
G = [ t + exp(x(1)) * (8*x(1) + 4*x(2)),
exp(x(1))*(4*x(1)+4*x(2)+2)];
Step 2: Write an M-file for the nonlinear constraints and the gradients of
the nonlinear constraints
function [c,ceq,DC,DCeq] = confungrad(x)
c(1) = 1.5 + x(1) * x(2) x(1) x(2); %inequality constraints
c(2) = x(1) * x(2)10;
% Gradient of the constraints
DC= [x(2)1, x(2);
x(1)1, x(1)];
% No nonlinear equality constraints
ceq=[];
DCeq = [ ];
G contai ns the parti al deri vati ves of the objecti ve functi on, f, returned by
objfungrad(x), wi th respect to each of the el ements i n x:
1 Tutorial
1-12
(1-4)
The col umns of DC contai n the parti al deri vati ves for each respecti ve constrai nt
(i .e., the ith col umn of DC i s the parti al deri vati ve of the ith constrai nt wi th
respect to x). So i n the above exampl e, DC i s
(1-5)
Si nce you are provi di ng the gradi ent of the objecti ve n objfungrad.m and the
gradi ent of the constrai nts i n confungrad.m, you must tel l fmincon that these
M-fi l es contai n thi s addi ti onal i nformati on. Use optimset to turn the
parameters GradObj and GradConstr to 'on' i n our al ready exi sti ng options
structure
options = optimset(options,'GradObj','on','GradConstr','on');
I f you do not set these parameters to 'on' i n the opti ons structure, fmincon wi l l
not use the anal yti c gradi ents.
The arguments lb and ub pl ace l ower and upper bounds on the i ndependent
vari abl es i n x. I n thi s exampl e we have no bound constrai nts and so they are
both set to [].
f
x
-----
e
x
1
4x
1
2
2x
2
2
4x
1
x
2
2x
2
1 + + + + ( ) e
x
1
8x
1
4x
2
+ ( ) +
e
x
1
4x
1
4x
2
2 + + ( )
=
c
1

x
1

--------
c
2

x
1

--------
c
1

x
2

--------
c
2

x
2

--------
x
2
1 x
2

x
1
1 x
1

=
Medium-Scale Examples
1-13
Step 3: Invoke constrained optimization routine
x0 = [1,1]; % Starting guess
options = optimset('LargeScale','off');
options = optimset(options,'GradObj','on','GradConstr','on');
lb = [ ]; ub = [ ]; % No upper or lower bounds
[x,fval] = fmincon('objfungrad',x0,[],[],[],[],lb,ub,...
'confungrad',options)
[c,ceq] = confungrad(x) % Check the constraint values at x
After 20 functi on eval uati ons, the sol uti on produced i s
x =
9.5474 1.0474
fval =
0.0236
c =
1.0e14 *
0.1110
0.1776
ceq =
[]
Gradient Check: Analytic Versus Numeric
When anal yti cal l y determi ned gradi ents are provi ded, you can compare the
suppl i ed gradi ents wi th a set cal cul ated by fi ni te-di fference eval uati on. Thi s i s
parti cul arl y useful for detecti ng mi stakes i n ei ther the objecti ve functi on or the
gradi ent functi on formul ati on.
I f such gradi ent checks are desi red, set the DerivativeCheck parameter to
'on' usi ng optimset:
options = optimset(options,'DerivativeCheck','on');
The fi rst cycl e of the opti mi zati on checks the anal yti cal l y determi ned gradi ents
(of the objecti ve functi on and, i f they exi st, the nonl i near constrai nts). I f they
do not match the fi ni te-di fferenci ng gradi ents wi thi n a gi ven tol erance, a
warni ng message i ndi cates the di screpancy and gi ves the opti on to abort the
opti mi zati on or to conti nue.
1 Tutorial
1-14
Equality Constrained Example
For routi nes that permi t equal i ty constrai nts, nonl i near equal i ty constrai nts
must be computed i n the M-fi l e wi th the nonl i near i nequal i ty constrai nts. For
l i near equal i ti es, the coeffi ci ents of the equal i ti es are passed i n through the
matri x Aeq and the ri ght-hand-si de vector beq.
For exampl e, i f you have the nonl i near equal i ty constrai nt and the
nonl i near i nequal i ty constrai nt , rewri te them as
and then, to sol ve the probl em:
Step 1: Write an M-file objfun.m
function f = objfun(x)
f = exp(x(1))*(4*x(1)^2+2*x(2)^2+4*x(1)*x(2)+2*x(2)+1);
Step 2: Write an M-file confuneq.m for the nonlinear constraints
function [c, ceq] = confuneq(x)
% nonlinear inequality constraints
c = x(1)*x(2) 10;
% nonlinear equality constraints
ceq = x(1)^2 + x(2) 1;
Step 3: Invoke constrained optimization routine
x0 = [1,1]; % Make a starting guess at the solution
options = optimset('LargeScale','off');
[x,fval] = fmincon('objfun',x0,[],[],[],[],[],[],...
'confuneq',options)
[c,ceq] = confuneq(x) % Check the constraint values at x
x
1
2
x
2
+ 1 =
x
1
x
2
10
x
1
2
x
2
1 + 0 =
x
1
x
2
10 0
Medium-Scale Examples
1-15
After 21 functi on eval uati ons, the sol uti on produced i s
x =
0.7529 0.4332
fval =
1.5093
c =
9.6739
ceq =
4.0684e010
Note that ceq i s equal to 0 wi thi n the defaul t tol erance on the constrai nts of
1.0e006 and that c i s l ess than or equal to zero as desi red.
Maximization
The opti mi zati on functi ons fminbnd, fminsearch, fminunc, fmincon,
fgoalattain, fminimax, lsqcurvefit, and lsqnonlin al l perform
mi ni mi zati on of the objecti ve functi on, . Maxi mi zati on i s achi eved by
suppl yi ng the routi nes wi th . Si mi l arl y, to achi eve maxi mi zati on for
quadprog suppl y H and f, and for linprog suppl y f.
Greater than Zero Constraints
The Opti mi zati on Tool box assumes nonl i near i nequal i ty constrai nts are of the
form . Greater than zero constrai nts are expressed as l ess than zero
constrai nts by mul ti pl yi ng them by 1. For exampl e, a constrai nt of the form
i s equi val ent to the constrai nt ; a constrai nt of the form
i s equi val ent to the constrai nt .
Additional Arguments: Avoiding Global Variables
Parameters that woul d otherwi se have to be decl ared as gl obal can be passed
di rectl y to M-fi l e functi ons usi ng addi ti onal arguments at the end of the cal l i ng
sequence.
For exampl e, enteri ng a number of vari abl es at the end of the cal l to fsolve:
[x,fval] = fsolve('objfun',x0,options,P1,P2,...)
f x ( )
f x ( )
C
i
x ( ) 0
C
i
x ( ) 0 C
i
x ( ) 0
C
i
x ( ) b C
i
x ( ) b + 0
1 Tutorial
1-16
passes the arguments di rectl y to the functi ons objfun when i t i s cal l ed from
i nsi de fsolve,
F = objfun(x,P1,P2, ... )
Consi der, for exampl e, fi ndi ng zeros of the functi on ellipj(u,m). The functi on
needs parameter m as wel l as i nput u. To l ook for a zero near u = 3, for m = 0.5
m = 0.5;
options = optimset('Display','off'); % Turn off Display
x = fsolve('ellipj',3,options,m)
returns
x =
3.7081
Then, sol ve for the functi on ellipj.
f = ellipj(x,m)
f =
2.9925e008
The cal l to optimset to get the defaul t opti ons for fsolve i mpl i es that defaul t
tol erances are used and that anal yti c gradi ents are not provi ded.
Multiobjective Examples
The previ ous exampl es i nvol ved probl ems wi th a si ngl e objecti ve functi on. Thi s
secti on demonstrates sol vi ng probl ems wi th mul ti objecti ve functi ons usi ng
lsqnonlin, fminimax, and fgoalattain. I ncl uded i s an exampl e of how to
opti mi ze parameters i n a Si mul i nk model .
Simulink Example
Lets say that you want to opti mi ze the control parameters i n the Si mul i nk
model optsim.mdl. (Thi s model can be found i n the Opti mi zati on Tool box
di rectory. Note that Si mul i nk must be i nstal l ed on your system to l oad thi s
model .) The model i ncl udes a nonl i near process pl ant model ed as a Si mul i nk
bl ock di agram shown i n Fi gure 1-1.
Medium-Scale Examples
1-17
Figure 1-1: Plant with Actuator Saturation
The pl ant i s an under-damped thi rd-order model wi th actuator l i mi ts. The
actuator l i mi ts are a saturati on l i mi t and a sl ew rate l i mi t. The actuator
saturati on l i mi t cuts off i nput val ues greater than 2 uni ts or l ess than 2 uni ts.
The sl ew rate l i mi t of the actuator i s 0.8 uni ts/sec. The open-l oop response of
the system to a step i nput i s shown i n Fi gure 1-2. (You can see thi s response by
openi ng the model (type optsim at the command l i ne), openi ng the Scope bl ock,
and runni ng the si mul ati on. The response pl ots to the Scope.)
Actuator Model
1
y Rate
1.5
50s +a2.s +a1.s+1
3 2
Plant
Limit
1
u
1 Tutorial
1-18
Figure 1-2: Open-Loop Response
The probl em i s to desi gn a feedback control l aw that tracks a uni t step i nput to
the system. The cl osed-l oop pl ant i s entered i n terms of the bl ocks where the
pl ant and actuator have been pl aced i n a hi erarchi cal Subsystem bl ock. A
Scope bl ock di spl ays output trajectori es duri ng the desi gn process. See
Fi gure 1-3.
0 10 20 30 40 50 60 70 80 90 100
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Time (secs)
Medium-Scale Examples
1-19
Figure 1-3: Closed-Loop Model
One way to sol ve thi s probl em i s to mi ni mi ze the error between the output and
the i nput si gnal . The vari abl es are the parameters of the PI D control l er. I f you
onl y need to mi ni mi ze the error at one ti me uni t, i t woul d be a si ngl e objecti ve
functi on. But the goal i s to mi ni mi ze the error for al l ti me steps from 0 to 100,
thus produci ng a mul ti objecti ve functi on (one functi on for each ti me step).
The routi ne lsqnonlin i s used to perform a l east squares fi t on the tracki ng of
the output. Thi s i s defi ned vi a a MATLAB functi on i n the fi l e tracklsq.m
shown bel ow that defi nes the error si gnal . The error si gnal i s yout, the output
computed by cal l i ng sim, mi nus the i nput si gnal 1.
The functi on tracklsq must run the si mul ati on. The si mul ati on can be run
ei ther i n the base workspace or the current workspace, i .e., the workspace of
the functi on cal l i ng sim, whi ch i n thi s case i s tracklsqs workspace. I n thi s
exampl e, the simset command i s used to tel l sim to run the si mul ati on i n the
current workspace by setti ng 'SrcWorkspace' to 'Current'.
To run the si mul ati on i n optsim, the vari abl es Kp, Ki, Kd, a1, and a2 (a1 and a2
are vari abl es i n the Pl ant bl ock) must al l be defi ned. Kp, Ki, and Kd are the
vari abl es we are opti mi zi ng. You can i ni ti al i ze a1 and a2 before cal l i ng
lsqnonlin and then pass these two vari abl es as addi ti onal arguments.
lsqnonlin wi l l then pass a1 and a2 to tracklsq each ti me i t i s cal l ed so you do
not have to use gl obal vari abl es.
After choosi ng a sol ver usi ng the simset functi on, the si mul ati on i s run usi ng
sim. The si mul ati on i s performed usi ng a fi xed-step fi fth-order method to 100
seconds. When the si mul ati on compl etes, the vari abl es tout, xout, and yout
are now i n the current workspace (that i s, the workspace of tracklsq). The
1 Tutorial
1-20
Outport bl ock i s used i n the bl ock di agram model to put yout i nto the current
workspace at the end of the si mul ati on.
Step 1: Write an M-file tracklsq.m
function F = tracklsq(pid,a1,a2)
Kp = pid(1); % Move variables into model parameter names
Ki = pid(2);
Kd = pid(3);
% Choose solver and set model workspace to this function
opt = simset('solver','ode5','SrcWorkspace','Current');
[tout,xout,yout] = sim('optsim',[0 100],opt);
F = yout1; % Compute error signal
Step 2: Invoke optimization routine
optsim % load the model
pid0 = [0.63 0.0504 1.9688] % Set initial values
a1 = 3; a2 = 43; % Initialize Plant variables in model
options = optimset('LargeScale','off','Display','iter',...
'TolX',0.001,'TolFun',0.001);
pid = lsqnonlin('tracklsq', pid0, [], [], options, a1, a2)
% put variables back in the base workspace
Kp = pid(1); Ki = pid(2); Kd = pid(3);
The vari abl e options passed to lsqnonlin defi nes the cri teri a and di spl ay
characteri sti cs. I n thi s case you ask for output, use the medi um-scal e
al gori thm, and gi ve termi nati on tol erances for the step and objecti ve functi on
on the order of 0.001.
Medium-Scale Examples
1-21
The opti mi zati on gi ves the sol uti on for the Proporti onal , I ntegral , and
Deri vati ve (Kp, Ki, Kd) gai ns of the control l er after 73 functi on eval uati ons:
Directional
Iteration Func-count Residual Step-size derivative Lambda
1 3 8.66531 1 3.48
2 10 6.78831 1 0.0634 3.4355
3 19 5.99204 5.5 0.0446 0.28612
4 28 4.74992 5.78 0.0213 0.0227966
5 36 4.51795 1.25 0.0222 0.0744258
6 43 4.5115 0.58 0.00633 0.03445
7 51 4.49455 2.99 0.000688 0.017225
8 58 4.4836 0.915 0.00203 0.0180998
9 66 4.47724 1.22 0.000845 0.00904992
10 73 4.47405 0.801 0.00072 0.0113409
Optimization terminated successfully:
Gradient in the search direction less than tolFun
Gradient less than 10*(tolFun+tolX)
pid =
2.9108 0.1443 12.8161
The resul ti ng cl osed-l oop step response i s shown i n Fi gure 1-4.
1 Tutorial
1-22
Figure 1-4: Closed-Loop Response using lsqnonlin
Note: The cal l to sim resul ts i n a cal l to one of the Si mul i nk ordi nary
di fferenti al equati on (ODE) sol vers. A choi ce must be made about the type of
sol ver to use. From the opti mi zati on poi nt of vi ew, a fi xed-step sol ver i s the
best choi ce i f that i s suffi ci ent to sol ve the ODE. However, i n the case of a sti ff
system, a vari abl e-step method may be requi red to sol ve the ODE. The
numeri cal sol uti on produced by a vari abl e-step sol ver, however, i s not a
smooth functi on of parameters because of step-si ze control mechani sms. Thi s
l ack of smoothness may prevent the opti mi zati on routi ne from convergi ng.
The l ack of smoothness i s not i ntroduced when a fi xed-step sol ver i s used. (For
a further expl anati on, see SolvingOrdinary Differential Equations I --
Nonstiff Problems, by E. Hai rer, S.P. Norsett, G. Wanner, Spri nger-Verl ag,
pages 183-184.) The NCD Blockset i s recommended for sol vi ng mul ti objecti ve
opti mi zati on probl ems i n conjuncti on wi th vari abl e-step sol vers i n Si mul i nk;
i t provi des a speci al numeri c gradi ent computati on that works wi th Si mul i nk
and avoi ds i ntroduci ng the l ack of smoothness probl em.
0 10 20 30 40 50 60 70 80 90 100
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Time (secs)
Medium-Scale Examples
1-23
Another sol uti on approach i s to use the fminimax functi on. I n thi s case, rather
than mi ni mi zi ng the error between the output and the i nput si gnal , you
mi ni mi ze the maxi mum val ue of the output at any ti me t between 0 and 100.
Then i n the functi on trackmmobj the objecti ve functi on i s si mpl y the output
yout returned by the sim command. But mi ni mi zi ng the maxi mum output at
al l ti me steps may force the output far bel ow uni ty for some ti me steps. To keep
the output above 0.95 after the fi rst 20 seconds, i n the constrai nt functi on
trackkmmcon add a constrai nt yout >= 0.95 from t=20 to t=100. Because
constrai nts must be i n the form g<=0, the constrai nt i n the functi on i s
g = yout(20:100)+.95.
Both trackmmobj and trackmmcon use the resul t yout from sim, cal cul ated from
the current pid val ues. The nonl i near constrai nt functi on i s always cal l ed
i mmedi atel y after the objecti ve functi on i n fmincon, fminimax, fgoalattain,
and fseminf wi th the same val ues. Thus you can avoi d cal l i ng the si mul ati on
twi ce by usi ng assignin to assi gn the current val ue of F to the vari abl e
F_TRACKMMOBJ i n the base workspace. Then the fi rst step i n trackmmcon i s to
use evalin to eval uate the vari abl e F_TRACKMMOBJ i n the base workspace, and
assi gn the resul t to F l ocal l y i n trackmmcon.
Step 1: Write an M-file trackmmobj.m to compute objective function
function F = trackmmobj(pid,a1,a2)
Kp = pid(1);
Ki = pid(2);
Kd = pid(3);
% Compute function value
opt = simset('solver','ode5','SrcWorkspace','Current');
[tout,xout,yout] = sim('optsim',[0 100],opt);
F = yout;
assignin('base','F_TRACKMMOBJ',F);
Step 2: Write an M-file trackmmcon.m to compute nonlinear constraints
function [c,ceq] = trackmmcon(pid,a1,a2)
F = evalin('base','F_TRACKMMOBJ');
% Compute constraints
c = F(20:100)+.95;
ceq = [ ];
1 Tutorial
1-24
Note that fminimax wi l l pass a1 and a2 to the objecti ve and constrai nt val ues,
so trackmmcon needs i nput arguments for these vari abl es even though i t does
not use them.
Step 3: Invoke constrained optimization routine
optsim
pid0 = [0.63 0.0504 1.9688]
a1 = 3; a2 = 43;
options = optimset('Display','iter',...
'TolX',0.001,'TolFun',0.001);
pid = fminimax('trackmmobj',pid0,[],[],[],[],[],[],...
'trackmmcon',options,a1,a2)
% put variables back in the base workspace
Kp = pid(1); Ki = pid(2); Kd = pid(3);
resul ti ng i n
Max Directional
Iter F-count {F,constraints} Step-size derivative Procedure
1 5 1.12 1 1.18
2 11 1.264 1 0.172
3 17 1.055 1 0.0128 Hessian
modified
twice
4 23 1.004 1 3.49e005 Hessian
modified
5 29 0.9997 1 1.36e006 Hessian
modified
twice
Optimization terminated successfully:
Search direction less than 2*options.TolX and
maximum constraint violation is less than options.TolCon
Active Constraints:
1
14
182
pid =
0.5894 0.0605 5.5295
Medium-Scale Examples
1-25
The l ast val ue shown i n the MAX{F,constraints} col umn of the output shows
the maxi mum val ue for al l the ti me steps i s 0.9997. The cl osed l oop response
wi th thi s resul t i s shown i n Fi gure 1-5.
Thi s sol uti on di ffers from the lsqnonlin sol uti on as you are sol vi ng di fferent
probl em formul ati ons.
Figure 1-5: Closed-Loop Response using fminimax
Signal Processing Example
Consi der desi gni ng a l i near-phase Fi ni te I mpul se Response (FI R) fi l ter. The
probl em i s to desi gn a l owpass fi l ter wi th magni tude one at al l frequenci es
between 0 and 0.1 Hz and magni tude zero between 0.15 and 0.5 Hz.
The frequency response H(f) for such a fi l ter i s defi ned by
0 10 20 30 40 50 60 70 80 90 100
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Time (secs)
1 Tutorial
1-26
(1-6)
where A(f) i s the magni tude of the frequency response. One sol uti on i s to appl y
a goal attai nment method to the magni tude of the frequency response. Gi ven a
functi on that computes the magni tude, the functi on fgoalattain wi l l attempt
to vary the magni tude coeffi ci ents a(n) unti l the magni tude response matches
the desi red response wi thi n some tol erance. The functi on that computes the
magni tude response i s gi ven i n filtmin.m. Thi s functi on takes a, the
magni tude functi on coeffi ci ents, and w, the di screti zati on of the frequency
domai n we are i nterested i n.
To set up a goal attai nment probl em, you must speci fy the goal and weights
for the probl em. For frequenci es between 0 and 0.1, the goal i s one. For
frequenci es between 0.15 and 0.5, the goal i s zero. Frequenci es between 0.1 and
0.15 are not speci fi ed so no goal s or wei ghts are needed i n thi s range.
Thi s i nformati on i s stored i n the vari abl e goal passed to fgoalattain. The
l ength of goal i s the same as the l ength returned by the functi on filtmin. So
that the goal s are equal l y sati sfi ed, usual l y weight woul d be set to abs(goal).
However, si nce some of the goal s are zero, the effect of usi ng weight=abs(goal)
wi l l force the objecti ves wi th weight 0 to be sati sfi ed as hard constrai nts, and
the objecti ves wi th weight 1 possi bl y to be underattai ned (see The Goal
Attai nment Method secti on of the I ntroduction toAlgorithms chapter).
Because al l the goal s are cl ose i n magni tude, usi ng a weight of uni ty for al l
goal s wi l l gi ve them equal pri ori ty. (Usi ng abs(goal) for the wei ghts i s more
i mportant when the magni tude of goal di ffers more si gni fi cantl y.) Al so, setti ng
options = optimset('GoalsExactAchieve',length(goal));
speci fi es that each objecti ve shoul d be as near as possi bl e to i ts goal val ue
(nei ther greater nor l ess than).
H f ( ) h n ( )e
j 2fn
n 0 =
2M

=
A f ( )e
j 2fM
=
A f ( ) a n ( ) 2fn ( ) cos
n 0 =
M 1

=
Medium-Scale Examples
1-27
Step 1: Write an M-file filtmin.m
function y = filtmin(a,w)
n = length(a);
y = cos(w'*(0:n1)*2*pi)*a ;
Step 2: Invoke optimization routine
% Plot with initial coefficients
a0 = ones(15,1);
incr = 50;
w = linspace(0,0.5,incr);
y0 = filtmin(a0,w);
clf, plot(w,y0.':');
drawnow;
% Set up the goal attainment problem
w1 = linspace(0,0.1,incr) ;
w2 = linspace(0.15,0.5,incr);
w0 = [w1 w2];
goal = [1.0*ones(1,length(w1)) zeros(1,length(w2))];
weight = ones(size(goal));
% Call fgoalattain
options = optimset('GoalsExactAchieve',length(goal));
[a,fval,attainfactor,exitflag] = fgoalattain('filtmin',...
a0,goal,weight,[],[],[],[],[],[],[],options,w0);
% Plot with the optimized (final) coefficients
y = filtmin(a,w);
hold on, plot(w,y,'r')
axis([0 0.5 3 3])
xlabel('Frequency (Hz)')
ylabel('Magnitude Response (dB)')
legend('initial', 'final')
grid on
Compare the magni tude response computed wi th the i ni ti al coeffi ci ents and
the fi nal coeffi ci ents (Fi gure 1-6). Note that you coul d use the remez functi on
i n the Signal ProcessingToolbox to desi gn thi s fi l ter.
1 Tutorial
1-28
Figure 1-6: Magnitude Response with Initial and Final Magnitude Coefficients
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
3
2
1
0
1
2
3
Frequency (Hz)
M
a
g
n
i
t
u
d
e

R
e
s
p
o
n
s
e

(
d
B
)
initial
final
Large-Scale Examples
1-29
Large-Scale Examples
Some of the opti mi zati on functi ons i ncl ude al gori thms for conti nuous
opti mi zati on probl ems especi al l y targeted to l arge probl ems wi th sparsi ty or
structure. The mai n l arge-scal e al gori thms are i terati ve, i .e., a sequence of
approxi mate sol uti ons i s generated. I n each i terati on a l i near system i s
(approxi matel y) sol ved. The l i near systems are sol ved usi ng the sparse matri x
capabi l i ti es of MATLAB and a vari ety of sparse l i near sol uti on techni ques,
both i terati ve and di rect.
General l y speaki ng the l arge-scal e opti mi zati on methods preserve structure
and sparsi ty, usi ng exact deri vati ve i nformati on wherever possi bl e. To sol ve
the l arge-scal e probl ems effi ci entl y, some probl em formul ati ons are restri cted
(such as onl y sol vi ng overdetermi ned l i near or nonl i near systems), or requi re
addi ti onal i nformati on (e.g., the nonl i near mi ni mi zati on al gori thm requi res
the gradi ent be computed i n the user-suppl i ed functi on).
Not al l possi bl e probl em formul ati ons are covered by the l arge-scal e
al gori thms. The fol l owi ng tabl e descri bes what functi onal i ty i s covered by the
l arge-scal e methods. For exampl e, for fmincon, the l arge-scal e al gori thm
covers the case where there are onl y bound constrai nts or onl y l i near
equal i ti es. For each probl em formul ati on, the thi rd col umn descri bes what
addi ti onal i nformati on i s needed for the l arge-scal e al gori thms. For fminunc
and fmincon, the gradi ent must be computed al ong wi th the objecti ve i n the
user-suppl i ed functi on (the gradi ent i s not requi red for the medi um-scal e
al gori thms).
Si nce these methods can al so be used on smal l - to medi um-scal e probl ems that
are not necessari l y sparse, the l ast col umn of the tabl e emphasi zes what
condi ti ons are needed for l arge-scal e probl ems to run effi ci entl y wi thout
exceedi ng your computer systems memory capabi l i ti es, e.g., the l i near
constrai nt matri ces shoul d be sparse. For smal l er probl ems the condi ti ons i n
the l ast col umn are unnecessary.
Several exampl es fol l ow thi s tabl e to further cl ari fy the contents of the tabl e.
1 Tutorial
1-30
Table 1-4: Large-Scale Problem Coverage and Requirements
Function Problem
Formulations
Additional
Information
Needed
For Large Problems
fminunc Must provi de
gradi ent for f(x)
i n fun.
Provi de sparsi ty
structure of the
Hessi an, or compute
the Hessi an i n fun.
The Hessi an shoul d be
sparse.
fmincon

Aeq i s an m-by-n matri x

where
Must provi de
gradi ent for f(x)
i n fun.
Provi de sparsi ty
structure of the
Hessi an, or compute
the Hessi an i n fun.
The Hessi an shoul d be
sparse.
lsqnonlin

F(x) must be overdetermi ned

(have at l east as many
equati ons as vari abl es).
Not appl i cabl e. Provi de sparsi ty
structure of the
Jacobi an, or compute
the Jacobi an i n fun.
The Jacobi an shoul d be
sparse.
f x ( )
x
min
f x ( )
x
min
such that l x u
where l u <
f x ( )
x
min
such that Aeq x beq =
m n.
1
2
--- F x ( )
2
2
x
min
1
2
--- F
i
x ( )
2
i

=
1
2
--- F x ( )
2
2
x
min
1
2
--- F
i
x ( )
2
i

=
such that l x u
where l u <
Large-Scale Examples
1-31
lsqcurvefit

F(x,xdata) must be
overdetermi ned (have at l east
as many equati ons as
vari abl es).
Not appl i cabl e. Provi de sparsi ty
structure of the
Jacobi an, or compute
the Jacobi an i n fun.
The Jacobi an shoul d be
sparse.
fsolve
F(x) must be overdetermi ned
(have at l east as many
equati ons as vari abl es).
Not appl i cabl e. Provi de sparsi ty struc-
ture of the Jacobi an or
compute the Jacobi an
i n fun.
The Jacobi an shoul d be
sparse.
lsqlin
C i s an m-by-n matri x where
i .e., the probl em must
be overdeter mi ned.
Not appl i cabl e. C shoul d be sparse.
Table 1-4: Large-Scale Problem Coverage and Requirements (Continued)
Function Problem
Formulations
Additional
Information
Needed
For Large Problems
1
2
--- F x xdata , ( ) ydata
2
2
x
min
1
2
--- F x xdata , ( ) ydata
2
2
x
min
such that l x u
where l u <
F x ( ) 0 =
C x d
2
x
min
2
such that l x u
where l u <
m n,
1 Tutorial
1-32
I n the exampl es bel ow, many of the M-fi l e functi ons are avai l abl e i n the
Opti mi zati on Tool box optim di rectory. Most of these do not have a fi xed
probl em si ze, i .e., the si ze of your starti ng poi nt xstart wi l l determi ne the si ze
probl em that i s computed. I f your computer system cannot handl e the si ze
suggested i n the exampl es bel ow, use a smal l er di mensi on start poi nt to run
the probl ems. I f the probl ems have upper or l ower bounds or equal i ti es, you
wi l l have to adjust the si ze of those vectors or matri ces as wel l .
linprog Not appl i cabl e. A and Aeq shoul d be
sparse.
quadprog

Aeq i s an m-by-n matri x

where
Not appl i cabl e. H shoul d be sparse.
Aeq shoul d be sparse.
Table 1-4: Large-Scale Problem Coverage and Requirements (Continued)
Function Problem
Formulations
Additional
Information
Needed
For Large Problems
f
T
x
x
min such that
A x b
Aeq x beq =
l x u
1
2
---x
T
Hx f
T
x +
x
min such that
l x u
where l u <
1
2
---x
T
Hx f
T
x +
x
min such that
Aeq x beq =
m n.
Large-Scale Examples
1-33
Nonlinear Equations with Jacobian
Consi der the probl em of fi ndi ng a sol uti on to a system of nonl i near equati ons
whose Jacobi an i s sparse. The di mensi on of the probl em i n thi s exampl e i s
1000. The goal i s to fi nd xsuch that F(x) =0. Assumi ng n=1000, the nonl i near
equati ons are
To sol ve a l arge nonl i near system of equati ons, F(x) = 0, use the l arge-scal e
method avai l abl e i n fsolve.
Step 1: Write an M-file nlsf1.m that computes the objective function values
and the Jacobian
function [F,J] = nlsf1(x);
% Evaluate the vector function
n = length(x);
F = zeros(n,1);
i = 2:(n1);
F(i) = (32*x(i)).*x(i)x(i1)2*x(i+1)1+ 1;
F(n) = (32*x(n)).*x(n)x(n1) + 1;
F(1) = (32*x(1)).*x(1)2*x(2) + 1;
% Evaluate the Jacobian if nargout > 1
if nargout > 1
d = 4*x + 3*ones(n,1); D = sparse(1:n,1:n,d,n,n);
c = 2*ones(n1,1); C = sparse(1:n1,2:n,c,n,n);
e = ones(n-1,1); E = sparse(2:n,1:n-1,e,n,n);
J = C + D + E;
end
Step 2: Call the system of equations solve routine
xstart = ones(1000,1);
fun = 'nlsf1';
options = optimset('Display','iter','Jacobian','on');
[x,fval,exitflag,output] = fsolve(fun,xstart,options);
F 1 ( ) 3x
1
2x
1
2
2x
2
1 + =
F i ( ) 3x
i
2x
i
2
x
i 1
2x
i 1 +
1 + =
F n ( ) 3x
n
2x
n
2
x
n 1
1 + =
1 Tutorial
1-34
A starti ng poi nt i s gi ven as wel l as the functi on name. The l arge-scal e method
i s the defaul t, so i t i s not necessary to speci fy thi s i n the options argument.
Output at each i terati on i s speci fi ed. Fi nal l y, so that fsolve wi l l use the
Jacobi an i nformati on avai l abl e i n nlsf1.m, you need to turn the Jacobian
parameter 'on' usi ng optimset.
These commands di spl ay thi s output:
Optimization terminated successfully:Relative function value
changing by less than OPTIONS.TolFun
A l i near system i s (approxi matel y) sol ved i n each major i terati on usi ng the
precondi ti oned conjugate gradi ent method. The defaul t val ue for the
PrecondBandWidth i s 0 i n opti ons, so a di agonal precondi ti oner i s used. (The
PrecondBandWidth speci fi es the bandwi dth of the precondi ti oni ng matri x. A
bandwi dth of 0 means there i s onl y one di agonal i n the matri x.)
From the fi rst-order opti mal i ty val ues, fast l i near convergence occurs. The
number of CG i terati ons requi red per major i terati on i s l ow, at most 5 for a
probl em of 1000 di mensi ons, i mpl yi ng the l i near systems are not very di ffi cul t
to sol ve i n thi s case (though more work i s requi red as convergence progresses).
I t i s possi bl e to overri de the defaul t choi ce of precondi ti oner (di agonal ) by
choosi ng a banded precondi ti oner through the use of the parameter
PrecondBandWidth. I f you want to use a tri di agonal precondi ti oner, i .e., a
precondi ti oni ng matr i x wi th thr ee di agonal s (or bandwi dth of one), set
PrecondBandWidth to the val ue 1:
options = optimset('Display','iter','Jacobian','on',...
'PrecondBandWidth',1) ;
[x,fval,exitflag,output] = fsolve(fun,xstart,options) ;
I n thi s case the output i s
Iteration Func-count f(x)
Norm of
step
First-order
optimality CG-iterations
1 2 1011 1 19 0
2 3 16.1942 7.91898 2.35 3
3 4 0.0228027 1.33142 0.291 3
4 5 0.000103359 0.0433329 0.0201 4
5 6 7.3792e007 0.0022606 0.000946 4
6 7 4.02299e010 0.000268381 4.12e005 5
Large-Scale Examples
1-35
Optimization terminated successfully:
Relative function value changing by less than OPTIONS.TolFun
Note that al though the same number of i terati ons takes pl ace, the number of
PCG i terati ons has dropped, so l ess work i s bei ng done per i terati on.
Nonlinear Equations with Jacobian Sparsity Pattern
I n the precedi ng exampl e the functi on nlsf1 computes the Jacobi an J, a sparse
matri x, al ong wi th the eval uati on of F. What i f the code to compute the
Jacobi an i s not avai l abl e? By defaul t, i f you do not i ndi cate the Jacobi an can be
computed i n nlsf1 (usi ng the Jacobian parameter i n options), fsolve,
lsqnolin, and lsqcurvefit wi l l i nstead use fi ni te-di fferenci ng to approxi mate
the Jacobi an.
I n order for thi s fi ni te-di fferenci ng to be as effi ci ent as possi bl e, the sparsi ty
pattern of the Jacobi an shoul d be suppl i ed, usi ng the JacobPattern parameter
i n options. That i s, suppl y a sparse matri x Jstr whose nonzero entri es
correspond to nonzeroes of the Jacobi an for al l x. I ndeed, the nonzeroes of Jstr
can correspond to a superset of the nonzero l ocati ons of J ; however, i n general
the computati onal cost of the sparse fi ni te-di fference procedure wi l l i ncrease
wi th the number of nonzeroes of Jstr.
Provi di ng the sparsi ty pattern can drasti cal l y reduce the ti me needed to
compute the fi ni te-di fferenci ng on l arge probl ems. I f the sparsi ty pattern i s not
provi ded (and the Jacobi an i s not computed i n objecti ve functi on ei ther) then,
i n thi s probl em nlsfs1, the fi ni te-di fferenci ng code wi l l attempt to compute al l
1000-by-1000 entri es i n the Jacobi an. But i n thi s case there are onl y 2998
nonzeros, substanti al l y l ess than the 1,000,000 possi bl e nonzeros the
fi ni te-di fferenci ng code wi l l attempt to compute. I n other words, thi s probl em
i s sol vabl e i f the sparsi ty pattern i s provi ded. I f not, most computers wi l l run
out of memory when the ful l dense fi ni te-di fferenci ng i s attempted. On most
smal l probl ems, i t i s not essenti al to provi de the sparsi ty structure.
Iteration Func-count f(x) Norm of
step
First-order
Optimality
CG-iterations
1 2 1011 1 19 0
2 3 16.0839 7.92496 1.92 1
3 4 0.0458181 1.3279 0.579 1
4 5 0.000101184 0.0631898 0.0203 2
5 6 3.16615e007 0.00273698 0.00079 2
6 7 9.72481e010 0.00018111 5.82e005 2
1 Tutorial
1-36
Suppose the sparse matri x Jstr, computed pr evi ousl y, has been saved i n fi l e
nlsdat1.mat. The fol l owi ng dri ver cal l s fsolve appl i ed to nlsf1a whi ch i s the
same as nlsf1 except onl y the functi on val ues are returned; sparse
fi ni te-di fferenci ng i s used to esti mate the sparse Jacobi an matri x as needed.
Step 1: Write an M-file nlsf1a.m that computes the objective function
values
function F = nlsf1a(x);
% Evaluate the vector function
n = length(x);
F = zeros(n,1);
i = 2:(n1);
F(i) = (32*x(i)).*x(i)x(i1)2*x(i+1) + 1;
F(n) = (32*x(n)).*x(n)x(n1) + 1;
F(1) = (32*x(1)).*x(1)2*x(2) + 1;
Step 2: Call the system of equations solve routine
xstart = ones(1000,1);
fun = 'nlsf1a';
load nlsdat1 % Get Jstr
options = optimset('Display','iter','JacobPattern',Jstr,...
'PrecondBandWidth',1);
[x,fval,exitflag,output] = fsolve(fun,xstart,options);
I n thi s case, the output di spl ayed i s
Optimization terminated successfully:
Relative function value changing by less than OPTIONS.TolFun
Al ternati vel y, i t i s possi bl e to choose a sparse di rect l i near sol ver (i .e., a sparse
QR factori zati on) by i ndi cati ng a compl ete precondi ti oner, i .e., i f we set
Iteration Func-count f(x)
Norm of
step
First-order
optimality CG-iterations
1 6 1011 1 19 0
2 11 16.0839 7.92496 1.92 1
3 16 0.0458181 1.3279 0.579 1
4 21 0.000101184 0.0631897 0.0203 2
5 26 3.16615e007 0.00273698 0.00079 2
6 31 9.72482e010 0.00018111 5.82e005 2
Large-Scale Examples
1-37
PrecondBandWidth to Inf, then a sparse di rect l i near sol ver wi l l be used
i nstead of a precondi ti oned conjugate gradi ent i terati on
xstart = ones(1000,1);
fun = 'nlsf1a';
load nlsdat1 % Get Jstr
options = optimset('Display','iter','JacobPattern',Jstr,...
'PrecondBandWidth',inf);
[x,fval,exitflag,output] = fsolve(fun,xstart,options);
and the resul ti ng di spl ay i s
Optimization terminated successfully:
Relative function value changing by less than OPTIONS.TolFun
When the sparse di rect sol vers are used, the CG i terati on wi l l be 1 for that
(major) i terati on, as shown i n the output under CG-iterations. Noti ce that the
fi nal opti mal i ty and f(x) val ue (whi ch for fsolve, f(x), i s the
sum-of-the-squares of the functi on val ues) are cl oser to zero than usi ng the
PCG method, whi ch i s often the case.
Nonlinear Least-Squares with Full Jacobian
Sparsity Pattern
The l arge-scal e methods i n lsqnonlin, lsqcurvefit, and fsolve can be used
wi th smal l - to medi um-scal e probl ems wi thout computi ng the Jacobi an i n fun
or provi di ng the Jacobi an sparsi ty pattern. (Thi s exampl e al so appl i es to the
case of usi ng fmincon or fminunc wi thout computi ng the Hessi an or suppl yi ng
the Hessi an sparsi ty pattern.) How smal l i s smal l - to medi um-scal e? No
absol ute answer i s avai l abl e, as i t depends on the amount of vi rtual memory
avai l abl e i n your computer system confi gurati on.
Suppose your probl em has m equati ons and n unknowns. I f the command
J = sparse(ones(m,n)) causes an OUT OF MEMORY error on your machi ne, then
Iteration Func-count f(x)
Norm of
step
First-order
optimality
CG-iterations
1 6 1011 1 19 0
2 11 15.9018 7.92421 1.89 1
3 16 0.0128163 1.32542 0.0746 1
4 21 1.73537e008 0.0397925 0.000196 1
5 26 1.13136e018 4.55542e005 2.76e009 1
1 Tutorial
1-38
thi s i s certai nl y too l arge a probl em. I f i t does not resul t i n an error, the
probl em may sti l l be too l arge, but you can onl y fi nd out by runni ng i t and
seei ng i f MATLAB i s abl e to run wi thi n the amount of vi rtual memory
avai l abl e on your system.
Lets say you have a smal l probl em wi th 10 equati ons and 2 unknowns, such as
fi nd x that mi ni mi zes
starti ng at the poi nt x = [0.3, 0.4].
Because lsqnonlin assumes that the sum-of-squares i s not expl i ci tl y formed i n
the user functi on, the functi on passed to lsqnonlin shoul d i nstead compute the
vector val ued functi on
for (that i s, F shoul d have k components).
Step 1: Write an M-file myfun.m that computes the objective function
values
function F = myfun(x)
k = 1:10;
F = 2 + 2*kexp(k*x(1))exp(k*x(2));
Step 2: Call the nonlinear least-squares routine
x0 = [0.3 0.4] % Starting guess
[x,resnorm] = lsqnonlin('myfun',x0) % Invoke optimizer
Si nce the Jacobi an i s not computed i n myfun.m (and the Jacobian parameter i n
options i s 'off' by defaul t), and no Jacobi an sparsi ty pattern i s provi ded
usi ng the JacobPattern parameter i n options, lsqnonlin wi l l cal l the
l arge-scal e method, the defaul t for lsqnonlin, wi th JacobPattern set to
Jstr = sparse(ones(10,2)). When the fi ni te-di fferenci ng routi ne i s cal l ed the
fi rst ti me, i t wi l l detect that Jstr i s actual l y a dense matri x, i .e., that no speed
benefi t i s der i ved fr om stori ng i t as a sparse matri x, and from then on wi l l use
Jstr = ones(10,2) (a ful l matri x) for the opti mi zati on computati ons.
2 2k e
kx
1
e
kx
2
+ ( )
2
k 1 =
10

F
k
x ( ) 2 2k e
kx
1
e
kx
2
+ =
k 1 to 10 =
Large-Scale Examples
1-39
After about 24 functi on eval uati ons, thi s exampl e gi ves the sol uti on:
x =
0.2578 0.2578
resnorm % residual or sum of squares
resnorm =
124.3622
Most computer systems wi l l be abl e to handl e much l arger ful l probl ems, say
i nto the 100s of equati ons and vari abl es. But if there i s some sparsi ty structure
i n the Jacobi an (or Hessi an) that can be taken advantage of, the l arge-scal e
methods wi l l al ways run faster i f thi s i nformati on i s provi ded.
Nonlinear Minimization with Gradient and Hessian
Thi s exampl e i nvol ves a sol vi ng a nonl i near mi ni mi zati on probl em wi th a
tri di agonal Hessi an matri x H(x) fi rst computed expl i ci tl y, and then by
provi di ng the Hessi ans sparsi ty structure for the fi ni te-di fferenci ng routi ne.
The probl em i s to fi nd x to mi ni mi ze
(1-7)
where n=1000.
Step 1: Write an M-file brownfgh.m that computes the objective function,
the gradient of the objective, and the sparse tridiagonal Hessian matrix
Thi s fi l e i s rather l engthy and i s not i ncl uded here. You can vi ew the code wi th
the command
type brownfgh
Because brownfgh computes the gradi ent and Hessi an val ues as wel l as the
objecti ve functi on, you need to use optimset to i ndi cate thi s i nformati on i s
avai l abl e i n brownfgh usi ng the GradObj and Hessian parameters.
f x ( ) x
i
2
( )
x
i 1 +
2
1 + ( )
x
i 1 +
2
( )
x
i
2
1 + ( )
+
i 1 =
n 1

=
1 Tutorial
1-40
Step 2: Call a nonlinear minimization routine with a starting point xstart
n = 1000;
xstart = ones(n,1);
xstart(2:2:n,1) = 1;
options = optimset('GradObj',...
'on','Hessian','on');
[x,fval,exitflag,output] = fminunc('brownfgh',xstart,options);
Thi s 1000 vari abl e probl em i s sol ved i n 8 i terati ons and 7 conjugate gradi ent
i terati ons wi th a posi ti ve exitflag i ndi cati ng convergence; the fi nal functi on
val ue and measure of opti mal i ty at the sol uti on x are both cl ose to zero (for
fminunc, the fi rst order opti mal i ty i s the gradi ent of the functi on, whi ch i s zero
at a l ocal mi ni mum):
exitflag =
1
fval =
2.8709e017
output.iterations
ans =
8
output.cgiterations
ans =
7
output.firstorderopt
ans =
4.7948e010
Nonlinear Minimization with Gradient and Hessian
Sparsity Pattern
Next we sol ve the same probl em but the Hessi an matri x i s now approxi mated
by sparse fi ni te-di fferences i nstead of expl i ci t computati on. To use the
l arge-scal e method i n fminunc, you must compute the gradi ent i n fun; i t i s not
optional as i n the medi um-scal e method.
The M-fi l e functi on brownfg computes the objecti ve functi on and gradi ent.
Large-Scale Examples
1-41
Step 1: Write an M-file brownfg.m that computes the objective function
and the gradient of the objective
function [f,g] = brownfg(x,dummy)
% BROWNFG Nonlinear minimization test problem
%
% Evaluate the function.
n=length(x); y=zeros(n,1);
i=1:(n1);
y(i)=(x(i).^2).^(x(i+1).^2+1) + ...
(x(i+1).^2).^(x(i).^2+1);
f=sum(y);
% Evaluate the gradient if nargout > 1
if nargout > 1
i=1:(n1); g = zeros(n,1);
g(i) = 2*(x(i+1).^2+1).*x(i).* ...
((x(i).^2).^(x(i+1).^2))+ ...
2*x(i).*((x(i+1).^2).^(x(i).^2+1)).* ...
log(x(i+1).^2);
g(i+1) = g(i+1) + ...
2*x(i+1).*((x(i).^2).^(x(i+1).^2+1)).* ...
log(x(i).^2) + ...
2*(x(i).^2+1).*x(i+1).* ...
((x(i+1).^2).^(x(i).^2));
end
To al l ow effi ci ent computati on of the sparse fi ni te-di fference approxi mati on of
the Hessi an matri x H(x), the sparsi ty structure of H must be predetermi ned.
I n thi s case assume thi s structure, Hstr, a sparse matri x, i s avai l abl e i n fi l e
brownhstr.mat. Usi ng the spy command you can see that Hstr i s i ndeed sparse
(onl y 2998 nonzeros). Use optimset to set the HessPattern parameter to Hstr.
When a probl em as l arge as thi s has obvi ous sparsi ty structure, not setti ng the
HessPattern parameter wi l l requi re a huge amount of unnecessary memory
and computati on si nce fminunc wi l l attempt to use fi ni te-di fferenci ng on a ful l
Hessi an matri x of one mi l l i on nonzero entri es.
You must al so set the GradObj parameter to 'on' usi ng optimset si nce the
gradi ent i s computed i n brownfg.m. Then to execute fminunc:
1 Tutorial
1-42
Step 2: Call a nonlinear minimization routine with a starting point xstart
fun = 'brownfg';
load brownhstr % get Hstr, structure of the Hessian
spy(Hstr) % view the sparsity structure of Hstr
n = 1000;
xstart = ones(n,1);
xstart(2:2:n,1) = 1;
options = optimset('GradObj','on','HessPattern',Hstr);
[x,fval,exitflag,output] = fminunc(fun,xstart,options);
Thi s 1000 vari abl e probl em i s sol ved i n 8 i terati ons and 7 conjugate gradi ent
i terati ons wi th a posi ti ve exitflag i ndi cati ng convergence; the fi nal functi on
val ue and measure of opti mal i ty at the sol uti on x are both cl ose to zero (for
fminunc, the fi rst order opti mal i ty i s the gradi ent of the functi on, whi ch i s zero
at a l ocal mi ni mum):
exitflag =
1
fval =
7.4738e017
output.iterations
ans =
8
output.cgiterations
ans =
7
output.firstorderopt
ans =
7.9822e010
Large-Scale Examples
1-43
Nonlinear Minimization with Bound Constraints and
Banded Preconditioner
The goal i n thi s probl em i s to mi ni mi ze the nonl i near functi on
such that , where n i s 800 (n shoul d be a mul ti pl e of 4),
p=7/ 3, and x
0
=x
n+1
=0.
Step 1: Write an M-file tbroyfg.m that computes the objective function and
the gradient of the objective
The M-fi l e functi on, tbroyfg.m, computes the functi on val ue and gradi ent.
Thi s fi l e i s rather l engthy and i s not i ncl uded here. You can see the code for thi s
functi on usi ng the command
type tbroyfg
f x ( ) 1 3 2x
i
( )x
i
x
i 1
x
i 1 +
1 +
p
i 1 =
n

x
i
x
i n 2 +
+
p
i 1 =
n
2
---

+ + =
10.0 x
i
10.0
1 Tutorial
1-44
The sparsi ty pattern of the Hessi an matri x has been predetermi ned and stored
i n the fi l e tbroyhstr.mat. The sparsi ty str ucture for the Hessi an of thi s
probl em i s banded, as you can see i n the spy pl ot bel ow:
load tbroyhstr
spy(Hstr):
Large-Scale Examples
1-45
The center stri pe i s i tsel f a 5-banded matri x
spy(Hstr(1:20,1:20))
Use optimset to set the HessPattern parameter to Hstr. When a probl em as
l arge as thi s has obvi ous sparsi ty structure, not setti ng the HessPattern
parameter wi l l requi re a huge amount of unnecessary memory and
computati on si nce fmincon wi l l attempt to use fi ni te-di fferenci ng on a ful l
Hessi an matri x of 640,000 nonzero entri es.
You must al so set the GradObj parameter to 'on' usi ng optimset si nce the
gradi ent i s computed i n tbroyfg.m. Then to execute fmincon:
Step 2: Call a nonlinear minimization routine with a starting point xstart
fun = 'tbroyfg';
load tbroyhstr % get Hstr, structure of the Hessian
n = 800;
xstart = ones(n,1); xstart(2:2:n) = 1;
lb = 10*ones(n,1); ub = lb;
options = optimset('GradObj','on','HessPattern',Hstr);
[x,fval,exitflag,output] = ...
fmincon('tbroyfg',xstart,[],[],[],[],lb,ub,[],options);
1 Tutorial
1-46
After ei ght i terati ons, the exitflag, fval and output val ues are:
exitflag =
1
fval =
270.4790
output =
iterations: 8
funcCount: 8
cgiterations: 18
firstorderopt: 0.0163
algorithm: 'large-scale: trust-region reflective Newton'
For bound constrai ned probl ems, the fi rst-order opti mal i ty i s the i nfi ni ty norm
of v.*g, where v i s defi ned as i n Box Constrai nts i n Chapter 3, and g i s the
gradi ent.
Because of the 5-banded center stri pe, you can i mprove the sol uti on by usi ng a
5-banded precondi ti oner i nstead of the defaul t di agonal precondi ti oner. Usi ng
the optimset functi on, reset the PrecondBandWidth parameter to two and sol ve
the probl em agai n. (The bandwi dth i s the number of upper (or l ower)di agonal s,
not counti ng the mai n di agonal .)
fun = 'tbroyfg';
load tbroyhstr % get Hstr, structure of the Hessian
n = 800;
xstart = ones(n,1); xstart(2:2:n,1) = 1;
lb = 10*ones(n,1); ub = lb;
options = optimset('GradObj','on','HessPattern',Hstr, ...
'PrecondBandWidth',2);
[x,fval,exitflag,output] = ...
fmincon('tbroyfg',xstart,[],[],[],[],lb,ub,[],options);
Large-Scale Examples
1-47
The number of i terati ons actual l y goes up by two; however the total number of
CG i terati ons drops from 18 to 15. The fi rst-order opti mal i ty measure i s
reduced by a factor of 1e3.
exitflag =
1
fval =
2.7048e+002
output =
iterations: 10
funcCount: 10
cgiterations: 15
firstorderopt: 7.5339e005
algorithm: 'large-scale: trust-region reflective Newton'
Nonlinear Minimization with Equality Constraints
The l arge-scal e method for fmincon can handl e equal i ty constrai nts i f no other
constrai nts exi st. Suppose you want to mi ni mi ze the same objecti ve i n Eq. 1-7,
whi ch i s coded i n the functi on brownfgh.m, where n =1000, such that
for Aeqthat has 100 equati ons (so Aeqi s a 100-by-1000 matri x).
Step 1: Write an M-file brownfgh.m that computes the objective function,
the gradient of the objective, and the sparse tridiagonal Hessian matrix
As before, thi s fi l e i s rather l engthy and i s not i ncl uded here. You can vi ew the
code wi th the command
type brownfgh
Because brownfgh computes the gradi ent and Hessi an val ues as wel l as the
objecti ve functi on, you need to use optimset to i ndi cate thi s i nformati on i s
avai l abl e i n brownfgh usi ng the GradObj and Hessian parameters.
The sparse matri x Aeq and vector beq are avai l abl e i n the fi l e browneq.mat
load browneq
Aeq x beq =
1 Tutorial
1-48
The l i near constrai nt system i s 100-by-1000, has unstructured sparsi ty (use
spy(Aeq) to vi ew the sparsi ty structure) and i s not too badl y i l l -condi ti oned:
condest(Aeq*Aeq')
ans =
2.9310e+006
Step 2: Call a nonlinear minimization routine with a starting point xstart
fun = 'brownfgh';
load browneq % get Aeq and beq, the linear equalities
n = 1000;
xstart = ones(n,1); xstart(2:2:n) = 1;
options = optimset('GradObj','on','Hessian','on', ...
'PrecondBandWidth', inf);
[x,fval,exitflag,output] = ...
fmincon('brownfgh',xstart,[],[],Aeq,beq,[],[],[],options);
Setti ng the parameter PrecondBandWidth to inf wi l l cause a sparse di rect
sol ver to be used i nstead of precondi ti oned conjugate gradi ents.
The exitflag i ndi cates convergence wi th the fi nal functi on val ue fval after 16
i terati ons
exitflag =
1
fval =
205.9313
output =
iterations: 16
funcCount: 16
cgiterations: 14
firstorderopt: 2.1434e004
algorithm: 'large-scale: projected trust-region Newton'
The l i near equal i ti es are sati sfi ed at x
norm(Aeq*x-beq)
ans =
1.1913e012
Large-Scale Examples
1-49
Quadratic Minimization with Bound Constraints
To mi ni mi ze a l arge-scal e quadrati c wi th upper and l ower bounds, you can use
the quadprog functi on.
The pr obl em stored i n the MAT-fi l e qpbox1.mat i s a posi ti ve defi ni te quadrati c,
and the Hessi an matri x H i s tri di agonal , subject to upper (ub) and l ower (lb)
bounds.
Load the Hessian and define f, lb, ub. Call a quadratic minimization
routine with a starting point xstart
load qpbox1 % Get H
lb = zeros(400,1); lb(400) = inf;
ub = 0.9*ones(400,1); ub(400) = inf;
f = zeros(400,1); f([1 400]) = 2;
xstart = 0.5*ones(400,1);
[x,fval,exitflag,output] = ...
quadprog(H,f,[],[],[],[],lb,ub,xstart);
Looki ng at the resul ti ng val ues of exitflag and output
exitflag =
1
output =
firstorderopt: 7.8435e006
iterations: 20
cgiterations: 1809
algorithm: 'large-scale: reflective trust-region'
you can see that whi l e convergence occurred i n 20 i terati ons, the hi gh number
of CG i terati ons i ndi cates that the cost of the l i near system sol ve i s hi gh. I n
l i ght of thi s cost, one strategy woul d be to l i mi t the number of CG i terati ons per
opti mi zati on i terati on. The defaul t number i s the di mensi on of the probl em
di vi ded by two, 200 for thi s probl em. Suppose you l i mi t i t to 50 usi ng the
MaxPCGIter fl ag i n options
options = optimset('MaxPCGIter',50);
[x,fval,exitflag,output] = ...
quadprog(H,f,[],[],[],[],lb,ub,xstart,options);
1 Tutorial
1-50
Thi s ti me convergence sti l l occurs and the total number of CG i terati ons (1547)
has dropped.
exitflag =
1
output =
firstorderopt: 2.3821e005
iterations: 36
cgiterations: 1547
algorithm: 'large-scale: reflective trust-region'
A second strategy woul d be to use a di rect sol ver at each i terati on by setti ng
the PrecondBandWidth parameter to inf.
options = optimset('PrecondBandWidth',inf);
[x,fval,exitflag,output] = ...
quadprog(H,f,[],[],[],[],lb,ub,xstart,options);
Now the number of i terati ons has dropped to 10.
exitflag =
1
output =
firstorderopt: 4.8955e007
iterations: 10
cgiterations: 9
algorithm: 'large-scale: reflective trust-region'
Usi ng a di rect sol ve at each i terati on usual l y causes the number of i terati ons
to decrease, but often takes more ti me per i terati on. For thi s probl em, the
trade-off i s benefi ci al as the ti me for quadprog to sol ve the probl em decreases
by a factor of 10.
Linear Least-Squares with Bound Constraints
Many si tuati ons gi ve ri se to sparse l i near l east-squares probl ems, often wi th
bounds on the vari abl es. The next probl em requi res that the vari abl es be
nonnegati ve. Thi s probl em comes from fi tti ng a functi on approxi mati on to a
pi ecewi se l i near spl i ne. Speci fi cal l y, parti cl es are scattered on the uni t square.
The functi on to be approxi mated i s eval uated at these poi nts, and a pi ecewi se
Large-Scale Examples
1-51
l i near spl i ne approxi mati on i s constructed under the condi ti on that (l i near)
coeffi ci ents are not negati ve. There are 2000 equati ons to fi t on 400 vari abl es.
load particle % Get C, d
lb = zeros(400,1);
[x,resnorm,residual,exitflag,output] = ...
lsqlin(C,d,[],[],[],[],lb);
The defaul t di agonal precondi ti oni ng works fai rl y wel l .
exitflag =
1
resnorm =
22.5794
output =
algorithm: 'large-scale: trust-region reflective Newton'
firstorderopt: 2.7870e-005
iterations: 10
cgiterations: 42
The fi rst-order opti mal i ty can be i mproved (decreased) by usi ng a sparse
QR-factori zati on i n each i terati on: set PrecondBandWidth to inf.
options = optimset('PrecondBandWidth',inf);
[x,resnorm,residual,exitflag,output] = ...
lsqlin(C,d,[],[],[],[],lb,[],[],options);
The number of i terati ons and the fi rst-order opti mal i ty both decrease
exitflag =
1
resnorm =
22.5794
output =
algorithm: 'large-scale: trust-region reflective Newton'
firstorderopt: 5.5907e015
iterations: 12
cgiterations: 11
1 Tutorial
1-52
Linear Programming with Equalities and
Inequalities
The probl em i s
and you can l oad the matri ces and vectors A, Aeq, b, beq, f and the l ower
bounds lb i nto the MATLAB workspace wi th
load sc50b
Thi s probl em i n sc50b.mat has 48 vari abl es, 30 i nequal i ti es and 20 equal i ti es.
You can use linprog to sol ve the probl em.
[x,fval,exitflag,output] = ...
linprog(f,A,b,Aeq,beq,lb,[],[],optimset('Display','iter'));
Si nce the i terati ve di spl ay was set usi ng optimset, the resul ts pri nted to the
command l i ne wi ndow are
Residuals: Primal Dual Duality Total
Infeas Infeas Gap Rel
A*x-b A'*y+z-f x'*z Error
---------------------------------------------------
Iter 0: 1.50e+003 2.19e+001 1.91e+004 1.00e+002
Iter 1: 1.15e+002 2.94e015 3.62e+003 9.90e001
Iter 2: 1.16e012 2.21e015 4.32e+002 9.48e001
Iter 3: 3.23e012 5.16e015 7.78e+001 6.88e001
Iter 4: 5.78e011 7.61e016 2.38e+001 2.69e001
Iter 5: 9.31e011 1.84e015 5.05e+000 6.89e002
Iter 6: 2.96e011 1.62e016 1.64e001 2.34e003
Iter 7: 1.51e011 2.74e016 1.09e005 1.55e007
Iter 8: 1.51e012 2.37e016 1.09e011 1.51e013
Optimization terminated successfully.
For thi s probl em the l arge-scal e l i near programmi ng al gori thm qui ckl y reduces
the scal ed resi dual s bel ow the defaul t tol erance of 1e08.
f
T
x min such that
Aeq x beq =
A x b
x 0
Large-Scale Examples
1-53
The exitflag val ue i s posi ti ve tel l i ng you linprog converged. You can al so get
the fi nal functi on val ue i n fval and the number of i terati ons i n
output.iterations.
exitflag =
1
fval =
70.0000
output =
iterations: 8
cgiterations: 0
algorithm: 'lipsol'
Linear Programming with Dense Columns in the
Equalities
The probl em i s
and you can l oad the matri ces and vectors Aeq, beq, f, lb, and ub i nto the
MATLAB workspace wi th
load densecolumns
The probl em i n densecolumns.mat has 1677 vari abl es and 627 equal i ti es wi th
l ower bounds on al l the vari abl es, and upper bounds on 399 of the vari abl es.
The equal i ty matri x Aeq has dense col umns among i ts fi rst 25 col umns, whi ch
i s easy to see wi th a spy pl ot.
spy(Aeq)
f
T
x min such that
Aeq x beq =
l b x ub
1 Tutorial
1-54
You can use linprog to sol ve the probl em.
[x,fval,exitflag,output] = ...
linprog(f,[],[],Aeq,beq,lb,ub,[],optimset('Display','iter'));
Si nce the i terati ve di spl ay was set usi ng optimset, the resul ts pri nted to the
command l i ne wi ndow are
Residuals: Primal Dual Upper Duality Total
Infeas Infeas Bounds Gap Rel
A*x-b A'*y+z-w-f {x}+s-ub x'*z+s'*w Error
-------------------------------------------------------------
Iter 0: 1.67e+003 8.11e+002 1.35e+003 5.30e+006 2.92e+001
Iter 1: 1.37e+002 1.33e+002 1.11e+002 1.27e+006 2.48e+000
Iter 2: 3.56e+001 2.38e+001 2.89e+001 3.42e+005 1.99e+000
Iter 3: 4.86e+000 8.88e+000 3.94e+000 1.40e+005 1.89e+000
Iter 4: 4.24e-001 5.89e-001 3.44e-001 1.91e+004 8.41e-001
Iter 5: 1.23e-001 2.02e-001 9.97e-002 8.41e+003 5.79e-001
Iter 6: 3.98e-002 7.91e-002 3.23e-002 4.05e+003 3.52e-001
Iter 7: 7.25e-003 3.83e-002 5.88e-003 1.85e+003 1.85e-001
Iter 8: 1.47e-003 1.34e-002 1.19e-003 8.12e+002 8.52e-002
Iter 9: 2.52e-004 3.39e-003 2.04e-004 2.78e+002 2.99e-002
Iter 10: 3.46e-005 1.08e-003 2.81e-005 1.09e+002 1.18e-002
Iter 11: 6.95e-007 1.53e-012 5.64e-007 1.48e+001 1.62e-003
Iter 12: 1.04e-006 2.26e-012 3.18e-008 8.32e-001 9.09e-005
Iter 13: 3.08e-006 1.23e-012 3.86e-009 7.26e-002 7.94e-006
Iter 14: 3.75e-007 1.09e-012 6.53e-012 1.11e-003 1.21e-007
Iter 15: 5.21e-008 1.30e-012 3.27e-013 8.62e-008 9.15e-010
Optimization terminated successfully.
You can see the returned val ues of exitflag, fval, and output .
exitflag =
1
fval =
9.1464e+003
output =
iterations: 15
cgiterations: 225
algorithm: 'lipsol'
Large-Scale Examples
1-55
Thi s ti me the number of PCG i terati ons (i n output.cgiterations) i s nonzero
because the dense col umns i n Aeq are detected. I nstead of usi ng a sparse
Chol esky factori zati on, linprog tri es to use the Sherman-Morri son formul a to
sol ve a l i near system i nvol vi ng Aeq*Aeq'. I f the Sherman-Morri son formul a
does not gi ve a sati sfactory resi dual , a PCG i terati on i s used. See Mai n
Al gori thm i n the Large-Scal e Li near Programmi ng secti on of Chapter 3.
1 Tutorial
1-56
Default Parameter Settings
The options structure contai ns parameters used i n the opti mi zati on routi nes.
I f, on the fi rst cal l to an opti mi zati on routi ne, the options structure i s not
provi ded, or i s empty, a set of defaul t parameters i s generated.
Some of the defaul t opti ons parameters are cal cul ated usi ng factors based on
probl em si ze, such as MaxFunEvals. Some parameters are dependent on the
speci fi c opti mi zati on routi ne and are documented i n Chapter 4. The
parameters i n the options structure are shown i n Tabl e 4-3 i n Chapter 4.
Changing the Default Settings
The functi on optimset creates or updates an OPTIONS vari abl e to pass to the
vari ous opti mi zati on functi ons. The arguments to the optimset functi on are
parameter name and parameter val ue pai rs, such as TolX and 1e4. Any
unspeci fi ed properti es have defaul t val ues. You need to type onl y enough
l eadi ng characters to defi ne the parameter name uni quel y. Case i s i gnored for
parameter names. For parameter val ues that are stri ngs, however, case and
the exact stri ng are necessary.
help optimset provi des i nformati on that defi nes the di fferent parameters and
descri bes how to use them.
Here are some exampl es of the use of optimset.
Returning All Parameters
optimset returns al l the parameters that can be set wi th typi cal val ues and
defaul t val ues.
Determining Parameters Used by a Function
The options structure defi nes the parameters that can be used by the
functi ons provi ded by the tool box. Because functi ons do not use al l the
parameters, i t may be useful to fi nd whi ch parameters are used by a parti cul ar
functi on.
To determi ne whi ch options structure fi el ds are used by a functi on, pass the
name of the functi on (i n thi s exampl e, fmincon) to optimset:
optimset('fmincon')
Default Parameter Settings
1-57
or
optimset fmincon
Thi s statement returns a structure. Fi el ds not used by the functi on have empty
val ues ([]); fi el ds used by the functi on are set to thei r defaul t val ues for the
gi ven functi on.
Displaying Output
To di spl ay output at each i terati on i nstead of just at termi nati on, enter:
options = optimset('Display', 'iter');
Thi s command sets the options.Display fi el d val ue to 'iter', whi ch causes
the tool box to di spl ay output at each i terati on.
Running Medium-Scale Optimization
For functi ons that support medi um- and l arge-scal e opti mi zati on probl ems, the
defaul t i s for the functi on to use the l arge-scal e al gori thm. To use the
medi um-scal e al gori thm, enter:
options = optimset('LargeScale', 'off');
Setting More Than One Parameter
You can speci fy mul ti pl e parameters wi th one cal l to optimset. For exampl e,
to reset the output opti on and the tol erance on x, enter:
options = optimset('Display', 'iter', 'TolX', 1e6);
Updating an options Structure
To update an exi sti ng options structure, cal l optimset and pass options as
the fi rst argument:
options = optimset(options, 'Display', 'iter', 'TolX', 1e6);
Retrieving Parameter Values
Use the optimget functi on to get parameter val ues from an options structure.
For exampl e, to get the current di spl ay opti on, enter:
verbosity = optimget(options, 'Display');
1 Tutorial
1-58
Displaying Iterative Output
Output Headings: Medium-Scale Algorithms
When the options Display parameter i s set to 'iter' for fminsearch,
fminbnd, fzero, fgoalattain, fmincon, lsqcurvefit, fminunc, fsolve,
lsqnonlin, fminimax, and fseminf, output i s produced i n col umn format.
For fminsearch the col umn headi ngs are
Iteration Func-count min f(x) Procedure
where
Iteration i s the i terati on number.
Func-count i s the number of functi on eval uati ons.
min f(x) i s the mi ni mum functi on val ue i n the current si mpl ex.
Procedure gi ves the current si mpl ex operati on: initial, expand, reflect,
shrink, contract inside and contract outside.
For fzero and fminbnd the col umn headi ngs are
Func-count x f(x) Procedure
where
Func-count i s the number of functi on eval uati ons (whi ch for fzero i s the
same as the number of i terati ons).
x i s the current poi nt.
f(x) i s the current functi on val ue at x.
Procedure gi ves the current operati on. For fzero these i ncl ude initial
(i ni ti al poi nt), search (search for a i nterval contai ni ng a zero), bisection
(bi secti on search), and interpolation. For fminbnd, the possi bl e operati ons
are initial, golden (gol den secti on search), and parabolic (parabol i c
i nterpol ati on).
Displaying Iterative Output
1-59
For fminunc, the col umn headi ngs are
Directional
Iteration Func-count f(x) Step-size derivative
where
Iteration i s the i terati on number.
Func-count i s the number of functi on eval uati ons.
f(x) i s the current functi on val ue.
Step-size i s the step si ze i n the current search di recti on.
Directional derivative i s the gradi ent of the functi on al ong the search
di recti on.
For fsolve, lsqnonlin, and lsqcurvefit the headi ngs are
Directional
Iteration Func-count Residual Step-size derivative Lambda
where Iteration, Func-count, Step-size, and Directional derivative are
the same as for fminunc, and
Residual i s the resi dual (sum-of-squares) of the functi on.
Lambda i s the val ue defi ned i n the Least Squares Opti mi zati on secti on
of the I ntroduction toAlgorithms chapter. (Thi s val ue i s pri nted when the
Levenberg-Marquardt method i s used and omi tted when the Gauss-Newton
method i s used.)
For fmincon and fseminf the headi ngs are
max Directional
Iter F-count f(x) constraint Step-size derivative Procedure
where
Iter i s the i terati on number.
F-count i s the number of functi on eval uati ons.
f(x) i s the current functi on val ue.
max constraint i s the maxi mum constrai nt vi ol ati on.
Step-size i s the step si ze i n the search di recti on.

k
1 Tutorial
1-60
Directional derivative i s the gradi ent of the functi on al ong the search
di recti on.
Procedures are messages about the Hessi an update and QP subprobl em.
The Procedures messages are di scussed i n the Updati ng the Hessi an Matri x
secti on of the I ntroduction toAlgorithms chapter.
For fgoalattain and fminimax, the headi ngs are the same as for fmincon
except f(x) and max constraint are combi ned i nto Max{F,constraints}
whi ch gi ves the maxi mum goal vi ol ati on or constrai nt vi ol ati on for
fgoalattain, and the maxi mum functi on val ue or constrai nt vi ol ati on for
fminimax.
Output Headings: Large-Scale Algorithms
For fminunc, the col umn headi ngs are
Norm of First-order
Iteration f(x) step optimality CG-iterations
where
Iteration i s the i terati on number.
f(x) i s the current functi on val ue.
Norm of step i s the norm of the current step-si ze.
First-order Optimality i s the i nfi ni ty norm of the current gradi ent.
CG-iterations i s the number of i terati ons taken by PCG (see
Precondi ti oned Conjugate Gradi ents i n Chapter 3) at the current
(opti mi zati on) i terati on.
For lsqnonlin, lsqcurvefit, and fsolve the col umn headi ngs are
Norm of First-order
Iteration Func-count f(x) step optimality CG-iterations
where
Iteration i s the i terati on number.
Func-count i s the number of functi on eval uati ons.
f(x) i s the sum-of-the-squares of the current functi on val ues.
Norm of step i s the norm of the current step-si ze.
Displaying Iterative Output
1-61
First-order optimality i s a measure of fi rst-order opti mal i ty. For bound
constrai ned probl ems, the fi rst-order opti mal i ty i s the i nfi ni ty norm of v.*g,
where v i s defi ned as i n Box Constrai nts i n Chapter 3 and g i s the gradi ent.
For unconstrai ned probl ems, i t i s the i nfi ni ty norm of the current gradi ent.
CG-iterations i s the number of i terati ons taken by PCG (see
Precondi ti oned Conjugate Gradi ents i n Chapter 3) at the current
(opti mi zati on) i terati on.
For fmincon, the col umn headi ngs are
Norm of First-order
Iteration f(x) step optimality CG-iterations
where
Iteration i s the i terati on number.
f(x) i s the current functi on val ue.
Norm of step i s the norm of the current step-si ze.
First-order optimality i s a measure of fi rst-order opti mal i ty. For bound
constrai ned probl ems, the fi rst-order opti mal i ty i s the i nfi ni ty norm of v.*g,
where v i s defi ned as i n Box Constrai nts i n Chapter 3 and g i s the gradi ent.
For equal i ty constrai ned probl ems, i t i s the i nfi ni ty norm of the projected
gradi ent. (The projected gradi ent i s the gradi ent projected i nto the nul l space
of Aeq.)
CG-iterations i s the number of i terati ons taken by PCG (see
Precondi ti oned Conjugate Gradi ents i n Chapter 3) at the current
(opti mi zati on) i terati on.
For linprog the col umn headi ngs are
Residuals: Primal Dual Upper Duality Total
Infeas Infeas Bounds Gap Rel
A*xb A'*y+zwf {x}+sub x'*z+s'*w Error
where
Primal Infeas A*x-b i s the norm of the resi dual A*x b.
Dual Infeas A'*y+zwf i s the norm of the resi dual A'*y+zwf (where w i s
al l zero i f there are no fi ni te upper bounds).
1 Tutorial
1-62
Upper Bounds x'*z+s'*w i s the norm of the resi dual
spones(s).*x+sub (whi ch i s defi ned to be zero i f al l vari abl es are
unbounded above). Thi s col umn i s not pri nted i f no fi ni te upper bounds exi st.
Duality Gap x'*z+s'*w i s the dual i ty gap (see Large-Scal e Li near
Programmi ng i n Chapter 3) between the pri mal objecti ve and the dual
objecti ve. s and w onl y appear i n thi s equati on i f there are fi ni te upper
bounds.
Total Rel Error i s the total rel ati ve error descri bed at the end of the Mai n
Al gori thm subsecti on of the secti on Large-Scal e Li near Programmi ng i n
Chapter 3.
Optimization of Inline Objects Instead of M-Files
1-63
Optimization of Inline Objects Instead of M-Files
The routi nes i n the Opti mi zati on Tool box al so perform opti mi zati on on i nl i ne
objects, avoi di ng the need to wri te M-fi l es to defi ne functi ons.
To represent a mathemati cal functi on at the command l i ne, create an inline
object from a stri ng expressi on. For exampl e, you can create an i nl i ne object of
the humps functi on (use the command type humps to see the M-fi l e functi on
humps.m):
f = inline('1./((x0.3).^2 + 0.01) + 1./((x0.9).^2 + 0.04)6');
You can then eval uate f at 2.0:
f(2.0)
ans =
4.8552
And you can pass f to an opti mi zati on routi ne to mi ni mi ze i t:
x = fminbnd(f, 3, 4)
You can al so create functi ons of more than one argument wi th inline by
speci fyi ng the names of the i nput arguments al ong wi th the stri ng expressi on.
For exampl e, to use lsqcurvefit, you need a functi on that takes two i nput
arguments, x and xdata:
f= inline('sin(x).*xdata +(x.^2).*cos(xdata)','x','xdata')
x = pi; xdata = pi*[4;2;3];
f(x, xdata)
ans =
9.8696e+000
9.8696e+000
9.8696e+000
and then cal l lsqcurvefit:
% assume ydata exists
x = lsqcurvefit(f,x,xdata,ydata)
Other exampl es usi ng thi s techni que fol l ow.
A matri x equati on
x = fsolve(inline('xxx[1,2;3,4]'),ones(2,2))
1 Tutorial
1-64
A nonl i near l east squares probl em
x = lsqnonlin(inline('xx[3 5;9 10]'),eye(2,2))
Fi nal l y, another exampl e usi ng fgoalattain where the functi on has addi ti onal
arguments to pass to the opti mi zati on routi ne. For exampl e, i f the functi on to
be mi ni mi zed has addi ti onal arguments A, B, and C,
fun = inline('sort(eig(A+B*x*C))','x','A','B','C');
x = fgoalattain(fun,ones(2,2),[5,3,1],[5, 3, 1],...
[ ],[ ],[ ],[ ],4*ones(2),4*ones(2),[ ],[ ],A,B,C);
sol ves the probl em detai l ed i n Chapter 4 for fgoalattain.
Practicalities
1-65
Practicalities
Opti mi zati on probl ems can take many i terati ons to converge and can be
sensi ti ve to numeri cal probl ems such as truncati on and round-off error i n the
cal cul ati on of fi ni te-di fference gradi ents. Most opti mi zati on probl ems benefi t
from good starti ng guesses. Thi s i mproves the executi on effi ci ency and can hel p
l ocate the gl obal mi ni mum i nstead of a l ocal mi ni mum.
Advanced probl ems are best sol ved by an evol uti onary approach whereby a
probl em wi th a smal l er number of i ndependent vari abl es i s sol ved fi rst.
Sol uti ons from l ower order probl ems can general l y be used as starti ng poi nts
for hi gher order probl ems by usi ng an appropri ate mappi ng.
The use of si mpl er cost functi ons and l ess stri ngent termi nati on cri teri a i n the
earl y stages of an opti mi zati on probl em can al so reduce computati on ti me.
Such an approach often produces superi or resul ts by avoi di ng l ocal mi ni ma.
The Opti mi zati on Tool box functi ons can be appl i ed to a l arge vari ety of
probl ems. Used wi th a l i ttl e conventi onal wi sdom, many of the l i mi tati ons
associ ated wi th opti mi zati on techni ques can be overcome. Addi ti onal l y,
probl ems that are not typi cal l y i n the standard form can be handl ed by usi ng
an appropri ate transformati on. Bel ow i s a l i st of typi cal probl ems and
recommendati ons for deal i ng wi th them:
Problem
The sol uti on does not appear to be a gl obal mi ni mum.
Recommendation
There i s no guarantee that you have a gl obal mi ni mum unl ess your probl em i s
conti nuous and has onl y one mi ni mum. Starti ng the opti mi zati on from a
number of di fferent starti ng poi nts may hel p to l ocate the gl obal mi ni mum or
veri fy that there i s onl y one mi ni mum. Use di fferent methods, where possi bl e,
to veri fy resul ts.
Problem
The fminunc functi on produces warni ng messages and seems to exhi bi t sl ow
convergence near the sol uti on.
1 Tutorial
1-66
Recommendation
I f you are not suppl yi ng anal yti cal l y determi ned gradi ents and the termi nati on
cri teri a are stri ngent, fminunc often exhi bi ts sl ow convergence near the
sol uti on due to truncati on error i n the gradi ent cal cul ati on. Rel axi ng the
termi nati on cri teri a produces faster, al though l ess accurate, sol uti ons. For the
medi um-scal e al gori thm, another opti on i s adjusti ng the fi ni te-di fference
perturbati on l evel s, DiffMinChange and DiffMaxChange, whi ch may i ncrease
the accuracy of gradi ent cal cul ati ons.
Problem
Someti mes an opti mi zati on probl em has val ues of x for whi ch i t i s i mpossi bl e
to eval uate the objecti ve functi on fun or the nonl i near constrai nts nonlcon.
Recommendation
Pl ace bounds on the i ndependent vari abl es or make a penal ty functi on to gi ve
a l arge posi ti ve val ue to f and g when i nfeasi bi l i ty i s encountered. For gradi ent
cal cul ati on the penal ty functi on shoul d be smooth and conti nuous.
Problem
The functi on that i s bei ng mi ni mi zed has di sconti nui ti es.
Recommendation
The deri vati on of the underl yi ng method i s based upon functi ons wi th
conti nuous fi rst and second deri vati ves. Some success may be achi eved for
some cl asses of di sconti nui ti es when they do not occur near sol uti on poi nts.
One opti on i s to smooth the functi on. For exampl e, the objecti ve functi on mi ght
i ncl ude a cal l to an i nterpol ati on functi on to do the smoothi ng.
Or, for the medi um-scal e al gori thms, the fi ni te-di fference parameters may be
adjusted i n order to jump over smal l di sconti nui ti es. The vari abl es
DiffMinChange and DiffMaxChange control the perturbati on l evel s for x used
i n the cal cul ati on of fi ni te-di fference gradi ents. The perturbati on,x, i s al ways
i n the range
DiffMinChange < Dx < DiffMaxChange
Problem
Warni ng messages are di spl ayed.
Practicalities
1-67
Recommendation
Thi s someti mes occurs when termi nati on cri teri a are overl y stri ngent, or when
the probl em i s parti cul arl y sensi ti ve to changes i n the i ndependent vari abl es.
Thi s usual l y i ndi cates truncati on or round-off errors i n the fi ni te-di fference
gradi ent cal cul ati on, or probl ems i n the pol ynomi al i nterpol ati on routi nes.
These warni ngs can usual l y be i gnored because the routi nes conti nue to make
steps toward the sol uti on poi nt; however, they are often an i ndi cati on that
convergence wi l l take l onger than normal . Scal i ng can someti mes i mprove the
sensi ti vi ty of a probl em.
Problem
The i ndependent vari abl es, x, onl y can take on di screte val ues, for exampl e,
i ntegers.
Recommendation
Thi s type of probl em occurs commonl y when, for exampl e, the vari abl es are the
coeffi ci ents of a fi l ter that are real i zed usi ng fi ni te-preci si on ari thmeti c or
when the i ndependent vari abl es represent materi al s that are manufactured
onl y i n standard amounts.
Al though the Opti mi zati on Tool box functi ons are not expl i ci tl y set up to sol ve
di screte probl ems, some di screte probl ems can be sol ved by fi rst sol vi ng an
equi val ent conti nuous probl em. Di screte vari abl es can be progressi vel y
el i mi nated from the i ndependent vari abl es, whi ch are free to vary.
El i mi nate a di screte vari abl e by roundi ng i t up or down to the nearest best
di screte val ue. After el i mi nati ng a di screte vari abl e, sol ve a reduced order
probl em for the remai ni ng free vari abl es. Havi ng found the sol uti on to the
reduced order probl em, el i mi nate another di screte vari abl e and repeat the
cycl e unti l al l the di screte vari abl es have been el i mi nated.
dfildemo i s a demonstrati on routi ne that shows how fi l ters wi th fi xed preci si on
coeffi ci ents can be desi gned usi ng thi s techni que.
Problem
The mi ni mi zati on routi ne appears to enter an i nfi ni te l oop or returns a sol uti on
that does not sati sfy the probl em constrai nts.
1 Tutorial
1-68
Recommendation
Your objecti ve, constrai nt or gradi ent functi ons may be returni ng I nf, NaN, or
compl ex val ues. The mi ni mi zati on routi nes expect onl y real numbers to be
returned. Any other val ues may cause unexpected resul ts. I nsert some
checki ng code i nto the user-suppl i ed functi ons to veri fy that onl y real numbers
are returned (use the functi on isfinite).
Problem
You do not get the convergence you expect from the lsqnonlin routi ne.
Recommendation
You may be formi ng the sum of squares expl i ci tl y and returni ng a scal ar val ue.
lsqnonlin expects a vector (or matri x) of functi on val ues that are squared and
summed i nternal l y.
Converting Your Code to Version 2.0 Syntax
1-69
Converting Your Code to Version 2.0 Syntax
Most of the functi on names and cal l i ng sequences have changed i n Versi on 2 to
accommodate new functi onal i ty and to cl ari fy the rol es of the i nput and output
vari abl es.
As a resul t, i f you want to use the new versi ons of these functi ons, you need to
modi fy any code that currentl y uses the ol d functi on names and cal l i ng
sequences.
Thi s tabl e l i sts the functi ons provi ded by the tool box and i ndi cates the
functi ons whose names have changed i n Versi on 2.
Old (Version 1.5) Name New (Version 2) Name
attgoal fgoalattain
conls lsqlin
constr fmincon
curvefit lsqcurvefit
fmin fminbnd
fmins fminsearch
fminu fminunc
fsolve fsolve (name unchanged)
fzero fzero (name unchanged)
leastsq lsqnonlin
minimax fminimax
nnls lsqnonneg
lp linprog
qp quadprog
seminf fseminf
1 Tutorial
1-70
Thi s secti on expl ai ns the reasons for the new cal l i ng sequences and expl ai ns
how to convert your code. I n addi ti on, i t provi des a detai l ed exampl e of
rewri ti ng a cal l to the constr functi on to cal l the new fmincon functi on i nstead.
I n addi ti on to the i nformati on i n thi s secti on, consul t the M-fi l e hel p for the
new functi ons for more i nformati on about the arguments they take. For
exampl e, to see the hel p for fmincon, type:
help fmincon
Using optimset and optimget
The optimset functi on repl aces foptions for overri di ng defaul t parameter
setti ngs. See Changi ng the Defaul t Setti ngs i n Chapter 1 for more
i nformati on on usi ng optimset and optimget.
New Calling Sequences
Versi on 2 of the tool box makes these changes i n the cal l i ng sequences:
Equal i ty constrai nts and i nequal i ty constrai nts are now suppl i ed as
separate i nput arguments.
Li near constrai nts and nonl i near constrai nts are now suppl i ed as separate
i nput arguments.
The gradi ent of the objecti ve i s computed i n the same functi on as the
objecti ve, rather than i n a separate functi on, i n order to provi de more
effi ci ent computati on (because the gradi ent and objecti ve often share si mi l ar
computati ons). Si mi l arl y, the gradi ent of the nonl i near constrai nts i s
computed by the (now separate) nonl i near constrai nt functi on.
The Hessi an matri x can be provi ded by the objecti ve functi on. (Thi s matri x
i s used onl y by the new l arge-scal e al gori thms.)
Fl ags are required to i ndi cate when extra i nformati on i s avai l abl e:
- OPTIONS.GradObj = 'on' i ndi cates the user-suppl i ed gradi ent of the
objecti ve functi on i s avai l abl e.
- OPTIONS.GradConstr = 'on' i ndi cates the user-suppl i ed gradi ent of the
constrai nts i s avai l abl e.
- OPTIONS.Hessian = 'on' i ndi cates the user-suppl i ed Hessi an of the
objecti ve functi on i s avai l abl e.
Converting Your Code to Version 2.0 Syntax
1-71
Each functi on takes an OPTIONS structure to adjust parameters to the
opti mi zati on functi ons (see optimset, optimget).
The new defaul t output gi ves i nformati on upon termi nati on (the ol d defaul t
was no output, the new defaul t i s OPTIONS.display = 'final').
Each functi on returns an EXITFLAG that denotes the termi nati on state.
The defaul t uses the new l ar ge-scal e methods when possi bl e. I f you want to
use the ol der al gori thms (referred to as medi um-scal e al gori thms i n other
parts of thi s Users Guide), set OPTIONS.LargeScale = 'off'.
Al gori thm termi nati ng condi ti ons have been fi ne tuned. The stoppi ng
condi ti ons rel ati ng to TolX and TolFun for the l arge-scal e and medi um-scal e
code are joi ned usi ng OR i nstead of AND for these functi ons: fgoalattain,
fmincon, fminimax, fminunc, fseminf, fsolve, and lsqnonlin. As a resul t, you
may need to speci fy stri cter tol erances; the defaul ts refl ect thi s change.
Each functi on now has an OUTPUT structure that contai ns i nformati on about
the probl em sol uti on rel evant to that functi on.
The LAMBDA i s now a structure where each fi el d i s the Lagrange mul ti pl i ers for
a type of constrai nt. For more i nformati on, see the i ndi vi dual functi ons entri es
i n Chapter 4.
The secti ons bel ow descri be how to convert from the ol d functi on names and
cal l i ng sequences to the new ones. The cal l s shown are the most general cases,
i nvol vi ng al l possi bl e i nput and output arguments. Note that many of these
arguments are opti onal ; see the onl i ne hel p for these functi ons for more
i nformati on.
Converting from attgoal to fgoalattain
I n Versi on 1.5, you used thi s cal l to attgoal:
OPTIONS = foptions;
[X,OPTIONS] = attgoal('FUN',x0,GOAL, WEIGHT, OPTIONS, VLB, VUB,
'GRADFUN', P1, P2,...);
wi th [F] = FUN(X,P1,...) and [DF] = GRADFUN(X,P1,...).
1 Tutorial
1-72
I n Versi on 2, you cal l fgoalattain l i ke thi s:
OPTIONS = optimset('fgoalattain');
[X,FVAL,ATTAINFACTOR,EXITFLAG,OUTPUT,LAMBDA] =
fgoalattain('FUN',x0,GOAL,WEIGHT,A,B,Aeq,Beq,VLB,VUB,
'NONLCON',OPTIONS,P1,P2,...);
wi th [F,DF] = FUN(X,P1,P2,...) and NONLCON = [].
The fgoalattain functi on now al l ows nonl i near constrai nts, so you can now
defi ne:
[Cineq,Ceq,DCineq,DCeq] = NONLCON(X,P1,...)
Converting from conls to lsqlin
I n Versi on 1.5, you used thi s cal l to conls:
[X,LAMBDA,HOW] = conls(A,b,C,d,VLB,VUB,X0,N,DISPLAY);
I n Versi on 2, convert the i nput arguments to the correct form for lsqlin by
separati ng the equal i ty and i nequal i ty constrai nts:
Ceq = C(1:N,:);
deq = d(1:N);
C = C(N+1:end,:);
d = d(N+1:end,:);
Now cal l lsqlin l i ke thi s:
OPTIONS = optimset('Display','final');
[X,RESNORM,RESIDUAL,EXITFLAG,OUTPUT,LAMBDA] =
lsqlin(A,b,C,d,Ceq,deq,VLB,VUB,X0,OPTIONS);
Converting from constr to fmincon
I n Versi on 1.5, you used thi s cal l to constr:
[X,OPTIONS,LAMBDA,HESS] =
constr('FUN',x0,OPTIONS,VLB,VUB,'GRADFUN',P1,P2,...);
wi th [F,C] = FUN(X,P1,...) and [G,DC] = GRADFUN(X,P1,...).
I n Versi on 2, repl ace FUN and GRADFUN wi th two new functi ons:
Converting Your Code to Version 2.0 Syntax
1-73
OBJFUN, whi ch returns the objecti ve functi on, the gradi ent (fi rst deri vati ve)
of thi s functi on, and i ts Hessi an matri x (second deri vati ve):
[F,G,H] = OBJFUN(X,P1,...)
NONLCON, whi ch returns the functi ons for the nonl i near constrai nts (both
i nequal i ty and equal i ty constrai nts) and thei r gradi ents:
[C,Ceq,DC,DCeq] = NONLCON(X,P1,...)
Now cal l fmincon l i ke thi s:
% OBJFUN supplies the objective gradient and Hessian;
% NONLCON supplies the constraint gradient.
OPTIONS =
optimset('GradObj','on','GradConstr','on','Hessian','on');
[X,FVAL,EXITFLAG,OUTPUT,LAMBDA,GRAD,HESSIAN] =
fmincon('OBJFUN',x0,A,B,Aeq,Beq,VLB,VUB,'NONLCON',OPTIONS,
P1,P2,...);
See Exampl e of Converti ng from constr to fmi ncon for a detai l ed exampl e of
converti ng from constr to fmincon.
Converting from curvefit to lsqcurvefit
I n Versi on 1.5, you used thi s cal l to curvefit:
[X,OPTIONS,FVAL,JACOBIAN] =
curvefit('FUN',x0,XDATA,YDATA,OPTIONS,'GRADFUN',P1,P2,...);
wi th F = FUN(X,P1,...) and G = GRADFUN(X,P1,...).
I n Versi on 2, repl ace FUN and GRADFUN wi th a si ngl e functi on that returns both
F and G (the objecti ve functi on and the gradi ent):
[F,G] = OBJFUN(X,P1,...)
Now cal l lsqcurvefit l i ke thi s:
OPTIONS = optimset('GradObj','on'); % Gradient is supplied
VLB = []; VUB = []; % New arguments not in curvefit
[X,RESNORM,F,EXITFLAG,OUTPUT,LAMBDA,JACOB] =
lsqcurvefit('OBJFUN',x0,XDATA,YDATA,VLB,VUB,OPTIONS,
P1,P2,...);
1 Tutorial
1-74
I f you have an exi sti ng FUN and GRADFUN that you do not want to rewri te, you
can pass them both to lsqcurvefit by pl aci ng them i n a cel l array:
OPTIONS = optimset('GradObj','on'); % Gradient is supplied
VLB = []; VUB = []; % New arguments not in curvefit
[X,RESNORM,F,EXITFLAG,OUTPUT,LAMBDA,JACOB] =
lsqcurvefit({'FUN','GRADFUN'},x0,XDATA,YDATA,VLB,VUB,
OPTIONS,P1,P2,...);
Converting from fmin to fminbnd
I n Versi on 1.5, you used thi s cal l to fmin:
[X,OPTIONS] = fmin('FUN',x1,x2,OPTIONS,P1,P2,...);
I n Versi on 2, you cal l fminbnd l i ke thi s:
[X,FVAL,EXITFLAG,OUTPUT] = fminbnd(FUN,x1,x2,OPTIONS,P1,P2,...);
Converting from fmins to fminsearch
I n Versi on 1.5, you used thi s cal l to fmins:
[X,OPTIONS] = fmins('FUN',x0,OPTIONS,[],P1,P2,...);
I n Versi on 2, you cal l fminsearch l i ke thi s:
[X,FVAL,EXITFLAG,OUTPUT] = fminsearch(FUN,x0,OPTIONS,P1,P2,...);
Converting from fminu to fminunc
I n Versi on 1.5, you used thi s cal l to fminu:
[X,OPTIONS] = fminu('FUN',x0,OPTIONS,'GRADFUN',P1,P2,...);
wi th F = FUN(X,P1, ...) and G = GRADFUN(X,P1, ...).
I n Versi on 2, repl ace FUN and GRADFUN wi th a si ngl e functi on that returns both
F and G (the objecti ve functi on and the gradi ent):
[F,G] = OBJFUN(X,P1, ...)
(Thi s functi on can al so return the Hessi an matri x as a thi rd output argument.)
Converting Your Code to Version 2.0 Syntax
1-75
Now cal l fminunc l i ke thi s:
OPTIONS = optimset('GradObj','on'); % Gradient is supplied
[X,FVAL,EXITFLAG,OUTPUT,GRAD,HESSIAN] =
fminunc('OBJFUN',x0,OPTIONS,P1,P2,...);
I f you have an exi sti ng FUN and GRADFUN that you do not want to rewri te, you
can pass them both to fminunc by pl aci ng them i n a cel l array:
OPTIONS = optimset('GradObj','on'); % Gradient is supplied
[X,FVAL,EXITFLAG,OUTPUT,GRAD,HESSIAN] =
fminunc({'FUN','GRADFUN'},x0,OPTIONS,P1,P2,...);
Converting to the new form of fsolve
I n Versi on 1.5, you used thi s cal l to fsolve:
[X,OPTIONS] =
fsolve('FUN',x0,XDATA,YDATA,OPTIONS,'GRADFUN',P1,P2,...);
wi th F = FUN(X,P1,...) and G = GRADFUN(X,P1,...).
I n Versi on 2, repl ace FUN and GRADFUN wi th a si ngl e functi on that returns both
F and G (the objecti ve functi on and the gradi ent):
[F,G] = OBJFUN(X,P1, ...)
Now cal l fsolve l i ke thi s:
OPTIONS = optimset('GradObj','on'); % Gradient is supplied
[X,FVAL,EXITFLAG,OUTPUT,JACOBIAN] =
fsolve('OBJFUN',x0,OPTIONS,P1,P2,...);
I f you have an exi sti ng FUN and GRADFUN that you do not want to rewri te, you
can pass them both to fsolve by pl aci ng them i n a cel l array:
OPTIONS = optimset('GradObj','on'); % Gradient is supplied
[X,FVAL,EXITFLAG,OUTPUT,JACOBIAN] =
fsolve({'FUN','GRADFUN'},x0,OPTIONS,P1,P2,...);
Converting to the new form of fzero
I n Versi on 1.5, you used thi s cal l to fzero:
X = fzero(F,X,TOL,TRACE,P1,P2,...);
1 Tutorial
1-76
I n Versi on 2, repl ace the TRACE and TOL arguments wi th:
if TRACE == 0,
val = 'none';
elseif TRACE == 1
val = 'iter';
end
OPTIONS = optimset('Display',val,'TolX',TOL);
Now cal l fzero l i ke thi s:
[X,FVAL,EXITFLAG,OUTPUT] = fzero(F,X,OPTIONS,P1,P2,...);
Converting from leastsq to lsqnonlin
I n Versi on 1.5, you used thi s cal l to leastsq:
[X,OPTIONS,FVAL,JACOBIAN] =
leastsq('FUN',x0,OPTIONS,'GRADFUN',P1,P2,...);
wi th F = FUN(X,P1,...) and G = GRADFUN(X,P1, ...).
I n Versi on 2, repl ace FUN and GRADFUN wi th a si ngl e functi on that returns both
F and G (the objecti ve functi on and the gradi ent):
[F,G] = OBJFUN(X,P1, ...)
Now cal l lsqnonlin l i ke thi s:
OPTIONS = optimset('GradObj','on'); % Gradient is supplied
VLB = []; VUB = []; % New arguments not in leastsq
[X,RESNORM,F,EXITFLAG,OUTPUT,LAMBDA,JACOBIAN] =
lsqnonlin('OBJFUN',x0,VLB,VUB,OPTIONS,P1,P2,...);
I f you have an exi sti ng FUN and GRADFUN that you do not want to rewri te, you
can pass them both to lsqnonlin by pl aci ng them i n a cel l array:
OPTIONS = optimset('GradObj','on'); % Gradient is supplied
VLB = []; VUB = []; % New arguments not in leastsq
[X,RESNORM,F,EXITFLAG,OUTPUT,LAMBDA,JACOBIAN] =
lsqnonlin({'FUN','GRADFUN'},x0,VLB,VUB,OPTIONS,P1,P2,...);
Converting Your Code to Version 2.0 Syntax
1-77
Converting from lp to linprog
I n Versi on 1.5, you used thi s cal l to lp:
[X,LAMBDA,HOW] = lp(f,A,b,VLB,VUB,X0,N,DISPLAY);
I n Versi on 2, convert the i nput arguments to the correct form for linprog by
separati ng the equal i ty and i nequal i ty constrai nts:
Aeq = A(1:N,:);
beq = b(1:N);
A = A(N+1:end,:);
b = b(N+1:end,:);
if DISPLAY
val = 'final';
else
val = 'none';
end
OPTIONS = optimset('Display',val);
Now cal l linprog l i ke thi s:
[X,FVAL,EXITFLAG,OUTPUT,LAMBDA] =
linprog(f,A,b,Aeq,beq,VLB,VUB,X0,OPTIONS);
Converting from minimax to fminimax
I n Versi on 1.5, you used thi s cal l to minimax:
[X,OPTIONS] =
minimax('FUN',x0,OPTIONS,VLB,VUB,'GRADFUN',P1,P2,...);
wi th F = FUN(X,P1,...) and G = GRADFUN(X,P1,...).
I n Versi on 2, you cal l fminimax l i ke thi s:
OPTIONS = optimset('fminimax');
[X,FVAL,MAXFVAL,EXITFLAG,OUTPUT,LAMBDA] =
fminimax('OBJFUN',x0,A,B,Aeq,Beq,VLB,VUB,'NONLCON',OPTIONS,
P1,P2,...);
wi th [F,DF] = OBJFUN(X,P1,...)
and [Cineq,Ceq,DCineq,DCeq] = NONLCON(X,P1,...).
1 Tutorial
1-78
Converting from nnls to lsqnonneg
I n Versi on 1.5, you used thi s cal l to nnls:
[X,LAMBDA] = nnls(A,b,tol);
I n Versi on 2, repl ace the tol argument wi th:
OPTIONS = optimset('Display','none','TolX',tol);
Now cal l lsqnonneg l i ke thi s:
[X,RESNORM,RESIDUAL,EXITFLAG,OUTPUT,LAMBDA] =
lsqnonneg(A,b,X0,OPTIONS);
Converting from qp to quadprog
I n Versi on 1.5, you used thi s cal l to qp:
[X,LAMBDA,HOW] = qp(H,f,A,b,VLB,VUB,X0,N,DISPLAY);
I n Versi on 2, convert the i nput arguments to the correct form for quadprog by
separati ng the equal i ty and i nequal i ty constrai nts:
Aeq = A(1:N,:);
beq = b(1:N);
A = A(N+1:end,:);
b = b(N+1:end,:);
if DISPLAY
val = 'final';
else
val = 'none';
end
OPTIONS = optimset('Display',val);
Now cal l quadprog l i ke thi s:
[X,FVAL,EXITFLAG,OUTPUT,LAMBDA] =
quadprog(H,f,A,b,Aeq,beq,VLB,VUB,X0,OPTIONS);
Converting from seminf to fseminf
I n Versi on 1.5, you used thi s cal l to seminf:
[X,OPTIONS] = seminf('FUN',N,x0,OPTIONS,VLB,VUB,P1,P2,...);
wi th [F,C,PHI1,PHI2,...,PHIN,S] = FUN(X,S,P1,P2,...).
Converting Your Code to Version 2.0 Syntax
1-79
I n Versi on 2, cal l fseminf l i ke thi s:
[X,FVAL,EXITFLAG,OUTPUT,LAMBDA] =
fseminf('OBJFUN',x0,N,'NONLCON',A,B,Aeq,Beq,VLB,VUB,OPTIONS,
P1,P2,...);
wi th F = OBJFUN(X,P1,...)
and [Cineq,Ceq,PHI1,PHI2,...,PHIN,S] = NONLCON(X,S,P1,...).
1 Tutorial
1-80
Example of Converting from constr to fmincon
Old Call to constr
OPTIONS = foptions;
OPTIONS(13) = 2; % two equality constraints
OPTIONS(1) = 1;
OPTIONS(9) = 1;
A1 = [ 1 4 3]; b1 = 2;
A2 = [ 2 5 0]; b2 = 9;
x0 = [1; .5; .8];
LB = []; UB = [];
[X,OPTIONS,LAMBDA,HESS] = ...
constr('myfuncon',x0,OPTIONS,LB,UB,'mygradcon',A1,b1,A2,b2);
% myfuncon.m
[F, C] = myfuncon(x,A1,b1,A2,b2)
F = x(1) + 0.0009*x(2)^3 + sin(x(3));
C(1,1) = A1*xb; % equality linear constraint
C(2,1) = 3*x(1)^21; % equality nonlinear constraint
C(3,1) = A2*xb2; % inequality linear constraint
C(4,1) = 7*sin(x(2))1; % inequality nonlinear constraint
% mygradcon.m
[G, DC] = mygradcon(x,alpha)
G = [1; % gradient of the objective
3*0.0009*x(2)^2;
cos(x(3))];
DC(:,1) = A1'; % gradient of the constraints
DC(:,2) = [6*x(1); 0; 0];
DC(:,3) = A2';
DC(:,4) = [0; 7*cos(x(2)); 0];
Converting Your Code to Version 2.0 Syntax
1-81
New Call to fmincon
OPTIONS = optimset(...
'Display', 'iter', ...
'GradCheck', 'on', ... % Check gradients.
'GradObj', 'on', ... % Gradient of objective is provided.
'GradConstr', 'on'); % Gradient of constraints is provided.
A1 = [ 1 4 3]; b1 = 2; % linear equalities
A2 = [ 2 5 0]; b2 = 9; % linear inequalities
x0 = [1; .5; .8];
LB = []; UB = [];
[X,FVAL,EXITFLAG,OUTPUT,LAMBDA,GRAD,HESSIAN] = ...
fmincon('myfun',x0,A2,b2,A1,b1,LB,UB,'mycon',OPTIONS);
% myfun.m
function [F,G] = myfun(x)
F = x(1) + 0.0009*x(2)^3 + sin(x(3));
G = [1;
3*0.0009*x(2)^2;
cos(x(3))];
% mycon.m
function [C,Ceq,DC,DCeq]= mycon(x)
Ceq(1,1) = 3*x(1)^21; % equality nonlinear constraint
C(1,1) = 7*sin(x(2))1; % inequality nonlinear constraint
DCeq(:,1) = [6*x(1); 0; 0]; % gradient of equality
% nonlinear constraint
DC(:,1) = [0; 7*cos(x(2)); 0]; % gradient of inequality
% nonlinear constraint
1 Tutorial
1-82
2
I ntroducti on to Al gori thms
Parametric Optimization . . . . . . . . . . . . . . 2-2
Unconstrained Optimization . . . . . . . . . . . . 2-4
Quasi -Newton Methods . . . . . . . . . . . . . . . . 2-5
Li ne Search . . . . . . . . . . . . . . . . . . . . . 2-8
Quasi-Newton Implementation . . . . . . . . . . . 2-11
Hessi an Update . . . . . . . . . . . . . . . . . . . 2-11
Li ne Search Procedures . . . . . . . . . . . . . . . . 2-11
Least Squares Optimization . . . . . . . . . . . . . 2-17
Gauss-Newton Method . . . . . . . . . . . . . . . . 2-18
Levenberg-Marquardt Method . . . . . . . . . . . . . 2-19
Nonlinear Least Squares Implementation . . . . . . 2-21
Gauss-Newton I mpl ementati on . . . . . . . . . . . . . 2-21
Levenberg-Marquardt I mpl ementati on . . . . . . . . . . 2-21
Constrained Optimization . . . . . . . . . . . . . . 2-23
Sequenti al Quadrati c Programmi ng (SQP) . . . . . . . . 2-24
QP Subprobl em . . . . . . . . . . . . . . . . . . . 2-24
SQP Implementation . . . . . . . . . . . . . . . . 2-27
Updati ng the Hessi an Matri x . . . . . . . . . . . . . . 2-27
Quadrati c Programmi ng Sol uti on . . . . . . . . . . . . 2-28
Li ne Search and Meri t Functi on . . . . . . . . . . . . 2-31
MultiobjectiveOptimization . . . . . . . . . . . . . 2-33
I ntroducti on . . . . . . . . . . . . . . . . . . . . . 2-33
Goal Attai nment Method . . . . . . . . . . . . . . . 2-39
Al gori thm I mprovements for Goal Attai nment Method . . . 2-40
Review . . . . . . . . . . . . . . . . . . . . . . 2-43
References . . . . . . . . . . . . . . . . . . . . . 2-44
2 Introduction to Algorithms
2-2
Parametric Optimization
Thi s chapter provi des an i ntroducti on to the di fferent opti mi zati on probl em
formul ati ons and descri bes the medi um-scal e al gori thms used i n the
Opti mi zati on Tool box. Medi um-scal e i s not a standard term and i s used here
onl y to di fferenti ate these al gori thms from the l arge-scal e al gori thms descri bed
i n the Large-scaleAlgorithms chapter.
Parametri c opti mi zati on i s used to fi nd a set of desi gn parameters,
, that can i n some way be defi ned as opti mal . I n a si mpl e
case thi s may be the mi ni mi zati on or maxi mi zati on of some system
characteri sti c that i s dependent on x. I n a more advanced formul ati on the
objecti ve functi on, f(x), to be mi ni mi zed or maxi mi zed, may be subject to
constrai nts i n the form of equal i ty constrai nts, ,
i nequal i ty constrai nts, , and/or parameter bounds,
.
A General Probl em (GP) descri pti on i s stated as
(2-1)
where x i s the vector of desi gn parameters, ( ), f(x) i s the objecti ve
functi on that returns a scal ar val ue ( ), and the vector functi on
G(x) returns the val ues of the equal i ty and i nequal i ty constrai nts eval uated at
x( ).
An effi ci ent and accurate sol uti on to thi s probl em i s not onl y dependent on the
si ze of the probl em i n terms of the number of constrai nts and desi gn vari abl es
but al so on characteri sti cs of the objecti ve functi on and constrai nts. When both
the objecti ve functi on and the constrai nts are l i near functi ons of the desi gn
vari abl e, the probl em i s known as a Li near Programmi ng probl em (LP).
Quadrati c Programmi ng (QP) concerns the mi ni mi zati on or maxi mi zati on of a
quadrati c objecti ve functi on that i s l i nearl y constrai ned. For both the LP and
QP probl ems, rel i abl e sol uti on procedures are readi l y avai l abl e. More di ffi cul t
to sol ve i s the Nonl i near Programmi ng (NP) probl em i n whi ch the objecti ve
functi on and constrai nts may be nonl i near functi ons of the desi gn vari abl es. A
x x
1
x
2
x
n
, , , { } =
G
i
x ( ) 0 = i 1 m
e
, , = ( )
G
i
x ( ) 0 i m
e
1 + m , , = ( )
x
l
x
u
,
f x ( )
x
n

minimize
subject to: G
i
x 0, = i 1 m
e
, , =
G
i
x ( ) 0 , i m
e
1 + m , , =
x
l
x x
u

x
n

f x ( ):
n

G x ( ):
n

m

Parametric Optimization
2-3
sol uti on of the NP probl em general l y requi res an i terati ve procedure to
establ i sh a di recti on of search at each major i terati on. Thi s i s usual l y achi eved
by the sol uti on of an LP, a QP, or an unconstrai ned sub-probl em.
2 Introduction to Algorithms
2-4
Unconstrained Optimization
Al though a wi de spectrum of methods exi sts for unconstrai ned opti mi zati on,
methods can be broadl y categori zed i n terms of the deri vati ve i nformati on that
i s, or i s not, used. Search methods that use onl y functi on eval uati ons (e.g., the
si mpl ex search of Nel der and Mead [33]) are most sui tabl e for probl ems that
are very nonl i near or have a number of di sconti nui ti es. Gradi ent methods are
general l y more effi ci ent when the functi on to be mi ni mi zed i s conti nuous i n i ts
fi rst deri vati ve. Hi gher order methods, such as Newtons method, are onl y
real l y sui tabl e when the second order i nformati on i s readi l y and easi l y
cal cul ated si nce cal cul ati on of second order i nformati on, usi ng numeri cal
di fferenti ati on, i s computati onal l y expensi ve.
Gradi ent methods use i nformati on about the sl ope of the functi on to di ctate a
di recti on of search where the mi ni mum i s thought to l i e. The si mpl est of these
i s the method of steepest descent i n whi ch a search i s performed i n a
di recti on, , (where i s the gradi ent of the objecti ve functi on). Thi s
method i s very i neffi ci ent when the functi on to be mi ni mi zed has l ong narrow
val l eys as, for exampl e, i s the case for Rosenbrocks functi on
(2-2)
The mi ni mum of thi s functi on i s at where . A contour map
of thi s functi on i s shown i n Fi gure , al ong wi th the sol uti on path to the
mi ni mum for a steepest descent i mpl ementati on starti ng at the poi nt [1.9,2].
The opti mi zati on was termi nated after 1000 i terati ons, sti l l a consi derabl e
di stance from the mi ni mum. The bl ack areas are where the method i s
conti nual l y zi g-zaggi ng from one si de of the val l ey to another. Note that
towards the center of the pl ot, a number of l arger steps are taken when a poi nt
l ands exactl y at the center of the val l ey.
f x ( ) f x ( )
f x ( ) 100 x
1
x
2
2
( )
2
1 x
1
( )
2
+ =
x 1 1 [ , ] = f x ( ) 0 =
Unconstrained Optimization
2-5
Figure 2-1: Steepest Descent Method on Rosenbrocks Function (Eq. 2-2)
Thi s type of functi on (Eq. 2-2), al so known as the banana functi on, i s notori ous
i n unconstrai ned exampl es because of the way the curvature bends around the
ori gi n. Eq. 2-2 i s used throughout thi s secti on to i l l ustrate the use of a vari ety
of opti mi zati on techni ques. The contours have been pl otted i n exponenti al
i ncrements due to the steepness of the sl ope surroundi ng the U-shaped val l ey.
Quasi-Newton Methods
Of the methods that use gradi ent i nformati on, the most favored are the
quasi -Newton methods. These methods bui l d up curvature i nformati on at each
i terati on to formul ate a quadrati c model probl em of the form
(2-3)
x
1
x
2
-1
-0.5
0
0.5
1
1.5
2
2.5
3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
o
Start Point
o
Solution
o
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
oo
oo
o
oo
oo
oo
oo
oo
oo
o oo oo oo oo oo oo oo oo oo oo oo oo oo oo oo oo oo oo oo oo oo oo o
1
2
---x
T
Hx c
T
x b + +
x
min
2 Introduction to Algorithms
2-6
where the Hessi an matri x, H, i s a posi ti ve defi ni te symmetri c matri x, c i s a
constant vector, and b i s a constant. The opti mal sol uti on for thi s probl em
occurs when the parti al deri vati ves of xgo to zero, i .e.,
(2-4)
The opti mal sol uti on poi nt, , can be wri tten as
(2-5)
Newton-type methods (as opposed to quasi -Newton methods) cal cul ate H
di rectl y and proceed i n a di recti on of descent usi ng a l i ne search method to
l ocate the mi ni mum after a number of i terati ons. Cal cul ati ng H numeri cal l y
i nvol ves a l arge amount of computati on. Quasi -Newton methods avoi d thi s by
usi ng the observed behavi or of f(x) and to bui l d up curvature
i nformati on to make an approxi mati on to H usi ng an appropri ate updati ng
techni que.
A l arge number of Hessi an updati ng methods have been devel oped. General l y,
the formul a of Broyden [3], Fl etcher [4], Gol dfarb [5], and Shanno [6] (BFGS)
i s thought to be the most effecti ve for use i n a general purpose method.
The formul a i s gi ven by
BFGS
(2-6)
where
As a starti ng poi nt, can be set to any symmetri c posi ti ve defi ni te matri x, for
exampl e, the i denti ty matri x I . To avoi d the i nversi on of the Hessi an H, you can
deri ve an updati ng method i n whi ch the di rect i nversi on of H i s avoi ded by
usi ng a formul a that makes an approxi mati on of the i nverse Hessi an at
each update. A wel l known procedure i s the DFP formul a of Davi don [7],
Fl etcher, and Powel l [8]. Thi s uses the same formul a as the above BFGS
method (Eq. ) except that i s substi tuted for .
f x

( ) Hx

c + 0 = =
x

H
1
c =
f x ( )
H
k 1 +
H
k
q
k
q
k
T
q
k
T
s
k
-------------
H
k
T
s
k
T
s
k
H
k
s
k
T
H
k
s
k
----------------------------- + =
s
k
x
k 1 +
x
k
=
q
k
f x
k 1 +
( ) f x
k
( ) =
H
0
H
1
q
k
s
k
Unconstrained Optimization
2-7
The gradi ent i nformati on i s ei ther suppl i ed through anal yti cal l y cal cul ated
gradi ents, or deri ved by parti al deri vati ves usi ng a numeri cal di fferenti ati on
method vi a fi ni te di fferences. Thi s i nvol ves perturbi ng each of the desi gn
vari abl es, x, i n turn and cal cul ati ng the rate of change i n the objecti ve functi on.
At each major i terati on, k, a l i ne search i s performed i n the di recti on
(2-7)
The quasi -Newton method i s i l l ustrated by the sol uti on path on Rosenbrocks
functi on (Eq. 2-2) i n Fi gure 2-2. The method i s abl e to fol l ow the shape of the
val l ey and converges to the mi ni mum after 140 functi on eval uati ons usi ng onl y
fi ni te di fference gradi ents.
Figure 2-2: BFGS Method on Rosenbrocks Function
d H
k
1
f x
k
( ) =
-1
-0.5
0
0.5
1
1.5
2
2.5
3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
o
Start Point
o
Solution
o
o
o
o
o
o
o
o
o
o
o
o o
o
o o
o
o
oo
o
oo
ooo
2 Introduction to Algorithms
2-8
Line Search
Most unconstrai ned and constrai ned methods use the sol uti on of a sub-probl em
to yi el d a search di recti on i n whi ch the sol uti on i s esti mated to l i e. The
mi ni mum al ong the l i ne formed from thi s search di recti on i s general l y
approxi mated usi ng a search procedure (e.g., Fi bonacci , Gol den Secti on) or by
a pol ynomi al method i nvol vi ng i nterpol ati on or extrapol ati on (e.g., quadrati c,
cubi c). Pol ynomi al methods approxi mate a number of poi nts wi th a uni vari ate
pol ynomi al whose mi ni mum can be cal cul ated easi l y. I nterpol ati on refers to
the condi ti on that the mi ni mum i s bracketed (i .e., the mi ni mum l i es i n the area
spanned by the avai l abl e poi nts), whereas extrapol ati on refers to a mi ni mum
l ocated outsi de the range spanned by the avai l abl e poi nts. Extrapol ati on
methods are general l y consi dered unrel i abl e for esti mati ng mi ni ma for
nonl i near functi ons. However, they are useful for esti mati ng step l ength when
tryi ng to bracket the mi ni mum as shown i n the Li ne Search Procedures
secti on. Pol ynomi al i nterpol ati on methods are general l y the most effecti ve i n
terms of effi ci ency when the functi on to be mi ni mi zed i s conti nuous. The
probl em i s to fi nd a new i terate of the form
(2-8)
where denotes the current i terate, d the search di recti on obtai ned by an
appropri ate method, and i s a scal ar step l ength parameter that i s the
di stance to the mi ni mum.
Quadratic Interpolation
Quadrati c i nterpol ati on i nvol ves a data fi t to a uni vari ate functi on of the form
(2-9)
where an extremum occurs at a step l ength of
(2-10)
Thi s poi nt may be a mi ni mum or a maxi mum. I t i s a mi ni mum when
i nterpol ati on i s performed (i .e., usi ng a bracketed mi ni mum) or when a i s
posi ti ve. Determi nati on of coeffi ci ents, a and b, can be found usi ng any
combi nati on of three gradi ent or functi on eval uati ons. I t may al so be carri ed
out wi th just two gradi ent eval uati ons. The coeffi ci ents are determi ned
through the formul ati on and sol uti on of a l i near set of si mul taneous equati ons.
x
k 1 +
x
k 1 +
x
k

d + =
x
k

m
q
( ) a
2
b c + + =

b
2a
------- =
Unconstrained Optimization
2-9
Vari ous si mpl i fi cati ons i n the sol uti on of these equati ons can be achi eved when
parti cul ar characteri sti cs of the poi nts are used. For exampl e, the fi rst poi nt
can general l y be taken as . Other si mpl i fi cati ons can be achi eved when
the poi nts are evenl y spaced. A general probl em formul a i s as fol l ows:
Gi ven three unevenl y spaced poi nts and thei r associ ated functi on
val ues the mi ni mum resul ti ng from a second-order fi t i s
gi ven by
Quadratic Interpolation
(2-11)
where
For i nterpol ati on to be performed, as opposed to extrapol ati on, the mi ni mum
must be bracketed so that the poi nts can be arranged to gi ve
Cubic Interpolation
Cubi c i nterpol ati on i s useful when gradi ent i nformati on i s readi l y avai l abl e or
when more than three functi on eval uati ons have been cal cul ated. I t i nvol ves a
data fi t to the uni vari ate functi on
(2-12)
where the l ocal extrema are roots of the quadrati c equati on
To fi nd the mi ni mum extremum, take the root that gi ves as posi ti ve.
Coeffi ci ents a and bcan be determi ned usi ng any combi nati on of four gradi ent
or functi on eval uati ons, or al ternati vel y, wi th just three gradi ent eval uati ons.
The coeffi ci ents are cal cul ated by the formul ati on and sol uti on of a l i near set of
si mul taneous equati ons. A general formul a, gi ven two poi nts, , thei r
0 =
x
1
x
2
x
3
, { , }
f x
1
( ) f x
2
( ) f x
3
( ) , { , }
x
k 1 +
1
2
---

23
f x
1
( )
31
f x
2
( )
12
f x
3
( ) + +

23
f x
1
( )
31
f x
2
( )
12
f x
3
( ) + +
------------------------------------------------------------------------------- =

i j
x
i
2
x
j
2
=

i j
x
i
x
j
=
f x
2
( ) f x
1
( ) < and f x
2
( ) f x
3
( ) <
m
c
( ) a
3
b
2
c d + + + =
3a
2
2b c + + 0 =
6a 2b +
x
1
x
2
{ , }
2 Introduction to Algorithms
2-10
correspondi ng gradi ents wi th respect to x, , and associ ated
functi on val ues, i s
(2-13)
where
f x
1
( ) f x
2
( ) { , }
f x
1
( ) f x
2
( ) { , }
x
k 1 +
x
2
x
2
x
1
( )
f x
2
( )
2

1
+
f x
2
( ) f x
1
( ) 2
2
+
-------------------------------------------------------- - =

1
f x
1
( ) f x
2
( ) 3
f x
1
( ) f x
2
( )
x
1
x
2

-------------------------------- + =

2

1
2
fx
1
f x
2
( ) ( )
1 2 /
. =
Quasi-Newton Implementation
2-11
Quasi-Newton Implementation
A quasi -Newton al gori thm i s used i n fminunc. The al gori thm consi sts of two
phases:
Determi nati on of a di recti on of search
Li ne search procedure
I mpl ementati on detai l s of the two phases are di scussed bel ow.
Hessian Update
The di recti on of search i s determi ned by a choi ce of ei ther the BFGS (Eq. ) or
the DFP method gi ven i n the Quasi -Newton Methods secti on (set the options
parameter HessUpdate to 'dfp' to sel ect the DFP method). The Hessi an, H, i s
al ways mai ntai ned to be posi ti ve defi ni te so that the di recti on of search, d, i s
al ways i n a descent di recti on. Thi s means that for some arbi trari l y smal l step,
, i n the di recti on, d, the objecti ve functi on decreases i n magni tude. Posi ti ve
defi ni teness of H i s achi eved by ensuri ng that H i s i ni ti al i zed to be posi ti ve
defi ni te and thereafter (from Eq. ) i s al ways posi ti ve. The term i s a
product of the l i ne search step l ength parameter, and a combi nati on of the
search di recti on, d, wi th past and present gradi ent eval uati ons,
(2-14)
The condi ti on that i s posi ti ve i s al ways achi eved by ensuri ng that a
suffi ci entl y accurate l i ne search i s performed. Thi s i s because the search
di recti on, d, i s a descent di recti on so that and are al ways
posi ti ve. Thus, the possi bl e negati ve term can be made as smal l
i n magni tude as requi red by i ncreasi ng the accuracy of the l i ne search.
Line Search Procedures
Two l i ne search strategi es are used dependi ng on whether gradi ent
i nformati on i s readi l y avai l abl e or whether i t must be cal cul ated usi ng a fi ni te
di fference method. When gradi ent i nformati on i s avai l abl e, the defaul t i s to use
a cubi c pol ynomi al method. When gradi ent i nformati on i s not avai l abl e, the
defaul t i s to use a mi xed quadrati c and cubi c pol ynomi al method.

q
k
T
s
k
q
k
T
s
k

k
q
k
T
s
k

k
f x
k 1 +
( )
T
d f x
k
( )
T
d ( ) =
q
k
T
s
k

k
f x
k
( )
T
d
f x
k 1 +
( )
T
d
2 Introduction to Algorithms
2-12
Cubic Polynomial Method
I n the proposed cubi c pol ynomi al method, a gradi ent and a functi on eval uati on
i s made at every i terati on, k. At each i terati on an update i s performed when a
new poi nt i s found, , whi ch sati sfi es the condi ti on that
(2-15)
At each i terati on a step, , i s attempted to form a new i terate of the form
(2-16)
I f thi s step does not sati sfy the condi ti on (Eq. 2-15) then i s reduced to form
a new step, . The usual method for thi s reducti on i s to use bi secti on (i .e.,
to conti nual l y hal ve the step l ength unti l a reducti on i s achi eved i n f(x).
However, thi s procedure i s sl ow when compared to an approach that i nvol ves
usi ng gradi ent and functi on eval uati ons together wi th cubi c i nterpol ati on/
extrapol ati on methods to i denti fy esti mates of step l ength.
When a poi nt i s found that sati sfi es the condi ti on (Eq. 2-15), an update i s
performed i f i s posi ti ve. I f i t i s not, then further cubi c i nterpol ati ons are
performed unti l the uni vari ate gradi ent term i s suffi ci entl y smal l
so that i s posi ti ve.
I t i s usual practi ce to reset to uni ty after every i terati on. However, note that
the quadrati c model (Eq. 2-3) i s general l y onl y a good one near to the sol uti on
poi nt. Therefore, , i s modi fi ed at each major i terati on to compensate for the
case when the approxi mati on to the Hessi an i s monotoni cal l y i ncreasi ng or
decreasi ng. To ensure that, as x
k
approaches the sol uti on poi nt, the procedure
reverts to a val ue of cl ose to uni ty, the val ues of and
are used to esti mate the cl oseness to the sol uti on poi nt and thus to control the
vari ati on i n .
After each update procedure, a step l ength i s attempted, fol l owi ng whi ch a
number of scenari os are possi bl e. Consi derati on of al l the possi bl e cases i s qui te
compl i cated and so they are represented pi ctori al l y i n Fi gure 2-3, where the
l eft-hand poi nt on the graphs represents the poi nt . The sl ope of the l i ne
bi secti ng each poi nt represents the sl ope of the uni vari ate gradi ent, ,
whi ch i s al ways negati ve for the l eft-hand poi nt. The ri ght-hand poi nt i s the
poi nt after a step of i s taken i n the di recti on d.
x
k 1 +
f x
k 1 +
( ) f x
k
( ) <

k
x
k 1 +
x
k

k
d + =

k 1 +
q
k
T
s
k
f x
k 1 +
( )
T
d
q
k
T
s
k

k
q
k
T
s
k
f x
k
( )
T
d
k 1 +

k
x
k
f x
k
( )
T
d
x
k 1 +

k
Quasi-Newton Implementation
2-13
Figure 2-3: Cubic Polynomial Line Search Procedures
Reduce step l ength.
Reduce step l ength.
Update H.
Reset d.

k 1 +

c
2 i f
k
0.1 <

c
otherwi se

'

=
0
0 <
q
k
T
s
k

k 1 +
1
c
, { } min =
k 1 +
0.9
c
=
Update H.
Reset d.
Change to steepest
descent method
temporari l y.
Reduce step l ength.
0
0 <
q
k
T
s
k

k 1 +
2 p 1.2
c
, , { } min =
where p 1 q
k
T
s
k
fx
k 1 +

T
d 0
k 1 +
, { } min + + =

k 1 +

c

k
2 , { } min =

k 1 +
2 1.5
k
, { } max
c
, , { } min =
Case 1:
Case 2:
Case 3:
Case 4:
f x
k 1 +
( ) f x
k
( ) > f x
k 1 +
( )
T
d 0 > ,

f x ( )
0

k 1 +

k
f x
k 1 +
( ) f x
k
( ) f x
k 1 +
( )
T
d 0 ,

f x ( )

k 1 +

k
0
f x
k 1 +
( ) f x
k
( ) < f x
k 1 +
( )
T
d 0 < ,

f x ( )

k 1 +

k
0
f x
k 1 +
( ) f x
k
( ) f x
k 1 +
( )
T
d 0 ,

f x ( )

k 1 +

k
0
2 Introduction to Algorithms
2-14
Cases 1 and 2 show the procedures performed when the val ue
i s posi ti ve. Cases 3 and 4 show the procedures performed when
the val ue i s negati ve. The notati on refers to the
smal l est val ue of the set .
At each i terati on a cubi cl y i nterpol ated step l ength i s cal cul ated and then
used to adjust the step l ength parameter . Occasi onal l y, for very
nonl i near functi ons may be negati ve, i n whi ch case i s gi ven a val ue of
. The methods for changi ng the step l ength have been refi ned over a peri od
of ti me by consi deri ng a l arge number of test probl ems.
Certai n robustness measures have al so been i ncl uded so that, even i n the case
when fal se gradi ent i nformati on i s suppl i ed, a reducti on i n f(x) can be achi eved
by taki ng a negati ve step. Thi s i s done by setti ng when
fal l s bel ow a certai n threshol d val ue (e.g., 1e8). Thi s i s i mportant when
extremel y hi gh preci si on i s requi red i f onl y fi ni te di fference gradi ents are
avai l abl e.
Mixed Cubic/Quadratic Polynomial Method
The cubi c i nterpol ati on/extrapol ati on method has proved successful for a l arge
number of opti mi zati on probl ems. However, when anal yti c deri vati ves are not
avai l abl e, the eval uati ng fi ni te di fference gradi ents i s computati onal l y
expensi ve. Therefore, another i nterpol ati on/extrapol ati on method i s
i mpl emented so that gradi ents are not needed at every i terati on. The approach
i n these ci rcumstances, when gradi ents are not readi l y avai l abl e, i s to use a
quadrati c i nterpol ati on method. The mi ni mum i s general l y bracketed usi ng
some form of bi secti on method. Thi s method, however, has the di sadvantage
that al l the avai l abl e i nformati on about the functi on i s not used. For i nstance,
a gradi ent cal cul ati on i s al ways performed at each major i terati on for the
Hessi an update. Therefore, gi ven three poi nts that bracket the mi ni mum, i t i s
possi bl e to use cubi c i nterpol ati on, whi ch i s l i kel y to be more accurate than
usi ng quadrati c i nterpol ati on. Further effi ci enci es are possi bl e i f, i nstead of
usi ng bi secti on to bracket the mi ni mum, extrapol ati on methods si mi l ar to
those used i n the cubi c pol ynomi al method are used.
Hence, the method that i s used i n fminunc, lsqnonlin, lsqcurvefit, and
fsolve i s to fi nd three poi nts that bracket the mi ni mum and to use cubi c
i nterpol ati on to esti mate the mi ni mum at each l i ne search. The esti mati on of
step l ength, at each mi nor i terati on, j, i s shown i n Fi gure for a number of poi nt
combi nati ons. The l eft-hand poi nt i n each graph represents the functi on val ue
and uni vari ate gradi ent obtai ned at the l ast update. The
f x
k 1 +
( )
T
d
f x
k 1 +
( )
T
d a b c , { , } min
a b c , { , }

k 1 +

c

c
2
k

k 1 +

k
2 =
k
f x
1
( ) f x
k
( )
Quasi-Newton Implementation
2-15
ri ght-hand poi nts represent the poi nts accumul ated i n the mi nor i terati ons of
the l i ne search procedure.
The terms and refer to the mi ni mum obtai ned from a respecti ve
quadrati c and cubi c i nterpol ati on or extrapol ati on. For hi ghl y nonl i near
functi ons, and may be negati ve, i n whi ch case they are set to a val ue of
so that they are al ways mai ntai ned to be posi ti ve. Cases 1 and 2 use
quadrati c i nterpol ati on wi th two poi nts and one gradi ent to esti mate a thi rd
poi nt that brackets the mi ni mum. I f thi s fai l s
, cases 3 and 4
represent the
possi bi l i ti es for changi ng the step l ength when at l east three poi nts are
avai l abl e.
When the mi ni mum i s fi nal l y bracketed, cubi c i nterpol ati on i s achi eved usi ng
one gradi ent and three functi on eval uati ons. I f the i nterpol ated poi nt i s greater
than any of the thr ee used for the i nter pol ati on, then i t i s repl aced wi th the
poi nt wi th the smal l est functi on val ue. Fol l owi ng the l i ne search procedure the
Hessi an update procedure i s performed as for the cubi c pol ynomi al l i ne search
method.
.

q

c

c

q
2
k
2 Introduction to Algorithms
2-16
Figure 2-4: Line Search Procedures with Only Gradient for the First Point
Case 1:
Case 2:

f x ( )

0

j

j 1 +

0
j

j 1 +

f x ( )
Reduce step l ength.
I ncrease step l ength.
f x
j
( ) f x
k
( )
f x
j
( ) f x
k
( ) <

j 1 +

q
=

j 1 +
1.2
q
=
Case 3:
Case 4:

0

j

j 1 +

j 2 +
f x ( )

0

j 2 +

j

j 1 +
f x ( )
I ncrease step l ength.
Reduce step l ength.
f x
j 1 +
( ) f x
k
( ) <
f x
j 1 +
( ) f x
k
( ) >

j 2 +
1.2
q
2
j 1 +
, { } max =

j 2 +

c
=
Least Squares Optimization
2-17
Least Squares Optimization
The l i ne search procedures used i n conjuncti on wi th a quasi -Newton method
are used i n the functi on fminunc. They are al so used as part of a nonl i near l east
squares (LS) opti mi zati on routi nes, lsqnonlin and lsqcurvefit. I n the l east
squares probl em a functi on, f(x) i s mi ni mi zed that i s a sum of squares.
LS
(2-17)
Probl ems of thi s type occur i n a l arge number of practi cal appl i cati ons
especi al l y when fi tti ng model functi ons to data, i .e., nonl i near parameter
esti mati on. They are al so preval ent i n control where you want the output,
to fol l ow some conti nuous model trajectory, , for vector xand scal ar
t. Thi s probl em can be expressed as
(2-18)
where and are scal ar functi ons.
When the i ntegral i s di screti zed usi ng a sui tabl e quadrature formul a, Eq. 2-18
can be formul ated as a l east squares probl em
(2-19)
where and i ncl ude the wei ghts of the quadrature scheme. Note that i n thi s
probl em the vector F(x) i s
f x ( ) =
1
2
--- F x ( )
2
2
x
n

min
1
2
--- F
i
x ( )
2
i

=
y x t , ( ) t ( )
mi n
x
n

y x t , ( ) t ( ) ( )
t
2
t
1

dt
2
y x t , ( ) t ( )
f x ( )
x
n

min y x t
i
, ( ) t
i
( ) ( )
2
i 1 =
m

=
y
F x ( )
y x t
1
, ( ) t
1
( )
y x t
2
, ( ) t
2
( )

y x t
m
, ( ) t
m
( )
=
2 Introduction to Algorithms
2-18
I n probl ems of thi s ki nd the resi dual i s l i kel y to be smal l at the
opti mum si nce i t i s general practi ce to set real i sti cal l y achi evabl e target
trajectori es. Al though the functi on i n LS (Eq. 2-18) can be mi ni mi zed usi ng a
general unconstrai ned mi ni mi zati on techni que as descri bed i n the
Unconstrai ned Opti mi zati on secti on, certai n characteri sti cs of the probl em
can often be expl oi ted to i mprove the i terati ve effi ci ency of the sol uti on
procedure. The gradi ent and Hessi an matri x of LS (Eq. 2-18) have a speci al
structure.
Denoti ng the Jacobi an matri x of F(x) as J (x), the gradi ent vector of f(x)
as , the Hessi an matri x of f(x) as , and the Hessi an matri x of each
as , we have
(2-20)
where
The matri x Q(x) has the property that when the resi dual tends to zero
as approaches the sol uti on, then Q(x) al so tends to zero. Thus when
i s smal l at the sol uti on, a ver y effecti ve method i s to use the Gauss-Newton
di r ecti on as a basi s for an opti mi zati on pr ocedur e.
Gauss-Newton Method
I n the Gauss-Newton method, a search di recti on, , i s obtai ned at each major
i terati on, k, that i s a sol uti on of the l i near l east-squares probl em
Gauss-Newton
(2-21)
The di recti on deri ved from thi s method i s equi val ent to the Newton di recti on
when the terms of Q(x) can be i gnored. The search di recti on can be used as
part of a l i ne search strategy to ensure that at each i terati on the functi on f(x)
decreases.
F x ( )
m n
G x ( ) H x ( )
F
i
x ( ) H
i
x ( )
G x ( ) 2J x ( )
T
F x ( ) =
H x ( ) 2J x ( )
T
J x ( ) 2Q x ( ) + =
Q x ( ) F
i
x ( ) H
i
x ( )
i 1 =
m

=
F x ( )
x
k
F x ( )
d
k
J x
k
( )d
k
F x
k
( )
2
2
x
n

min
d
k
Least Squares Optimization
2-19
To consi der the effi ci enci es that are possi bl e wi th the Gauss-Newton method,
Fi gure shows the path to the mi ni mum on Rosenbrocks functi on (Eq. 2-2)
when posed as a l east squares probl em. The Gauss-Newton method converges
after onl y 48 functi on eval uati ons usi ng fi ni te di fference gradi ents compared to
140 i terati ons usi ng an unconstrai ned BFGS method.
The Gauss-Newton method often encounters probl ems when the second order
term Q(x) i n Eq. 2-20 i s si gni fi cant. A method that overcomes thi s probl em i s
the Levenberg-Marquardt method.
Figure 2-5: Gauss-Newton Method on Rosenbrocks Function
Levenberg-Marquardt Method
The Levenberg-Marquardt [18,19] method uses a search di recti on that i s a
sol uti on of the l i near set of equati ons
(2-22)
where the scal ar control s both the magni tude and di recti on of . When
i s zero, the di recti on i s i denti cal to that of the Gauss-Newton method. As
tends to i nfi ni ty, tends towards a vector of zeros and a steepest descent
-1
-0.5
0
0.5
1
1.5
2
2.5
3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
o
Start Point
o
Solution
o
o
o
o
o
o
o
o
o
o o
J x
k
( )
T
J x ( )
k
I + ( )d
k
J x
k
( )F x
k
( ) =

k
d
k

k
d
k

k
d
k
2 Introduction to Algorithms
2-20
di recti on. Thi s i mpl i es that for some suffi ci entl y l arge , the term
hol ds true. The term can therefore be control l ed to
ensure descent even when second order terms, whi ch restri ct the effi ci ency of
the Gauss-Newton method, are encountered.
The Levenberg-Marquardt method therefore uses a search di recti on that i s a
cross between the Gauss-Newton di recti on and the steepest descent. Thi s i s
i l l ustrated i n Fi gure 2-6 bel ow. The sol uti on for Rosenbrocks functi on (Eq. 2-2)
converges after 90 functi on eval uati ons compared to 48 for the Gauss-Newton
method. The poorer effi ci ency i s partl y because the Gauss-Newton method i s
general l y more effecti ve when the resi dual i s zero at the sol uti on. However,
such i nformati on i s not al ways avai l abl e beforehand, and occasi onal poorer
effi ci ency of the Levenberg-Marquardt method i s compensated for by i ts
i ncreased robustness.
Figure 2-6: Levenberg-Marquardt Method on Rosenbrocks Function

k
F x
k
d
k
+ ( ) F x
k
( ) <
k
-1
-0.5
0
0.5
1
1.5
2
2.5
3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
o
Start Point
o
Solution
o
o
oo
o
o
o
o
o
o
o
o
o
o
o
o
ooo
Nonlinear Least Squares Implementation
2-21
Nonlinear Least Squares Implementation
For a general survey of nonl i near l east squares methods see Denni s [21].
Speci fi c detai l s on the Levenberg-Marquardt method can be found i n
Mor [20]. Both the Gauss-Newton method and the Levenberg-Marquardt
method are i mpl emented i n the Opti mi zati on Tool box. Detai l s of the
i mpl ementati ons are di scussed bel ow.
Gauss-Newton Implementation
The Gauss-Newton method i s i mpl emented usi ng si mi l ar pol ynomi al l i ne
search strategi es di scussed for unconstrai ned opti mi zati on. I n sol vi ng the
l i near l east squares probl em (Prob. 2.18), exacerbati on of the condi ti oni ng of
the equati ons i s avoi ded by usi ng the QR decomposi ti on of and appl yi ng
the decomposi ti on to (usi ng the MATLAB \ operator). Thi s i s i n contrast
to i nverti ng the expl i ci t matri x, , whi ch can cause unnecessary
errors to occur.
Robustness measures are i ncl uded i n the method. These measures consi st of
changi ng the al gori thm to the Levenberg-Marquardt method when ei ther the
step l ength goes bel ow a threshol d val ue (i n thi s i mpl ementati on 1e15) or
when the condi ti on number of i s bel ow 1e10. The condi ti on number i s
a rati o of the l argest si ngul ar val ue to the smal l est.
Levenberg-Marquardt Implementation
The mai n di ffi cul ty i n the i mpl ementati on of the Levenberg-Marquardt
method i s an effecti ve strategy for control l i ng the si ze of at each i terati on so
that i t i s effi ci ent for a broad spectrum of probl ems. The method used i n thi s
i mpl ementati on i s to esti mate the rel ati ve nonl i neari ty of f(x) usi ng a l i near
predi cted sum of squares and a cubi cl y i nterpol ated esti mate of the
mi ni mum . I n thi s way the si ze of i s determi ned at each i terati on.
The l i near predi cted sum of squares i s cal cul ated as
(2-23)
and the term i s obtai ned by cubi cl y i nterpol ati ng the poi nts and
. A step l ength parameter

i s al so obtai ned from thi s i nterpol ati on,

whi ch i s the esti mated step to the mi ni mum. I f i s greater than ,
then i s reduced, otherwi se i t i s i ncreased. The justi fi cati on for thi s i s that
J x
k
( )
F x
k
( )
J x
k
( )
T
J x
k
( )
J x
k
( )

k
f
p
x
k
( )
f
k
x
*
( )
k
f
p
x
k
( ) J x
k 1
( ) ( )
T
= d
k 1
F x ( ) +
f
k
x
*
( ) f x
k
( )
f x
k 1
( )
f
p
x
k
( ) f
k
x
*
( )

k
2 Introduction to Algorithms
2-22
the di fference between and i s a measure of the effecti veness of
the Gauss-Newton method and the l i neari ty of the probl em. Thi s determi nes
whether to use a di recti on approachi ng the steepest descent di recti on or the
Gauss-Newton di recti on. The formul as for the reducti on and i ncrease i n ,
whi ch have been devel oped through consi derati on of a l arge number of test
probl ems, are shown i n Fi gure 2-7 bel ow.
Figure 2-7: Updating
Fol l owi ng the update of , a sol uti on of Eq. 2-22 i s used to obtai n a search
di recti on, . A step l ength of uni ty i s then taken i n the di recti on , whi ch i s
fol l owed by a l i ne search procedure si mi l ar to that di scussed for the
unconstrai ned i mpl ementati on. The l i ne search procedure ensures that
at each major i terati on and the method i s therefore a descent
method.
The i mpl ementati on has been successful l y tested on a l arge number of
nonl i near probl ems. I t has proved to be more robust than the Gauss-Newton
method and i terati vel y more effi ci ent than an unconstrai ned method. The
Levenberg-Marquardt al gori thm i s the defaul t method used by lsqnonlin. The
Gauss-Newton method can be sel ected by setti ng the options parameter
LevenbergMarquardt to 'off'.
f
p
x
k
( ) f
k
x
*
( )

k
No Yes
f
p
x
k
( ) f
k
x
*
( ) >
I ncrease
k
Reduce
k

k

k
f
k
x
*
( ) f
p
x
k
( )

*
------------------------------------- + =

k

k
1
*
+
---------------- =

k
d
k
d
k
f x
k 1 +
( ) f x
k
( ) <
Constrained Optimization
2-23
Constrained Optimization
I n constrai ned opti mi zati on, the general ai m i s to transform the probl em i nto
an easi er subprobl em that can then be sol ved and used as the basi s of an
i terati ve process. A characteri sti c of a l arge cl ass of earl y methods i s the
transl ati on of the constrai ned probl em to a basi c unconstrai ned probl em by
usi ng a penal ty functi on for constrai nts, whi ch are near or beyond the
constrai nt boundary. I n thi s way the constrai ned probl em i s sol ved usi ng a
sequence of parameteri zed unconstrai ned opti mi zati ons, whi ch i n the l i mi t (of
the sequence) converge to the constrai ned probl em. These methods are now
consi dered rel ati vel y i neffi ci ent and have been repl aced by methods that have
focused on the sol uti on of the Kuhn-Tucker (KT) equati ons. The KT equati ons
are necessary condi ti ons for opti mal i ty for a constrai ned opti mi zati on probl em.
I f the probl em i s a so-cal l ed convex programmi ng probl em, that i s, f(x) and
, are convex functi ons, then the KT equati ons are both
necessary and suffi ci ent for a gl obal sol uti on poi nt.
Referri ng to GP (Eq. 2-1), the Kuhn-Tucker equati ons can be stated as
(2-24)
The fi rst equati on descri bes a cancel i ng of the gradi ents between the objecti ve
functi on and the acti ve constrai nts at the sol uti on poi nt. For the gradi ents to
be cancel ed, Lagrange Mul ti pl i ers ( ) are necessary to bal ance
the devi ati ons i n magni tude of the objecti ve functi on and constrai nt gradi ents.
Si nce onl y acti ve constrai nts are i ncl uded i n thi s cancel i ng operati on,
constrai nts that are not acti ve must not be i ncl uded i n thi s operati on and so
are gi ven Lagrange mul ti pl i ers equal to zero. Thi s i s stated i mpl i ci tl y i n the
l ast two equati ons of Eq. 2-24.
The sol uti on of the KT equati ons forms the basi s to many nonl i near
programmi ng al gori thms. These al gori thms attempt to compute di rectl y the
Lagrange mul ti pl i ers. Constrai ned quasi -Newton methods guarantee
superl i near convergence by accumul ati ng second order i nformati on regardi ng
the KT equati ons usi ng a quasi -Newton updati ng procedure. These methods
G
i
x ( ) i , 1 m , , =
f x

( )
i

G
i
x

( )
i 1 =
m

+ 0 =
G
i
x

( ) 0 = i 1 m
e
, , =

0 i m
e
1 m , , + =

i
i , 1 m , =
2 Introduction to Algorithms
2-24
are commonl y referred to as Sequenti al Quadrati c Programmi ng (SQP)
methods si nce a QP sub-probl em i s sol ved at each major i terati on (al so known
as I terati ve Quadrati c Programmi ng, Recursi ve Quadrati c Programmi ng, and
Constrai ned Vari abl e Metri c methods).
Sequential Quadratic Programming (SQP)
SQP methods represent state-of-the-art i n nonl i near programmi ng methods.
Schi ttowski [22], for exampl e, has i mpl emented and tested a versi on that out
performs every other tested method i n terms of effi ci ency, accuracy, and
percentage of successful sol uti ons, over a l arge number of test probl ems.
Based on the work of Bi ggs [9], Han [10], and Powel l [11,12], the method al l ows
you to cl osel y mi mi c Newtons method for constrai ned opti mi zati on just as i s
done for unconstrai ned opti mi zati on. At each major i terati on an approxi mati on
i s made of the Hessi an of the Lagrangi an functi on usi ng a quasi -Newton
updati ng method. Thi s i s then used to generate a QP sub-probl em whose
sol uti on i s used to form a search di recti on for a l i ne search procedure. An
over vi ew of SQP i s found i n Fl etcher [2], Gi l l et al . [1], Powel l [13], and
Schi ttowski [14]. The general method, however , i s stated her e.
Gi ven the probl em descri pti on i n GP (Eq. 2.1) the pri nci pal i dea i s the
formul ati on of a QP sub-probl em based on a quadrati c approxi mati on of the
Lagrangi an functi on.
(2-25)
Here Eq. 2.1 i s si mpl i fi ed by assumi ng that bound constrai nts have been
expressed as i nequal i ty constrai nts. The QP sub-probl em i s obtai ned by
l i neari zi ng the nonl i near constrai nts.
QP Subproblem
(2-26)
L x , ( ) f x ( )
i
g
i
x ( )
i 1 =
m

+ =
1
2
---d
T
H
k
d f x
k
( )
T
+ d
d
n

minimize
g
i
x
k
( )
T
d g
i
x
k
( ) + 0 = i 1 m
e
, =
g
i
x
k
( )
T
d g
i
x
k
( ) + 0 i m
e
1 + m , =
Constrained Optimization
2-25
Thi s sub-probl em can be sol ved usi ng any QP al gori thm (see, for i nstance, the
Quadrati c Programmi ng Sol uti on secti on). The sol uti on i s used to form a new
i terate
The step l ength parameter i s determi ned by an appropri ate l i ne search
procedure so that a suffi ci ent decrease i n a meri t functi on i s obtai ned (see the
Updati ng the Hessi an Matri x secti on). The matri x i s a posi ti ve defi ni te
approxi mati on of the Hessi an matri x of the Lagrangi an functi on (Eq. 2-25).
can be updated by any of the quasi -Newton methods, al though the BFGS
method (see the secti on Updati ng the Hessi an Matri x) appears to be the most
popul ar.
A nonl i nearl y constrai ned probl em can often be sol ved i n fewer i terati ons than
an unconstrai ned probl em usi ng SQP. One of the reasons for thi s i s that,
because of l i mi ts on the feasi bl e area, the opti mi zer can make wel l -i nformed
deci si ons regardi ng di recti ons of search and step l ength.
Consi der Rosenbrocks functi on (Eq. 2-2) wi th an addi ti onal nonl i near
i nequal i ty constrai nt, g(x)
(2-27)
Thi s was sol ved by an SQP i mpl ementati on i n 96 i terati ons compared to 140
for the unconstrai ned case. Fi gure shows the path to the sol uti on poi nt
starti ng at .
x
k 1 +
x
k

k
d
k
+ =

k
H
k
H
k
x
1
2
x
2
2
1.5 + 0
x 0.9072 0.8228 [ , ] = x 1.9 2 [ , ] =
2 Introduction to Algorithms
2-26
Figure 2-8: SQP Method on Nonlinear Linearly Constrained Rosenbrocks
Function (Eq.2-2)
-1
-0.5
0
0.5
1
1.5
2
2.5
3
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
o
Start Point
o
o
o
o
o
o
o oo
o
o
o
ooo
Constraint boundary
Infeasible region
Feasible region
Solution
SQP Implementation
2-27
SQP Implementation
The MATLAB SQP i mpl ementati on consi sts of three mai n stages, whi ch are
di scussed bri efl y i n the fol l owi ng sub-secti ons:
Updati ng of the Hessi an matri x of the Lagrangi an functi on
Quadrati c programmi ng probl em sol uti on
Li ne search and meri t functi on cal cul ati on
Updating the Hessian Matrix
At each major i terati on a posi ti ve defi ni te quasi -Newton approxi mati on of the
Hessi an of the Lagrangi an functi on, H,i s cal cul ated usi ng the BFGS method
where
i
(i = 1,...,m) i s an esti mate of the Lagrange mul ti pl i ers.
Hessian Update(BFGS)
(2-28)
Powel l [11] recommends keepi ng the Hessi an posi ti ve defi ni te even though i t
may be posi ti ve i ndefi ni te at the sol uti on poi nt. A posi ti ve defi ni te Hessi an i s
mai ntai ned provi di ng i s posi ti ve at each update and that H i s i ni ti al i zed
wi th a posi ti ve defi ni te matri x. When i s not posi ti ve, i s modi fi ed on
an el ement by el ement basi s so that . The general ai m of thi s
modi fi cati on i s to di stort the el ements of , whi ch contri bute to a posi ti ve
defi ni te update, as l i ttl e as possi bl e. Therefore, i n the i ni ti al phase of the
modi fi cati on, the most negati ve el ement of i s repeatedl y hal ved. Thi s
procedure i s conti nued unti l i s greater than or equal to 1e-5. I f after thi s
procedure, i s sti l l not posi ti ve, i s modi fi ed by addi ng a vector v
mul ti pl i ed by a constant scal ar w, that i s,
(2-29)
H
k 1 +
H
k
q
k
q
k
T
q
k
T
s
k
-------------
H
k
T
H
k
s
k
T
H
k
s
k
-------------------- + = where
s
k
x
k 1 +
x
k
=
q
k
f x
k 1 +
( )
i
g
i
x
k 1 +
( )
i 1 =
n

f x
k
( )
i
g
i
x
k
( )
i 1 =
n

+
,


_
+ =
q
k
T
s
k
q
k
T
s
k
q
k
q
k
T
s
k
0 >
q
k
q
k
.s

k
q
k
T
s
k
q
k
T
s
k
q
k
q
k
q
k
wv + =
2 Introduction to Algorithms
2-28
where
and wi s systemati cal l y i ncreased unti l becomes posi ti ve.
The functi ons fmincon, fminimax, fgoalattain, and fseminf al l use SQP. I f the
options parameter Display i s set to 'iter', then vari ous i nformati on i s gi ven
such as functi on val ues and the maxi mum constrai nt vi ol ati on. When the
Hessi an has to be modi fi ed usi ng the fi rst phase of the procedure descri bed
above to keep i t posi ti ve defi ni te, then Hessian modified i s di spl ayed. I f the
Hessi an has to be modi fi ed agai n usi ng the second phase of the approach
descri bed above, then Hessian modified twice i s di spl ayed. When the QP
sub-probl em i s i nfeasi bl e, then i nfeasi bl e wi l l be di spl ayed. Such di spl ays are
usual l y not a cause for concern but i ndi cate that the probl em i s hi ghl y
nonl i near and that convergence may take l onger than usual . Someti mes the
message no update i s di spl ayed i ndi cati ng that i s nearl y zero. Thi s can
be an i ndi cati on that the probl em setup i s wrong or you are tryi ng to mi ni mi ze
a nonconti nuous functi on.
Quadratic Programming Solution
At each major i terati on of the SQP method a QP probl em i s sol ved of the form
where refers to the ith row of the m-by-n matri x A.
QP
(2-30)
The method used i n the Opti mi zati on Tool box i s an acti ve set strategy (al so
known as a projecti on method)si mi l ar to that of Gi l l et al ., descri bed i n [16] and
[17]. I t has been modi fi ed for both LP and QP probl ems.
The sol uti on procedure i nvol ves two phases: the fi rst phase i nvol ves the
cal cul ati on of a feasi bl e poi nt (i f one exi sts), the second phase i nvol ves the
generati on of an i terati ve sequence of feasi bl e poi nts that converge to the
sol uti on. I n thi s method an acti ve set i s mai ntai ned, , whi ch i s an esti mate
v
i
g
i
x
k 1 +
( ) g
i
x
k 1 +
( ) g
i
x
k
( ) g
i
x
k
( ), =
i f q
k
( )
i
w 0 < and q
k
( )
i
s
k
( )
i
0 < i 1 m , = ( )
v
i
0 otherwi se =
q
k
T
s
k
q
k
T
s
k
A
i
q d ( )
d
n

minimize
1
2
---d
T
Hd c
T
d + =
A
i
d b
i
= i 1 m
e
, , =
A
i
d b
i
i m
e
1 + m , , =
A
k
SQP Implementation
2-29
of the acti ve constrai nts (i .e., whi ch are on the constrai nt boundari es) at the
sol uti on poi nt. Vi rtual l y al l QP al gori thms are acti ve set methods. Thi s poi nt
i s emphasi zed because there exi st many di fferent methods that are very
si mi l ar i n structure but that are descri bed i n wi del y di fferent terms.
i s updated at each i terati on, k, and thi s i s used to form a basi s for a search
di recti on . Equal i ty constrai nts al ways remai n i n the acti ve set, . The
notati on for the vari abl e, , i s used here to di sti ngui sh i t from i n the major
i terati ons of the SQP method. The search di recti on, , i s cal cul ated and
mi ni mi zes the objecti ve functi on whi l e remai ni ng on any acti ve constrai nt
boundari es. The feasi bl e subspace for i s formed from a basi s, whose
col umns are orthogonal to the esti mate of the acti ve set (i .e., ).
Thus a search di recti on, whi ch i s formed from a l i near summati on of any
combi nati on of the col umns of , i s guaranteed to remai n on the boundari es
of the acti ve constrai nts.
The matri x i s formed from the l ast m-l col umns of the QR decomposi ti on of
the matri x , where l i s the number of acti ve constrai nts and l <m. That i s,
i s gi ven by
(2-31)
Havi ng found , a new search di recti on i s sought that mi ni mi zes
where i s i n the nul l space of the acti ve constrai nts, that i s, i s a l i near
combi nati on of the col umns of : for some vector p.
Then i f we vi ew our quadrati c as a functi on of p, by substi tuti ng for , we
have
(2-32)
Di fferenti ati ng thi s wi th respect to p yi el ds
(2-33)
i s referred to as the projected gradi ent of the quadrati c functi on because
i t i s the gr adi ent pr ojected i n the subspace defi ned by . The ter m i s
cal l ed the projected Hessi an. Assumi ng the Hessi an matri x H i s posi ti ve
Ak
d
k
Ak
d
k
d
k
d
k
d
k
Z
k
Ak AkZ
k
0 =
Z
k
Z
k
A
k
T
Z
k
Z
k
Q : l 1:m + , [ ] =
where Q
T
A
k
T
R
0
=
Z
k
d
k
q d ( )
d
k
d
k
Z
k
d
k
Z
k
p =
d
k
q p ( )
1
2
---p
T
Z
k
T
HZ
k
p c
T
Z
k
p + =
q p ( ) Z
k
T
HZ
k
p Z
k
T
c + =
q p ( )
Z
k
Z
k
T
HZ
k
2 Introduction to Algorithms
2-30
defi ni te (whi ch i s the case i n thi s i mpl ementati on of SQP), then the mi ni mum
of the functi on q(p) i n the subspace defi ned by occurs when ,
whi ch i s the sol uti on of the system of l i near equati ons
(2-34)
A step i s then taken of the for m
(2-35)
At each i terati on, because of the quadrati c nature of the objecti ve functi on,
there ar e onl y two choi ces of step l ength . A step of uni ty al ong i s the exact
step to the mi ni mum of the functi on restri cted to the nul l space of . I f such
a step can be taken, wi thout vi ol ati on of the constrai nts, then thi s i s the
sol uti on to QP (Eq. 2.31). Otherwi se, the step al ong to the nearest
constrai nt i s l ess than uni ty and a new constrai nt i s i ncl uded i n the acti ve set
at the next i terate. The di stance to the constrai nt boundari es i n any di recti on
i s gi ven by
(2-36)
whi ch i s defi ned for constrai nts not i n the acti ve set, and where the di recti on
i s towards the constrai nt boundary, i .e., .
When n i ndependent constrai nts are i ncl uded i n the acti ve set, wi thout
l ocati on of the mi ni mum, Lagrange mul ti pl i ers, are cal cul ated that sati sfy
the nonsi ngul ar set of l i near equati ons
(2-37)
I f al l el ements of are posi ti ve, i s the opti mal sol uti on of QP (Eq. 2.31).
However, i f any component of i s negati ve, and i t does not correspond to an
equal i ty constrai nt, then the correspondi ng el ement i s del eted from the acti ve
set and a new i terate i s sought.
Initialization
The al gori thm requi res a feasi bl e poi nt to start. I f the current poi nt from the
SQP method i s not feasi bl e, then a poi nt can be found by sol vi ng the l i near
programmi ng probl em
Z
k
q p ( ) 0 =
Z
k
T
HZ
k
p Z
k
T
c =
x
k 1 +
x
k
d
k
+ = where d
k
Z
k
T
p =
d
k
Ak
d
k
d
k

A
i
x
k
b
i
( )
A
i
d
k
-------------------------------

' ;

i
min = i 1 m , , = ( )
d
k
A
i
d
k
0 i , > 1 m , , =

k
Ak
T

k
c =

k
x
k

k
SQP Implementation
2-31
(2-38)
The notati on i ndi cates the ith row of the matri x A. A feasi bl e poi nt (i f one
exi sts) to Eq. 2.38 can be found by setti ng x to a val ue that sati sfi es the
equal i ty constrai nts. Thi s can be achi eved by sol vi ng an under- or
over-determi ned set of l i near equati ons formed from the set of equal i ty
constrai nts. I f there i s a sol uti on to thi s probl em, then the sl ack vari abl e i s
set to the maxi mum i nequal i ty constrai nt at thi s poi nt.
The above QP al gori thm i s modi fi ed for LP probl ems by setti ng the search
di recti on to the steepest descent di recti on at each i terati on where i s the
gradi ent of the objecti ve functi on (equal to the coeffi ci ents of the l i near
objecti ve functi on)
(2-39)
I f a feasi bl e poi nt i s found usi ng the above LP method, the mai n QP phase i s
entered. The search di recti on i s i ni ti al i zed wi th a search di recti on found
from sol vi ng the set of l i near equati ons
(2-40)
where i s the gradi ent of the objecti ve functi on at the current i terate (i .e.,
).
I f a feasi bl e sol uti on i s not found for the QP probl em, the di recti on of search for
the mai n SQP routi ne i s taken as one that mi ni mi zes

.
Line Search and Merit Function
The sol uti on to the QP sub-probl em produces a vector , whi ch i s used to form
a new i terate
(2-41)
The step l ength parameter i s determi ned i n order to produce a suffi ci ent
decrease i n a meri t functi on. The meri t functi on used by Han [15] and
Powel l [15] of the form bel ow has been used i n thi s i mpl ementati on

x
n
,
minimize
A
i
x b
i
= i 1 m
e
, , =
A
i
x b
i
i m
e
1 + m , , =
A
i

g
k
d
k
Z
k
Z
k
T
g
k
=
d
k
d
1
Hd
1
g
k
=
g
k
x
k
Hx
k
c +
d
k
d
k
x
k 1 +
x
k
d
k
+ =

k
2 Introduction to Algorithms
2-32
Merit Function
(2-42)
Powel l recommends setti ng the penal ty parameter
(2-43)
Thi s al l ows posi ti ve contri buti on form constrai nts that are i nacti ve i n the QP
sol uti on but were recentl y acti ve. I n thi s i mpl ementati on, i ni ti al l y the penal ty
parameter i s set to
(2-44)
where represents the Eucl i dean norm.
Thi s ensures l arger contri buti ons to the penal ty parameter from constrai nts
wi th smal l er gradi ents, whi ch woul d be the case for acti ve constrai nts at the
sol uti on poi nt.
x ( ) f x ( ) r
i
g
i
x ( )
i 1 =
m
e

r
i
0 g
i
x ( ) , { } max
i m
e
1 + =
m

+ + =
r
i
r
k 1 +
( )
i

i
1
2
--- r
k
( )
i

i
+ ( ) ,

' ;

,
i
max = = i 1 m , , =
r
i
r
i
f x ( )
g
i
x ( )
-------------------------- =

Multiobjective Optimization
2-33
Multiobjective Optimization
The ri gi di ty of the mathemati cal probl em posed by the general opti mi zati on
formul ati on gi ven i n GP (Eq. 2-1) i s often remote from that of a practi cal desi gn
probl em. Rarel y does a si ngl e objecti ve wi th several hard constrai nts
adequatel y represent the probl em bei ng faced. More often there i s a vector of
objecti ves that must be traded off i n some
way. The rel ati ve i mportance of these objecti ves i s not general l y known unti l
the systems best capabi l i ti es are determi ned and trade-offs between the
objecti ves ful l y understood. As the number of objecti ves i ncreases, trade-offs
are l i kel y to become compl ex and l ess easi l y quanti fi ed. There i s much rel i ance
on the i ntui ti on of the desi gner and hi s or her abi l i ty to express preferences
throughout the opti mi zati on cycl e. Thus, requi rements for a mul ti objecti ve
desi gn strategy are to enabl e a natural probl em formul ati on to be expressed,
yet be abl e to sol ve the probl em and enter preferences i nto a numeri cal l y
tractabl e and real i sti c desi gn probl em.
Thi s secti on begi ns wi th an i ntroducti on to mul ti objecti ve opti mi zati on,
l ooki ng at a number of al ternati ve methods. Attenti on i s focused on the Goal
Attai nment method, whi ch can be posed as a nonl i near programi ng probl em.
Al gori thm i mprovements to the SQP method are presented for use wi th the
Goal Attai nment method.
Introduction
Mul ti objecti ve opti mi zati on i s concerned wi th the mi ni mi zati on of a vector of
objecti ves F(x) that may be the subject of a number of constrai nts or bounds.
MO
(2-45)
Note that, because F(x) i s a vector , i f any of the components of F(x) are
competi ng, there i s no uni que sol uti on to thi s probl em. I nstead, the concept of
noni nferi ori ty [25] (al so cal l ed Pareto opti mal i ty [24], [26]) must be used to
characteri ze the objecti ves. A noni nferi or sol uti on i s one i n whi ch an
F x ( ) F
1
x ( ) F
2
x ( ) F
m
x ( ) , , , { } =
F x ( )
x
n

minimize
G
i
x ( ) 0 = i 1 m
e
, , =
G
i
x ( ) 0 i m
e
1 m , , + =
x
l
x x
u

2 Introduction to Algorithms
2-34
i mprovement i n one objecti ve requi res a degradati on of another. To defi ne thi s
concept more preci sel y, consi der a feasi bl e regi on, , i n the parameter space
that sati sfi es al l the constrai nts, i .e.,
(2-46)
Thi s al l ows us to defi ne the correspondi ng feasi bl e regi on for the objecti ve
functi on space
(2-47)
The performance vector, F(x), maps parameter space i nto objecti ve functi on
space as i s represented for a two-di mensi onal case i n Fi gure 2-9 bel ow.
Figure 2-9: Mapping from Parameter Space into Objective Function Space
A noni nferi or sol uti on poi nt can now be defi ned.
Definition: A poi nt i s a noni nferi or sol uti on i f for some nei ghborhood of
there does not exi st a such that and
(2-48)
x
n

x
n
{ } =
subject to g
i
x ( ) 0 = i 1 m
e
, , =
g
i
x ( ) 0 i m
e
1 + m , , =
x
l
x x
u

y
m
{ } = where y F x ( ) = subject to x .
x
1
x
2
F1
F
2

x
*

x
*
x x
*
x + ( )
F
i
x
*
x + ( ) F
i
x
*
( ) i 1 m , , =
F
j
x
*
x + ( ) F
j
x
*
( ) < for some j .
Multiobjective Optimization
2-35
I n the two-di mensi onal representati on of Fi gure 2-10 the set of noni nferi or
sol uti ons l i es on the curve between C and D. Poi nts A and B represent speci fi c
noni nferi or poi nts.
Figure 2-10: Set of Noninferior Solutions
A and B are cl earl y noni nferi or sol uti on poi nts because an i mprovement i n one
objecti ve, , requi res a degradati on i n the other objecti ve, , i .e.,
.
Si nce any poi nt i n that i s not a noni nferi or poi nt represents a poi nt i n whi ch
i mprovement can be attai ned i n al l the objecti ves, i t i s cl ear that such a poi nt
i s of no val ue. Mul ti objecti ve opti mi zati on i s, therefore, concerned wi th the
generati on and sel ecti on of noni nferi or sol uti on poi nts. The techni ques for
mul ti objecti ve opti mi zati on are wi de and vari ed and al l the methods cannot be
covered wi thi n the scope of thi s tool box. However, some of the techni ques are
descri bed bel ow.
Weighted Sum Strategy
The wei ghted sum strategy converts the mul ti objecti ve probl em of mi ni mi zi ng
the vector i nto a scal ar probl em by constructi ng a wei ghted sum of al l the
objecti ves.

F
1
F
2

noni nferi or
sol uti ons

A
C
B
D

F
1A
F
1B
F
2A
F
2B
F
1
F
2
F
1B
F
1A
< F
2B
F
2A
> ,
F x ( )
2 Introduction to Algorithms
2-36
Weighted Sum
(2-49)
The probl em can then be opti mi zed usi ng a standard unconstrai ned
opti mi zati on al gori thm. The probl em here i s i n attachi ng wei ghti ng
coeffi ci ents to each of the objecti ves. The wei ghti ng coeffi ci ents do not
necessari l y correspond di rectl y to the rel ati ve i mportance of the objecti ves or
al l ow trade-offs between the objecti ves to be expressed. Further, the
noni nferi or sol uti on boundary may be nonconcurrent so that certai n sol uti ons
are not accessi bl e.
Thi s can be i l l ustrated geometri cal l y. Consi der the two objecti ve case i n
Fi gure . I n the objecti ve functi on space a l i ne, L, i s drawn. The
mi ni mi zati on of Eq. 2-49 can be i nterpreted as fi ndi ng the val ue of cfor whi ch
L just touches the boundary of as i t proceeds outwards from the ori gi n.
Sel ecti on of wei ghts and , therefore, defi nes the sl ope of L, whi ch i n turn
l eads to the sol uti on poi nt where L touches the boundary of .
Figure 2-11: Geometrical Representation of the Weighted Sum Method
The aforementi oned convexi ty probl em ari ses when the l ower boundary of i s
nonconvex as shown i n Fi gure 2-12. I n thi s case the set of noni nferi or sol uti ons
between A and B i s not avai l abl e.
f x ( ) w
i
F
i
x ( )
2

i 1 =
m

=
x
minimize
w
T
F x ( ) c =
w
1
w
2
L

F
2
F
1
w
T
F x ( ) c =
Multiobjective Optimization
2-37
Figure 2-12: Nonconvex Solution Boundary
-Constraint Method
A procedure that overcomes some of the convexi ty probl ems of the wei ghted
sum techni que i s the -constrai nt method. Thi s i nvol ves mi ni mi zi ng a pri mary
objecti ve, , and expressi ng the other objecti ves i n the form of i nequal i ty
constrai nts
(2-50)
Fi gure 2-13 shows a two-di mensi onal representati on of the -constrai nt
method for a two objecti ve probl em.
B
A
L

F
2
F
1

F
p
F
p
x ( )
x
minimize
subject to F
i
x ( )
i
i 1 m , , = i p

2 Introduction to Algorithms
2-38
Figure 2-13: Geometrical Representation of -Constraint Method
Thi s approach i s abl e to i denti fy a number of noni nferi or sol uti ons on a
nonconvex boundary that are not obtai nabl e usi ng the wei ghted sum
techni que, for exampl e, at the sol uti on poi nt and . A
probl em wi th thi s method i s, however, a sui tabl e sel ecti on of to ensure a
feasi bl e sol uti on. A further di sadvantage of thi s approach i s that the use of
hard constrai nts i s rarel y adequate for expressi ng true desi gn objecti ves.
Si mi l ar methods exi st, such as that of Wal tz [31], whi ch pri ori ti ze the
objecti ves. The opti mi zati on proceeds wi th reference to these pri ori ti es and
al l owabl e bounds of acceptance. The di ffi cul ty here i s i n expressi ng such
i nformati on at earl y stages of the opti mi zati on cycl e.
I n order for the desi gners true preferences to be put i nto a mathemati cal
descri pti on, the desi gners must express a ful l tabl e of thei r preferences and
sati sfacti on l evel s for a range of objecti ve val ue combi nati ons. A procedure
must then be real i zed that i s abl e to fi nd a sol uti on wi th reference to thi s. Such
methods have been deri ved for di scr ete functi ons usi ng the br anches of
stati sti cs known as deci si on theory and game theory (for a basi c i ntroducti on,
see [28]). I mpl ementati on for conti nuous functi ons requi res sui tabl e
di screti zati on strategi es and compl ex sol uti on methods. Si nce i t i s rare for the
desi gner to know such detai l ed i nformati on, thi s method i s deemed i mpracti cal
for most practi cal desi gn probl ems. I t i s, however, seen as a possi bl e area for
further research.

F
1s
F
1
F
2

2
F
1
x ( )
x
minimize subject to: F
2
x
2

F
1
F
1s
= F
2

2
=

Multiobjective Optimization
2-39
What i s requi red i s a formul ati on that i s si mpl e to express, retai ns the
desi gners preferences, and i s numeri cal l y tractabl e.
Goal Attainment Method
The method descri bed here i s the Goal Attai nment method of Gembi cki [27].
Thi s i nvol ves expressi ng a set of desi gn goal s, , whi ch i s
associ ated wi th a set of objecti ves, . The
probl em formul ati on al l ows the objecti ves to be under- or over-achi eved
enabl i ng the desi gner to be rel ati vel y i mpreci se about i ni ti al desi gn goal s. The
rel ati ve degree of under- or over-achi evement of the goal s i s control l ed by a
vector of wei ghti ng coeffi ci ents, , and i s expressed as a
standard opti mi zati on probl em usi ng the fol l owi ng formul ati on:
Goal Attainment
(2-51)
The term i ntroduces an el ement of slackness i nto the probl em, whi ch
otherwi se i mposes that the goal s be ri gi dl y met. The wei ghti ng vector, w,
enabl es the desi gner to express a measure of the rel ati ve trade-offs between the
objecti ves. For i nstance, setti ng the wei ghti ng vector, w, equal to the i ni ti al
goal s i ndi cates that the same percentage under- or over-attai nment of the
goal s, , i s achi eved. Hard constrai nts can be i ncorporated i nto the desi gn by
setti ng a parti cul ar wei ghti ng factor to zero (i .e., ). The Goal Attai nment
method provi des a conveni ent i ntui ti ve i nterpretati on of the desi gn probl em,
whi ch i s sol vabl e usi ng standard opti mi zati on procedures. I l l ustrati ve
exampl es of the use of Goal Attai nment method i n control system desi gn can
be found i n Fl emi ng [29,30].
The Goal Attai nment method i s represented geometri cal l y i n Fi gure 2-14 for
the two-di mensi onal probl em.
F
*
F
1
*
F
2
*
F
m
*
, , , { } =
F x ( ) F
1
x ( ) F
2
x ( ) F
m
x ( ) , , , { } =
w w
1
w
2
w
m
, , , { } =

x ,
minimize
such that F
i
x ( ) w
i
F
i
*
i 1 m , , =
w
i

F
*
w
i
0 =
2 Introduction to Algorithms
2-40
Figure 2-14: Geometrical Representation of Goal Attainment Method
Speci fi cati on of the goal s, , defi nes the goal poi nt, P. The wei ghti ng
vector defi nes the di recti on of search from P to the feasi bl e functi on space,
. Duri ng the opti mi zati on

i s vari ed, whi ch changes the si ze of the feasi bl e
regi on. The constrai nt boundari es converge to the uni que sol uti on poi nt
.
Algorithm Improvements for Goal Attainment
Method
The Goal Attai nment method has the advantage that i t can be posed as a
nonl i near programmi ng probl em. Characteri sti cs of the probl em can al so be
expl oi ted i n a nonl i near programmi ng al gori thm. I n Sequenti al Quadrati c
Programmi ng (SQP) the choi ce of meri t functi on for the l i ne search i s not easy
because, i n many cases, i t i s di ffi cul t to defi ne the rel ati ve i mportance
between i mprovi ng the objecti ve functi on and reduci ng constrai nt vi ol ati ons.
Thi s has resul ted i n a number of di fferent schemes for constructi ng the meri t
functi on (see, for exampl e, Schi ttowski [22]). I n Goal Attai nment programmi ng
there may be a more appropri ate meri t functi on, whi ch can be achi eved by
posi ng Eq. 2-51 as the mi ni max probl em
w
F
2
F
1
F
1s
F
2s

P
()

x ,
minimize subject to: F
1
x ( ) w
1
F
1
*

F
2
x ( ) w
2
F
2
*

F
1
*
F
2
*
F
1
*
F
2
*
, { }
( )
F
1s
F
2s
, }
Multiobjective Optimization
2-41
(2-52)
Fol l owi ng the argument of Brayton et al . [32] for mi ni max opti mi zati on usi ng
SQP, usi ng the meri t functi on of Eq. 2-43 for the Goal Attai nment probl em of
Eq. 2-52, gi ves
(2-53)
When the meri t functi on of Eq. 2-53 i s used as the basi s of a l i ne search
procedure, then, al though may decrease for a step i n a gi ven search
di recti on, the functi on max may paradoxi cal l y i ncrease. Thi s i s accepti ng a
degradati on i n the worst case objecti ve. Si nce the worst case objecti ve i s
responsi bl e for the val ue of the objecti ve functi on

, thi s i s accepti ng a step that
ul ti matel y i ncreases the objecti ve functi on to be mi ni mi zed. Conversel y,
may i ncrease when max decreases i mpl yi ng a rejecti on of a step that
i mpr oves the worst case objecti ve.
Fol l owi ng the l i nes of Brayton et al . [32], a sol uti on i s therefore to set
equal to the worst case objecti ve, i .e.,
(2-54)
A probl em i n the Goal Attai nment method i s that i t i s common to use a
wei ghti ng coeffi ci ent equal to zero to i ncorporate hard constrai nts. The meri t
functi on of Eq. 2-54 then becomes i nfi ni te for arbi trary vi ol ati ons of the
constrai nts. To overcome thi s probl em whi l e sti l l retai ni ng the features of Eq.
2-54 the meri t functi on i s combi ned wi th that of Eq. 2-43 gi vi ng the fol l owi ng:
(2-55)
Another feature that can be expl oi ted i n SQP i s the objecti ve functi on

. From
the KT equati ons (Eq. 2-24) i t can be shown that the approxi mati on to the

i
{ }
i
max
x
n

minimize
where
i
F
i
x ( ) F
i
*

w
i
-------------------------- = i 1 m , , =
x , ( ) r
i
0, F
i
x ( ) w
i
F
i
*
{ } max
i 1 =
m

+ =
x , ( )

i
x , ( )
i
x ( )
x ( )
i
i
max . =
x ( )
r
i
0, F
i
x ( ) w
i
F
i
*
{ } max i f w
i
0 =

i
i
max i , 1 m , , = otherwi se

'

i 1 =
m

=
2 Introduction to Algorithms
2-42
Hessi an of the Lagrangi an, H, shoul d have zeros i n the rows and col umns
associ ated wi th the vari abl e . By i ni ti al i zi ng H as the i denti ty matri x, thi s
property does not appear. H i s therefore i ni ti al i zed and mai ntai ned to have
zeros i n the rows and col umns associ ated wi th .
These changes make the Hessi an, H, i ndefi ni te, therefore H i s set to have zeros
i n the rows and col umns associ ated wi th , except for the di agonal el ement,
whi ch i s set to a smal l posi ti ve number (e.g., 1e10). Thi s al l ows use of the fast
convergi ng posi ti ve defi ni te QP method descri bed i n the Quadrati c
Programmi ng Sol uti on secti on.
The above modi fi cati ons have been i mpl emented i n fgoalattain and have
been found to make the method mor e r obust. However , due to the rapi d
convergence of the SQP method, the requi rement that the meri t functi on
stri ctl y decrease someti mes requi res more functi on eval uati ons than an
i mpl ementati on of SQP usi ng the meri t functi on of (Eq. 2-43).
Review
2-43
Review
A number of di fferent opti mi zati on strategi es have been di scussed. The
al gori thms used (e.g., BFGS, Levenberg-Marquardt and SQP) have been
chosen for thei r robustness and i terati ve effi ci ency. The choi ce of probl em
formul ati on (e.g., unconstrai ned, l east squares, constrai ned, mi ni max,
mul ti objecti ve, or goal attai nment) depends on the probl em bei ng consi dered
and the requi red executi on effi ci ency.
2 Introduction to Algorithms
2-44
References
[1] Gi l l , P.E., W. Murray, and M.H.Wri ght, Practical Optimization, Academi c
Press, London, 1981.
[2] Fl etcher, R., Practi cal Methods of Opti mi zati on, Vol. 1, Unconstrained
Optimization, and Vol. 2, Constrained Optimization, John Wi l ey and Sons.,
1980.
[3] Broyden, C.G., The Convergence of a Cl ass of Doubl e-rank Mi ni mi zati on
Al gori thms, J .I nst. Maths. Applics., Vol . 6, pp. 76-90, 1970.
[4] Fl etcher, R., A New Approach to Vari abl e Metri c Al gori thms, Computer
J ournal, Vol . 13, pp. 317-322, 1970.
[5] Gol dfarb, D., A Fami l y of Vari abl e Metri c Updates Deri ved by Vari ati onal
Means, Mathematics of Computing, Vol . 24, pp. 23-26, 1970.
[6] Shanno, D.F., Condi ti oni ng of Quasi -Newton Methods for Functi on
Mi ni mi zati on, Mathematics of Computing, Vol . 24, pp. 647-656, 1970.
[7] Davi don, W.C., Vari abl e Metri c Method for Mi ni mi zati on, A.E.C. Research
and Development Report, ANL-5990, 1959.
[8] Fl etcher, R. and M.J.D. Powel l , A Rapi dl y Convergent Descent Method for
Mi ni mi zati on, Computer J ., Vol . 6, pp. 163-168, 1963.
[9] Bi ggs, M.C., Constrai ned Mi ni mi zati on Usi ng Recursi ve Quadrati c
Programmi ng, Towards Global Optimization (L.C.W.Di xon and G.P.Szergo,
eds.), North-Hol l and, pp. 341-349, 1975.
[10] Han, S.P., A Gl obal l y Convergent Method for Nonl i near Programmi ng,
J . Optimization Theory and Applications, Vol . 22, p. 297, 1977.
[11] Powel l , M.J.D., A Fast Al gori thm for Nonl i nearl y Constrai ned
Opti mi zati on Cal cul ati ons, Numerical Analysis, G.A.Watson ed., Lecture
Notes i n Mathemati cs, Spri nger Verl ag, Vol . 630, 1978.
[12] Powel l , M.J.D., The Convergence of Vari abl e Metri c Methods for
Nonl i nearl y Constrai ned Opti mi zati on Cal cul ati ons, Nonlinear Programming
3, (O.L. Mangasari an, R.R. Meyer and S.M. Robi nson, eds.), Academi c Press,
1978.
References
2-45
[13] Powel l , M.J.D., Vari abl e Metri c Methods for Constrai ned Opti mi zati on,
Mathematical Programming: TheStateof theArt, (A.Bachem, M.Grotschel
and B.Korte, eds.) Spri nger Verl ag, pp. 288-311, 1983.
[14] Hock, W. and K. Schi ttowski , A Comparati ve Performance Eval uati on of
27 Nonl i near Programmi ng Codes, Computing, Vol . 30, p. 335, 1983.
[15] Dantzi g, G., Linear Programmingand Extensions, Pri nceton Uni versi ty
Press, Pri nceton, 1963.
[16] Gi l l , P.E., W. Murray, and M.H. Wri ght, Numerical Linear Algebra and
Optimization, Vol . 1, Addi son Wesl ey, 1991.
[17] Gi l l , P.E., W. Murray, M.A. Saunders, and M.H. Wri ght, Procedures for
Opti mi zati on Probl ems wi th a Mi xture of Bounds and General Li near
Constrai nts, ACM Trans. Math. Software, Vol . 10, pp. 282-298, 1984.
[18] Levenberg, K., A Method for the Sol uti on of Certai n Probl ems i n Last
Squares, Quart. Appl. Math. Vol . 2, pp. 164-168, 1944.
[19] Marquardt, D., An Al gori thm for Least-Squares Esti mati on of Nonl i near
Parameters, SI AM J . Appl. Math. Vol . 11, pp. 431-441, 1963.
[20] Mor, J.J., The Levenberg-Marquardt Al gori thm: I mpl ementati on and
Theor y, Numerical Analysis, ed. G. A. Watson, Lecture Notes i n Mathemati cs
630, Spri nger Verl ag, pp. 105-116, 1977.
[21] Denni s, J.E., Jr., Nonl i near Least Squares, Stateof theArt in Numerical
Analysis ed. D. Jacobs, Academi c Press, pp. 269-312, 1977.
[22] Schi ttowski , K., NLQPL: A FORTRAN-Subrouti ne Sol vi ng Constrai ned
Nonl i near Programmi ng Probl ems, Annalsof Operations Research, Vol . 5, pp.
485-500, 1985.
[23] NAG Fortran Library Manual, Mark 12, Vol . 4, E04UAF, p. 16.
[24] Censor, Y., Pareto Opti mal i ty i n Mul ti objecti ve Probl ems, Appl. Math.
Optimiz., Vol . 4, pp. 41-59, 1977.
[25] Zadeh, L.A., Opti mal i ty and Nonscal ar-val ued Performance Cri teri a,
I EEE Trans. Automat. Contr., Vol . AC-8, p. 1, 1963.
[26] Da Cunha, N.O. and E. Pol ak, Constrai ned Mi ni mi zati on Under
Vector-val ued Cri teri a i n Fi ni te Di mensi onal Spaces, J .Math. Anal. Appl.,
Vol . 19, pp. 103-124, 1967.
2 Introduction to Algorithms
2-46
[27] Gembi cki , F.W., Vector Opti mi zati on for Control wi th Performance and
Parameter Sensi ti vi ty I ndi ces, Ph.D. Di ssertati on, Case Western Reserve
Uni v., Cl evel and, Ohi o, 1974.
[28] Hol l i ngdal e, S.H., Methods of Operational Analysis in Newer Uses of
Mathematics (James Li ghthi l l , ed.), Pengui n Books, 1978.
[29] Fl emi ng, P.J., Appl i cati on of Mul ti objecti ve Opti mi zati on to Compensator
Desi gn for SI SO Control Systems, Electronics Letters, Vol . 22, No. 5, pp.
258-259, 1986.
[30] Fl emi ng, P.J., Computer-Ai ded Control System Desi gn of Regul ators
usi ng a Mul ti objecti ve Opti mi zati on Approach, Proc. I FAC Control
Applications of Nonlinear Porg. and Optim., Capri , I tal y, pp. 47-52, 1985.
[31] Wal tz, F.M., An Engi neeri ng Approach: Hi erarchi cal Opti mi zati on
Cri teri a, I EEE Trans., Vol . AC-12, pp. 179-180, Apri l , 1967.
[32] Brayton, R.K., S.W. Di rector, G.D. Hachtel , and L.Vi di gal , A New
Al gori thm for Stati sti cal Ci rcui t Desi gn Based on Quasi -Newton Methods and
Functi on Spl i tti ng, I EEE Transactions on Circuits and Systems, Vol . CAS-26,
pp. 784-794, Sept. 1979.
[33] Nel der, J.A. and R. Mead, A Si mpl ex Method for Functi on Mi ni mi zati on,
Computer J ., Vol .7, pp. 308-313, 1965.
[34] Grace, A.C.W., ComputerAi ded Control System Desi gn Usi ng
Opti mi zati on Techni ques, Ph.D. Thesi s, Uni versi ty of Wal es, Bangor,
Gwynedd, UK, 1989.
[35] Madsen, K. and H. Schjaer-Jacobsen, Algorithms for Worst Case
ToleranceOptimization, I EEE Transactions of Circuits and Systems, Vol .
CAS-26, Sept. 1979.
[36] Forsythe, G.F., M.A. Mal col m, and C.B. Mol er, Computer Methods for
Mathematical Computations, Prenti ce Hal l , 1976.
[37] Dantzi g, G.B., A. Orden, and P. Wol fe, General i zed Si mpl ex Method for
Mi ni mi zi ng a Li near from Under Li near I nequal i ty Constrai nts, Pacific J .
Math. Vol . 5, pp. 183-195.
3
Large-Scal e Al gori thms
Large-ScaleOptimization . . . . . . . . . . . . . . 3-2
Trust Region Methods for Nonlinear Minimization . . 3-3
Preconditioned ConjugateGradients . . . . . . . . . 3-6
Linearly Constrained Problems . . . . . . . . . . . 3-8
Li near Equal i ty Constrai nts . . . . . . . . . . . . . . 3-8
Box Constrai nts . . . . . . . . . . . . . . . . . . . 3-8
Nonlinear Least-Squares . . . . . . . . . . . . . . 3-11
Quadratic Programming . . . . . . . . . . . . . . 3-12
Linear Least-Squares . . . . . . . . . . . . . . . . 3-13
Large-ScaleLinear Programming . . . . . . . . . . 3-14
Mai n Al gori thm . . . . . . . . . . . . . . . . . . . 3-14
Preprocessi ng . . . . . . . . . . . . . . . . . . . . 3-17
References . . . . . . . . . . . . . . . . . . . . . 3-18
3 Large-Scale Algorithms
3-2
Large-Scale Optimization
To sol ve l arge-scal e opti mi zati on probl ems, speci al techni ques are needed. Thi s
chapter descri bes the techni ques used i n the Opti mi zati on Tool box for the
l arge-scal e methods, i n parti cul ar trust regi on methods, precondi ti oned
conjugate gradi ents, projecti on methods for equal i ty constrai nts, and refl ecti ve
Newton methods for bound constrai nts. Fi nal l y, we descri be i nteri or-poi nt
techni ques for l i near programmi ng, speci fi cal l y a pri mal -dual Mehrotra
predi ctor-corrector method.
Trust Region Methods for Nonlinear Minimization
3-3
Trust Region Methods for Nonlinear Minimization
Many of the methods used i n the Opti mi zati on Tool box are based on the trust
regi on i dea, a si mpl e yet powerful concept i n opti mi zati on.
To understand the trust regi on approach to opti mi zati on, consi der the
unconstrai ned mi ni mi zati on probl em, mi ni mi ze where the functi on takes
vector arguments and returns scal ars. Suppose we are at a poi nt i n n-space
and we want to i mprove, i .e., move to a poi nt wi th a l ower functi on val ue. The
basi c i dea i s to approxi mate wi th a si mpl er functi on whi ch reasonabl y
refl ects the behavi or of functi on i n a nei ghborhood around the poi nt x. Thi s
nei ghborhood i s the trust region. A tri al step i s computed by mi ni mi zi ng (or
approxi matel y mi ni mi zi ng) over N. Thi s i s the trust regi on subprobl em,
(3-1)
The current poi nt i s updated to be i f otherwi se, the current
poi nt remai ns unchanged and N, the regi on of trust, i s shrunk and the tri al
step computati on i s repeated.
The key questi ons i n defi ni ng a speci fi c trust regi on approach to mi ni mi zi ng
are how to choose and compute the approxi mati on (defi ned at the
current poi nt ), how to choose and modi fy the trust regi on N, and how
accuratel y to sol ve the trust regi on subprobl em. I n thi s secti on we focus on the
unconstrai ned probl em; addi ti onal compl i cati ons due to the presence of
constrai nts on the vari abl es are di scussed i n l ater secti ons.
I n the standard trust regi on method ([2]), the quadrati c approxi mati on i s
defi ned by the fi rst two terms of the Tayl or approxi mati on to at x; the
nei ghborhood i s usual l y spheri cal or el l i psoi dal i n shape. Mathemati cal l y
the trust regi on subprobl em i s typi cal l y stated
(3-2)
where i s the gradi ent of at the current poi nt x, i s the Hessi an matri x
(the symmetri c matri x of second deri vati ves), i s a di agonal scal i ng matri x,
i s a posi ti ve scal ar, and
.
i s the 2-norm. Good al gori thms exi st for sol vi ng
Eq. 3-2 (see [2]); such al gori thms typi cal l y i nvol ve the computati on of a ful l
ei gensystem and a Newton process appl i ed to the secul ar equati on
f x ( ),
x
f q,
f N
s
mi n
s
q s ( )such that s N { }
x s + f x s + ( ) f x ( ); <
f x ( ) q
x
q
f
N
mi n
1
2
---s
T
Hs s
T
g + such that Ds

' ;

g f H
D
3 Large-Scale Algorithms
3-4
Such al gori thms provi de an accurate sol uti on to Eq. 3-2, however they requi re
ti me proporti onal to several factori zati ons of H; therefore, for l arge-scal e
probl ems a di fferent approach i s needed. Several approxi mati on and heuri sti c
strategi es, based on Eq. 3-2, have been proposed i n the l i terature ([2],[10]). The
approxi mati on approach we fol l ow i n the Opti mi zati on Tool box i s to restri ct the
trust regi on subprobl em to a two-di mensi onal subspace ([2],[1]). Once the
subspace has been computed, the work to sol ve Eq. 3-2 i s tri vi al even i f ful l
ei genval ue/ei genvector i nformati on i s needed (si nce i n the subspace, the
probl em i s onl y two-di mensi onal ). The domi nant work has now shi fted to the
determi nati on of the subspace.
The two-di mensi onal subspace i s determi ned wi th the ai d of a
precondi ti oned conjugate gradi ent process descri bed bel ow. The tool box
assi gns where i s i n the di recti on of the gradi ent g, and i s
ei ther an approxi mate Newton di recti on, i .e., a sol uti on to
(3-3)
or a di recti on of negati ve curvature
(3-4)
The phi l osophy behi nd thi s choi ce of i s to force gl obal convergence (vi a the
steepest descent di recti on or negati ve curvature di recti on) and achi eve fast
l ocal convergence (vi a the Newton step, when i t exi sts).
A framework for the Opti mi zati on Tool box approach to unconstrai ned
mi ni mi zati on usi ng trust regi on i deas i s now easy to descri be:
Formul ate the two-di mensi onal trust regi on subprobl em
Sol ve Eq. 3-2 to determi ne the tri al step
I f then
Adjust
These four steps are repeated unti l convergence. The trust regi on di mensi on
i s adjusted accordi ng to standard rul es. I n parti cul ar, i t i s decreased i f the tri al
step i s not accepted, i .e., See [9],[6] for a di scussi on of thi s
aspect.
1

---
1
s
------- 0 =
S
S
S
S s
1
s
2
, = s
1
s
2
H s
2
g =
s
2
T
H s
2
0 <
S
s
f x s + ( ) f x ( ) ( ) x x s + =

f x s + ( ) f x ( ).
Trust Region Methods for Nonlinear Minimization
3-5
The Opti mi zati on Tool box treats a few i mportant speci al cases of f wi th
speci al i zed functi ons: nonl i near l east-squares, quadrati c functi ons, and l i near
l east-squares. However, the underl yi ng al gori thmi c i deas are the same as for
the general case. These speci al cases are di scussed i n l ater secti ons.
3 Large-Scale Algorithms
3-6
Preconditioned Conjugate Gradients
A popul ar way to sol ve l arge symmetri c posi ti ve defi ni te systems of l i near
equati ons i s the method of precondi ti oned conjugate gradi ents
(PCG). Thi s i terati ve approach requi res the abi l i ty to perform matri x-vector
products of the form where i s an arbi trary vector. The matri x posi ti ve
defi ni te matri x M i s a preconditioner for H, that i s, where
i s wel l -condi ti oned matri x or a matri x wi th cl ustered ei genval ues.
Algorithm PCG
% Initializations
r = g; p = zeros(n,1);
% Precondition
z = M\r; inner1 = r'*z; inner2 = 0; d = z;
% Conjugate gradient iteration
for k = 1:kmax
if k > 1
beta = inner1/inner2;
d = z + beta*d;
end
w = H*d; denom = d'*w;
if denom <= 0
p = d/norm(d); % Direction of negative/zero curvature
break % Exit if zero/negative curvature detected
else
alpha = inner1/denom;
p = p + alpha*d;
r = r alpha*w;
end
z = M\r;
if norm(z) <= tol % Exit if Hp=g solved within tolerance
break
end
inner2 = inner1;
inner1 = r'*z;
end
I n a mi ni mi zati on context you can assume that the Hessi an matri x i s
symmetri c. However, i s guaranteed to be posi ti ve defi ni te onl y i n the
nei ghborhood of a strong mi ni mi zer. Al gori thm PCG exi ts when a di recti on of
Hp g =
H v v
M C
2
= C
1
HC
1
H
H
Preconditioned Conjugate Gradients
3-7
negati ve (or zero) curvature i s encountered, i .e., . The PCG output
di recti on, p, i s ei ther a di recti on of negati ve curvature or an approxi mate (tol
contr ol s how appr oxi mate) sol uti on to the Newton system I n ei ther
case i s used to hel p defi ne the two-di mensi onal subspace used i n the trust
regi on approach di scussed i n the secti on Trust Regi on Methods for Nonl i near
Mi ni mi zati on.
d
T
Hd 0
Hp g. =
p
3 Large-Scale Algorithms
3-8
Linearly Constrained Problems
Li near constrai nts compl i cate the si tuati on descri bed for unconstrai ned
mi ni mi zati on; however, the underl yi ng i deas descri bed above can be carri ed
through i n a cl ean and effi ci ent way. The l arge-scal e methods i n the
Opti mi zati on Tool box gener ate stri ctl y feasi bl e i ter ates: a pr ojecti on techni que
i s used for l i near equal i ty constrai nts, refl ecti ons are used wi th si mpl e box
constrai nts.
Linear Equality Constraints
The general l i near equal i ty constrai ned mi ni mi zati on probl em can be wri tten
(3-5)
where i s an m-by-n matri x ( ). The Opti mi zati on Tool box preprocesses
to remove stri ct l i near dependenci es usi ng a techni que based on the
LU-factori zati on of ([6]). Here we wi l l assume i s of rank m.
Our method to sol ve Eq. 3-5 di ffers from our unconstrai ned approach i n two
si gni fi cant ways. Fi rst, an i ni ti al feasi bl e poi nt i s computed, usi ng a sparse
l east-squares step, so that . Second, Al gori thm PCG i s repl aced wi th
Reduced Precondi ti oned Conjugate Gradi ents (RPCG), see [6], i n order to
compute an approxi mate reduced Newton step (or a di recti on of negati ve
curvature i n the nul l space of ). The key l i near al gebra step i nvol ves sol vi ng
systems of the form
(3-6)
where approxi mates (smal l nonzeros of are set to zero provi ded rank i s
not l ost) and i s a sparse symmetri c posi ti ve-defi ni te approxi mati on to H, i .e.,
. See [6] for more detai l s.
Box Constraints
The box constrai ned probl em i s of the form
(3-7)
mi n f x ( ) such that Ax b = { }
A m n
A
A
T
A
x
0
Ax
0
b =
A
C A

T
A

0
s
t
r
0
=
A

A A
C
C
T
C H
mi n f x ( ) such that l x u { }
Linearly Constrained Problems
3-9
where l i s a vector of l ower bounds, and u i s a vector of upper bounds. Some (or
al l ) of the components of may be equal to and some (or al l ) of the
components of may be equal to The method gener ates a sequence of
stri ctl y feasi bl e poi nts. Two techni ques are used to mai ntai n feasi bi l i ty whi l e
achi evi ng robust convergence behavi or. Fi rst, a scal ed modi fi ed Newton step
repl aces the unconstrai ned Newton step (to defi ne the two-di mensi onal
subspace ). Second, refl ecti ons are used to i ncrease the stepsi ze.
The scal ed modi fi ed Newton step ari ses from exami ni ng the Kuhn-Tucker
necessary condi ti ons for Eq. 3-7
(3-8)
where
and the vector i s defi ned bel ow, for each
I f and then
I f and then
I f and then
I f and then
The nonl i near system Eq. 3-8 i s not di fferenti abl e everywhere;
nondi fferenti abi l i ty occurs when Hence we avoi d such poi nts by
mai ntai ni ng stri ct feasi bi l i ty, i .e., restri cti ng
The scal ed modi fi ed Newton step for Eq. 3-8 i s defi ned as the sol uti on to the
l i near system
(3-9)
where
(3-10)
and
l
u .
S
D x ( ) ( )
2
g 0 =
D x ( ) di ag v
k
1
2
---
,

_
=
v x ( ) 1 i n:
g
i
0 < u
i
< v
i
x
i
u
i
=
g
i
0 l
i
> v
i
x
i
l
i
=
g
i
0 < u
i
= v
i
1 =
g
i
0 l
i
= v
i
1 =
v
i
0. =
l x u. < <
s
k
N
M

Ds
N
g =
g D
1
g di ag v
1
2
---
,

_
g = =
3 Large-Scale Algorithms
3-10
(3-11)
Here pl ays the rol e of the Jacobi an of Each di agonal component of the
di agonal matri x equal s 0, -1 or 1. I f al l the components of l and u are fi ni te,
At a poi nt where may not be di fferenti abl e. We
defi ne at such a poi nt. Nondi fferenti abi l i ty of thi s type i s not a cause
for concern because, for such a component, i t i s not si gni fi cant whi ch val ue
takes. Further wi l l sti l l be di sconti nuous at thi s poi nt, but the functi on
i s conti nuous.
Second, refl ecti ons are used to i ncrease the stepsi ze. A (si ngl e) refl ecti on step
i s defi ned as fol l ows. Gi ven a step that i ntersects a bound constrai nt,
consi der the fi rst bound constrai nt crossed by p; assume i t i s the ith bound
constrai nt (ei ther the ith upper or ith l ower bound). Then the refl ecti on step
except i n the ith component where
M

D
1
HD
1
di ag(g )J
v
+ =
J
v
v .
J
v
J
v
di ag si gn g ( ) ( ). = g
i
0, = v
i
J
i i
v
0 =
v
i
v
i
v
i
g
i

p
p
R
p = p
i
R
p
i
. =
Nonlinear Least-Squares
3-11
Nonlinear Least-Squares
An i mportant speci al case for f(x) i s the nonl i near l east-squares probl em
(3-12)
where i s a vector-val ued functi on wi th component i of equal to
The basi c method used to sol ve thi s probl em i s the same as i n the general case
descri bed i n Trust Regi on Methods for Nonl i near Mi ni mi zati on. However,
the structure of the nonl i near l east-squares probl em i s expl oi ted to enhance
effi ci ency. I n parti cul ar, an approxi mate Gauss-Newton di recti on, i .e., a
sol uti on s to
(3-13)
(where J i s the Jacobi an of ) i s used to hel p defi ne the two-di mensi onal
subspace . Second deri vati ves of the component functi on are not used.
I n each i terati on the method of precondi ti oned conjugate gradi ents i s used to
approxi matel y sol ve the normal equati ons, i .e.,
al though the normal equati ons are not expl i ci tl y formed.
f x ( )
1
2
--- f
i
2
x ( )
i

1
2
--- F x ( )
2
2
= =
F x ( ) F x ( ) f
i
x ( ).
mi n J s F +
2
2
F x ( )
S f
i
x ( )
J
T
J s J
T
F =
3 Large-Scale Algorithms
3-12
Quadratic Programming
I n thi s case the functi on f(x) i s the quadrati c equati on
The subspace trust regi on method i s used to determi ne a search di recti on.
However, i nstead of restri cti ng the step to (possi bl y) one refl ecti on step as i n
the nonl i near mi ni mi zati on case, a pi ecewi se refl ecti ve l i ne search i s conducted
at each i terati on. See [5] for detai l s of the l i ne search.
q x ( )
1
2
---x
T
Hx f
T
x + =
Linear Least-Squares
3-13
Linear Least-Squares
I n thi s case the functi on f(x) to be sol ved i s
The al gori thm generates stri ctl y feasi bl e i terates convergi ng, i n the l i mi t, to a
l ocal sol uti on. Each i terati on i nvol ves the approxi mate sol uti on of a l arge
l i near system (of order n, where ni s the l ength of x); the i terati on matri ces have
the structure of the matri x C. I n parti cul ar, the method of precondi ti oned
conjugate gradi ents i s used to approxi matel y sol ve the normal equati ons, i .e,
al though the normal equati ons are not expl i ci tl y formed.
The subspace trust regi on method i s used to determi ne a search di recti on.
However, i nstead of restri cti ng the step to (possi bl y) one refl ecti on step as i n
the nonl i near mi ni mi zati on case, a pi ecewi se refl ecti ve l i ne search i s conducted
at each i terati on, as i n the quadrati c case. See [5] for detai l s of the l i ne search.
Ul ti matel y, the l i near systems represent a Newton approach capturi ng the
fi rst-order opti mal i ty condi ti ons at the sol uti on: resul ti ng i n strong l ocal
convergence rates.
f x ( )
1
2
---x
T
C
T
Cx x
T
C
T
d + =
C
T
Cx C
T
d =
3 Large-Scale Algorithms
3-14
Large-Scale Linear Programming
Li near programmi ng i s defi ned as
(3-14)
The l arge-scal e method i s based on LI PSOL ([11]), whi ch i s a vari ant of
Mehrotras predi ctor-corrector al gori thm ([7]), a pri mal -dual i nteri or-poi nt
method.
Main Algorithm
The al gori thm begi ns by appl yi ng a seri es of preprocessi ng steps (see
Preprocessi ng on page 3-17). After preprocessi ng, the probl em has the form
(3-15)
The upper bounds constrai nts are i mpl i ci tl y i ncl uded i n the constrai nt matri x
A wi th the addi ti on of pri mal sl ack vari abl es s, Eq. 3-15 becomes
(3-16)
whi ch i s referred to as the primal probl em: xconsi sts of the pri mal vari abl es
and s consi sts of the pri mal sl ack vari abl es. The dual probl em i s
(3-17)
where yand wconsi st of the dual vari abl es and z consi sts of the dual sl acks.
The opti mal i ty condi ti ons for thi s l i near program, i .e., the pri mal Eq. 3-16 and
the dual Eq. 3-17, are
mi n f
T
x such that
Aeq x beq =
Ai neq x bi neq
l x u ,


_
mi n f
T
x such that
A x b =
0 x u
,
_
mi n f
T
x such that
A x b =
x s + u =
x 0 s 0 ,
,


_
max b
T
y u
T
w such that
A
T
y w z + f =
z 0 w 0 ,
Large-Scale Linear Programming
3-15
(3-18)
where and denote component-wi se mul ti pl i cati on.
The quadrati c equati ons and are cal l ed the
complementaritycondi ti ons for the l i near program; the other (l i near)equati ons
are cal l ed the feasibilitycondi ti ons. The quanti ty
i s the dualitygap, whi ch measures the resi dual of the compl ementari ty porti on
of F when .
The al gori thm i s a primal-dual algorithmmeani ng that both the pri mal and
the dual programs are sol ved si mul taneousl y. I t can be consi dered a
Newton-l i ke method, appl i ed to the l i near-quadrati c system
i n Eq. 3-18, whi l e at the same ti me keepi ng the i terates x,
z, w, and s posi ti ve, thus the name i nteri or-poi nt method. (The i terates are i n
the stri ctl y i nteri or regi on represented by the i nequal i ty constrai nts i n Eq.
3-16.)
The al gori thm i s a vari ant of the predi ctor-corrector al gori thm proposed by
Mehrotra. Consi der an i terate , where
We fi rst compute the so-cal l ed prediction di recti on
whi ch i s the Newton di recti on; then the so-cal l ed corrector di recti on
where i s cal l ed the centeringparameter and must be chosen careful l y.
i s a zero-one vector wi th the ones correspondi ng to the quadrati c equati ons i n
F(v), i .e., the perturbati ons are onl y appl i ed to the compl ementari ty condi ti ons,
F x y z s w , , , , ( )
A x b
x s u +
A
T
y w z f +
x
i
z
i
s
i
w
i ,







_
0 = =
x 0 z 0 s 0 w 0 , , ,
x
i
z
i
s
i
w
i
x
i
z
i
0 = s
i
w
i
0 =
x
T
z s
T
w +
x z s w , , , ( ) 0
F x y z s w , , , , ( ) 0 =
v x y z s w ; ; ; ; [ ] = x z s w ; ; ; [ ] 0. >
v
p
F
T
v ( ) ( )
1
F v ( ) =
v
c
F
T
v ( ) ( )
1
F v v
p
+ ( ) ( ) e =
0 > e
3 Large-Scale Algorithms
3-16
whi ch are al l quadrati c, but not to the feasi bi l i ty condi ti ons, whi ch are al l
l i near. We combi ne the two di recti ons wi th a step-l ength parameter and
update v to obtai n the new i terate
where the step-l ength parameter i s chosen so that
sati sfi es
I n sol vi ng for the steps above, the al gori thm computes a (sparse) di rect
factori zati on on a modi fi cati on of the Chol esky factors of I f A has dense
col umns, i t i nstead uses the Sherman-Morri son formul a, and i f that sol uti on i s
not adequate (the resi dual i s too l arge), i t uses precondi ti oned conjugate
gradi ents to fi nd a sol uti on.
The al gori thm then repeats these steps unti l the i terates converge. The mai n
stoppi ng cri teri a i s a standard one
where
are the pri mal resi dual , dual resi dual , and upper-bound feasi bi l i ty
respecti vel y, and
i s the di fference between the pri mal and dual objecti ve val ues, and tol i s some
tol erance. The sum i n the stoppi ng cri teri a measures the total rel ati ve errors
i n the opti mal i ty condi ti ons i n Eq. 3-18.
0 >
v
+
v
+
v v
p
v
c
+ ( ) + =

v
+
x
+
y
+
z
+
s
+
w
+
; ; ; ; [ ] =
x
+
z
+
s
+
w
+
; ; ; [ ] 0 >
A A
T
.
r
b
1 b , ( ) max
------------------------------
r
f
1 f , ( ) max
-----------------------------
r
u
1 u , ( ) max
------------------------------
f
T
x b
T
y u
T
w +
1 f
T
x b
T
y u
T
w , , ( ) max
-------------------------------------------------------------------- + + + tol
r
b
Ax b =
r
f
A
T
y w z f + =
r
u
x s u + =
f
T
x b
T
y u
T
w +
Large-Scale Linear Programming
3-17
Preprocessing
A number of preprocessi ng steps occur before the actual i terati ve al gori thm
begi ns. The resul ti ng transformed probl em i s one where:
Al l vari abl es are bounded bel ow by zero.
Al l constrai nts are equal i ti es.
Fi xed vari abl es, those wi th equal upper and l ower bounds, are removed.
Rows of al l zeros i n the constrai nt matri x are removed.
The constrai nt matri x has ful l structural rank.
Col umns of al l zeros i n the constrai nt matri x are removed.
When a si gni fi cant number of si ngl eton rows exi st i n the constrai nt matri x,
the associ ated vari abl es are sol ved for and the rows removed.
Whi l e these preprocessi ng steps can do much to speed up the i terati ve part of
the al gori thm, i f the Lagrange mul ti pl i ers are requi red, the preprocessi ng
steps must be undone si nce the mul ti pl i ers cal cul ated duri ng the al gori thm
are for the transformed, and not the ori gi nal , probl em. Thus, i f the mul ti pl i ers
are not requested, thi s transformati on back wi l l not be computed, and may
save some ti me computati onal l y.
3 Large-Scale Algorithms
3-18
References
[1] Branch, M.A., T.F. Col eman, Y. Li , A Subspace, I nteri or, and Conjugate
Gradi ent Method for Large-Scal e Bound-Constrai ned Mi ni mi zati on
Probl ems, to appear i n a future SI AM J ournal on Scientific Computing.
[2] Byrd, R.H., R.B. Schnabel , and G.A. Shul tz, Approxi mate Sol uti on of the
Trust Regi on Probl em by Mi ni mi zati on over Two-Di mensi onal
Subspaces, Mathematical Programming, Vol . 40, pp. 247-263, 1988.
[3] Col eman, T.F. and Y. Li , On the Convergence of Refl ecti ve Newton
Methods for Large-Scal e Nonl i near Mi ni mi zati on Subject to Bounds,
Mathematical Programming, Vol . 67, Number 2, pp. 189-224, 1994.
[4] Col eman, T.F. and Y. Li , An I nteri or, Trust Regi on Approach for
Nonl i near Mi ni mi zati on Subject to Bounds, SI AM J ournal on
Optimization, Vol . 6, pp. 418-445, 1996.
[5] Col eman, T.F. and Y. Li , A Refl ecti ve Newton Method for Mi ni mi zi ng a
Quadrati c Functi on Subject to Bounds on some of the Vari abl es, SI AM
J ournal on Optimization, Vol . 6, Number 4, pp. 1040-1058, 1996.
[6] Col eman, T.F. and A. Verma, On Precondi ti oni ng Conjugate Gradi ents
for Li near Equal i ty Constrai ned Mi ni mi zati on, submi tted to SI AM
J ournal on Scientific Computing.
[7] Mehrotra, S., On the I mpl ementati on of a Pri mal -Dual I nteri or Poi nt
Method, SI AM J ournal on Optimization, Vol . 2, pp. 575-601, 1992.
[8] Mor, J.J. and D.C. Sorensen, Computi ng a Trust Regi on Step, SI AM
J ournal on Scientific and Statistical Computing, Vol . 3, pp. 553-572,
1983.
[9] Sorensen, D.C., Mi ni mi zati on of a Large Scal e Quadrati c Functi on
Subject to an El l i psoi dal Constrai nt, Department of Computational and
Applied Mathematics, Ri ce Uni versi ty, Techni cal Report TR94-27, 1994.
[10] Stei haug, T., The Conjugate Gradi ent Method and Trust Regi ons i n
Large Scal e Opti mi zati on, SI AM J ournal on Numerical Analysis, Vol .
20, pp. 626-637, 1983.
[11] Zhang, Y., Sol vi ng Large-Scal e Li near Programs by I nteri or-Poi nt
Methods Under the MATLAB Envi ronment, Department of
Mathemati cs and Stati sti cs, Uni versi ty of Maryl and, Bal ti more County,
Bal ti more, MD, Techni cal Report TR96-01, Jul y, 1995.
4
Reference
4 Reference
4-2
Thi s chapter contai ns descri pti ons of the Opti mi zati on Tool box functi ons,
l i sted al phabeti cal l y. The chapter starts wi th tabl es l i sti ng the types of
opti mi zati on and whi ch functi ons appl y. The next set of tabl es l i sts general
descri pti ons of al l the i nput and output arguments and the parameters i n the
opti mi zati on opti ons structure, and whi ch functi ons use those arguments or
parameters. I nformati on speci fi c to a functi on about i nput arguments, output
arguments, and opti ons i s l i sted wi thi n that functi ons reference pages under
the Arguments secti on, after the functi on descri pti on.
I nformati on i s al so avai l abl e through the onl i ne Hel p faci l i ty.
Optimization Functions
4-3
Optimization Functions
Minimization
Equation Solving
Function Purpose
fgoalattain Mul ti objecti ve goal attai nment
fminbnd Scal ar nonl i near mi ni mi zati on wi th bounds
fmincon Constrai ned nonl i near mi ni mi zati on
fminimax Mi ni max opti mi zati on
fminsearch,
fminunc
Unconstrai ned nonl i near mi ni mi zati on
fseminf Semi -i nfi ni te mi ni mi zati on
linprog Li near programmi ng
quadprog Quadrati c programmi ng
Function Purpose
\ Li near equati on sol vi ng (see the onl i ne MATLAB
Function Referencegui de)
fsolve Nonl i near equati on sol vi ng
fzero Scal ar nonl i near equati on sol vi ng
4 Reference
4-4
Least-Squares (Curve Fitting)
Utility
Demonstrations of Large-Scale Methods
Function Purpose
\ Li near l east squares (see the onl i ne MATLAB
Function Referencegui de)
lsqlin Constrai ned l i near l east squares
lsqcurvefit Nonl i near curve fi tti ng
lsqnonlin Nonl i near l east squares
lsqnonneg Nonnegati ve l i near l east squares
Function Purpose
optimset,
optimget
Parameter setti ng
Function Purpose
circustent Quadrati c programmi ng to fi nd shape of a ci rcus
tent
molecule Mol ecul e conformati on sol uti on usi ng
unconstrai ned nonl i near mi ni mi zati on
optdeblur I mage debl urri ng usi ng bounded l i near
l east-squares
Optimization Functions
4-5
Demonstrations of Medium-Scale Methods
Function Purpose
bandemo Mi ni mi zati on of the banana functi on
dfildemo Fi ni te-preci si on fi l ter desi gn (requi res Signal
ProcessingToolbox)
goaldemo Goal attai nment exampl e
optdemo Menu of demonstrati on routi nes
tutdemo Tutori al wal k-through
4 Reference
4-6
Function Arguments
These tabl es descri be arguments used by Opti mi zati on Tool box functi ons: the
fi rst descri bes i nput arguments, the second descri bes the output arguments,
and the thi rd descri bes the opti mi zati on opti ons parameters structure
options.
Thi s tabl e summari zes the i nput arguments. Not al l functi ons use al l
arguments. See the i ndi vi dual reference pages for functi on-speci fi c detai l s
about these arguments.
Table 4-1: Input Arguments
Argument Description Used by Functions
A, b The matri x A and vector b are, respecti vel y, the coeffi ci ents
of the l i near i nequal i ty constrai nts and the correspondi ng
ri ght-hand si de vector: A
.
x<=b.
fgoalattain,
fmincon, fminimax,
fseminf, linprog,
lsqlin, quadprog
Aeq, beq The matri x Aeq and vector beq are, respecti vel y, the
coeffi ci ents of the l i near equal i ty constrai nts and the
correspondi ng ri ght-hand si de vector: Aeq
.
x=beq.
fgoalattain,
fmincon, fminimax,
fseminf, linprog,
lsqlin, quadprog
C, d The matri x C and vector d are, respecti vel y, the coeffi ci ents
of the over- or under-determi ned l i near system and the
ri ght-hand-si de vector to be sol ved.
lsqlin, lsqnonneg
f The vector of coeffi ci ents for the l i near ter m i n the l i near
equati on f'*x or the quadrati c equati on x'*H*x+f'*x.
linprog, quadprog
fun The functi on to be opti mi zed. fun may be an i nl i ne object,
or the name of an M-fi l e, bui l t-i n functi on, or MEX-fi l e. See
the i ndi vi dual functi on reference pages for more
i nformati on on fun.
fgoalattain,
fminbnd, fmincon,
fminimax,
fminsearch,
fminunc, fseminf,
fsolve, fzero,
lsqcurvefit,
lsqnonlin
Function Arguments
4-7
goal Vector of val ues that the objecti ves attempt to attai n. The
vector i s the same l ength as the number of objecti ves.
fgoalattain
H The matri x of coeffi ci ents for the quadrati c terms i n the
quadrati c equati on x'*H*x+f'*x. H must be symmetri c.
quadprog
lb, ub Lower and upper bound vectors (or matri ces). The
arguments are normal l y the same si ze as x. However, i f lb
has fewer el ements than x, say onl y m, then onl y the fi rst m
el ements i n x are bounded bel ow; upper bounds i n ub can
be defi ned i n the same manner. Unbounded vari abl es may
al so be speci fi ed usi ng Inf (for l ower bounds) or Inf (for
upper bounds). For exampl e, i f lb(i) = Inf then the
vari abl e x(i) i s unbounded bel ow.
fgoalattain,
fmincon, fminimax,
fseminf, linprog,
lsqcurvefit,
lsqlin, lsqnonlin,
quadprog
nonlcon The functi on that computes the nonl i near i nequal i ty and
equal i ty constrai nts. nonlcon i s the name of an M-fi l e or
MEX-fi l e. See the i ndi vi dual reference pages for more
i nformati on on nonlcon.
fgoalattain,
fmincon, fminimax
ntheta The number of semi -i nfi ni te constrai nts. fseminf
options An opti mi zati on opti ons parameter structure that defi nes
parameters used by the opti mi zati on functi ons. For
i nformati on about the parameters, see Tabl e 4-3 or the
i ndi vi dual functi on reference pages.
Al l functi ons
Table 4-1: Input Arguments (Continued)
Argument Description Used by Functions
4 Reference
4-8
P1, P2,... Addi ti onal arguments to be passed to fun, nonlcon (i f i t
exi sts), or seminfcon (i f i t exi sts), that i s, when the
opti mi zati on functi on cal l s the functi ons fun, nonlcon, or
seminfcon, the cal l s are
f = feval(fun,x,P1,P2,...)
[c, ceq] = feval(nonlcon,x,P1,P2,...)
[c,ceq,K1,K2,...,Kn,s]= ...
feval(seminfcon,x,s,P1,P2,...)
Usi ng thi s feature, the same fun (or nonlcon or seminfcon)
can sol ve a number of si mi l ar probl ems wi th di fferent
parameters, avoi di ng the need to use gl obal vari abl es.
fgoalattain,
fminbnd, fmincon,
fminimax,
fminsearch,
fminunc, fseminf,
fsolve, fzero,
lsqcurvefit,
lsqnonlin
seminfcon The functi on that computes the nonl i near i nequal i ty and
equal i ty constrai nts and the semi -i nfi ni te constrai nts.
seminfcon i s the name of an M-fi l e or MEX-fi l e. See the
i ndi vi dual functi on reference pages for fseminf more
i nformati on on seminfcon.
fseminf
weight A wei ghti ng vector to control the rel ati ve under-attai nment
or over-attai nment of the objecti ves.
fgoalattain
xdata,
ydata
The i nput data xdata and the observed output data ydata
that i s to be fi t to an equati on.
lsqcurvefit
x0 Starti ng poi nt (a scal ar, vector or matri x).
(For fzero, x0 can al so be a two-el ement vector
representi ng an i nterval that i s known to contai n a zero.)
Al l functi ons except
fminbnd
x1, x2 The i nterval over whi ch the functi on i s mi ni mi zed. fminbnd
Table 4-1: Input Arguments (Continued)
Argument Description Used by Functions
Function Arguments
4-9
Table 4-2: Output Arguments
Argument Description Used by Functions
attainfactor The attai nment factor at the sol uti on x. fgoalattain
exitflag The exi t condi ti on. For the meani ng of a parti cul ar
val ue, see the functi on reference pages.
Al l functi ons
fval The val ue of the objecti ve functi on fun at the sol uti on x. fgoalattain,
fminbnd, fmincon,
fminimax,
fminsearch,
fminunc, fseminf,
fsolve, fzero,
linprog, quadprog
grad The val ue of the gradi ent of fun at the sol uti on x. fmincon, fminunc
hessian The val ue of the Hessi an of fun at the sol uti on x. fmincon, fminunc
jacobian The val ue of the Jacobi an of fun at the sol uti on x. lsqcurvefit,
lsqnonlin, fsolve
lambda The Lagrange mul ti pl i ers at the sol uti on x. lambda i s a
structure where each fi el d i s for a di fferent constrai nt
type. For structure fi el d names, see i ndi vi dual functi on
descri pti ons. (For lsqnonneg, lambda i s si mpl y a vector
as lsqnonneg onl y handl es one ki nd of constrai nt.)
fgoalattain,
fmincon,
fminimax,
fseminf, linprog,
lsqcurvefit,
lsqlin,
lsqnonlin,
lsqnonneg,
quadprog
maxfval max{fun(x)} at the sol uti on x. fminimax
output An output structure that contai ns i nformati on about
the resul ts of the opti mi zati on. For structure fi el d
names, see i ndi vi dual functi on descri pti ons.
Al l functi ons
4 Reference
4-10
Thi s tabl e descri bes fi el ds i n the opti mi zati on parameters structure, options.
The col umn l abel ed L, M, B provi des thi s i nformati on about each parameter:
L The parameter onl y appl i es to l arge-scal e methods.
M The parameter onl y appl i es to medi um-scal e methods.
B The parameter appl i es to both l arge- and medi um-scal e methods.
residual The val ue of the resi dual at the sol uti on x. lsqcurvefit,
lsqlin,
lsqnonlin,
lsqnonneg
resnorm The val ue of the squared 2-norm of the resi dual at the
sol uti on x.
lsqcurvefit,
lsqlin,
lsqnonlin,
lsqnonneg
x The sol uti on found by the opti mi zati on functi on. I f
exitflag > 0 then x i s a sol uti on; otherwi se, x i s the
val ue the opti mi zati on routi ne was at when i t
termi nated prematurel y.
Al l functi ons
Table 4-2: Output Arguments (Continued)
Argument Description Used by Functions
Table 4-3: Optimization Options Parameters
Parameter Name Description L, M, B Used by Functions
DerivativeCheck Compare user-suppl i ed anal yti c
deri vati ves (gradi ents or
Jacobi an) to fi ni te di fferenci ng
deri vati ves.
M fgoalattain, fmincon,
fminimax, fminunc,
fseminf, fsolve,
lsqcurvefit, lsqnonlin
Diagnostics Pri nt di agnosti c i nformati on
about the functi on to be
mi ni mi zed or sol ved.
B Al l but fminbnd,
fminsearch, fzero, and
lsqnonneg
Function Arguments
4-11
DiffMaxChange Maxi mum change i n vari abl es
for fi ni te di fference deri vati ves.
M fgoalattain, fmincon,
fminimax, fminunc,
fseminf, fsolve,
lsqcurvefit, lsqnonlin
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te di fference deri vati ves.
M fgoalattain, fmincon,
fminimax, fminunc,
fseminf, fsolve,
lsqcurvefit, lsqnonlin
Display Level of di spl ay. 'off' di spl ays
no output; 'iter' di spl ays
output at each i terati on; 'final'
di spl ays just the fi nal output.
B Al l
GoalsExactAchieve Number of goal s to achi eve
exactl y (do not over- or
underachi eve).
M fgoalattain
GradConstr Gradi ents for the nonl i near
constrai nts defi ned by user.
M fgoalattain, fmincon,
fminimax
GradObj Gradi ent(s) for the objecti ve
functi on(s) defi ned by user.
B fgoalattain, fmincon,
fminimax, fminunc,
fseminf
Hessian Hessi an for the objecti ve
functi on defi ned by user.
L fmincon, fminunc
HessPattern Sparsi ty pattern of the Hessi an
for fi ni te di fferenci ng.
L fmincon, fminunc
HessUpdate Quasi -Newton updati ng scheme. M fminunc
Jacobian Jacobi an for the objecti ve
functi on defi ned by user.
B fsolve, lsqcurvefit,
lsqnonlin
JacobPattern Sparsi ty pattern of the Jacobi an
for fi ni te di fferenci ng.
L fsolve, lsqcurvefit,
lsqnonlin
Table 4-3: Optimization Options Parameters (Continued)
Parameter Name Description L, M, B Used by Functions
4 Reference
4-12
LargeScale Use l arge-scal e al gori thm i f
possi bl e.
B fmincon, fminunc,
fsolve, linprog,
lsqcurvefit, lsqlin,
lsqnonlin, quadprog
LevenbergMarquardt Chooses Levenberg-Marquardt
over Gauss-Newton al gori thm.
M fsolve, lsqcurvefit,
lsqnonlin
LineSearchType Li ne search al gori thm choi ce. M fminunc, fsolve,
lsqcurvefit, lsqnonlin
MaxFunEvals Maxi mum number of functi on
eval uati ons al l owed.
B fgoalattain, fminbnd,
fmincon, fminimax,
fminsearch, fminunc,
fseminf, fsolve,
lsqcurvefit, lsqnonlin
MaxIter Maxi mum number of i terati ons
al l owed.
B Al l but fzero and
lsqnonneg
MaxPCGIter Maxi mum number of PCG
i terati ons al l owed.
L fmincon, fminunc,
fsolve, lsqcurvefit,
lsqlin, lsqnonlin,
quadprog
MeritFunction Use goal attai nment/mi ni max
meri t functi on (mul ti objecti ve)
vs. fmincon (si ngl e objecti ve).
M fgoalattain, fminimax
MinAbsMax Number of F(x) to mi ni mi ze the
worst case absol ute val ues
M fminimax
PrecondBandWidth Upper bandwi dth of
precondi ti oner for PCG.
L fmincon, fminunc,
fsolve, lsqcurvefit,
lsqlin, lsqnonlin,
quadprog
Table 4-3: Optimization Options Parameters (Continued)
Parameter Name Description L, M, B Used by Functions
Function Arguments
4-13
TolCon Termi nati on tol erance on the
constrai nt vi ol ati on.
B fgoalattain, fmincon,
fminimax, fseminf
TolFun Termi nati on tol erance on the
functi on val ue.
B Al l but fminbnd, fzero,
and lsqnonneg
TolPCG Termi nati on tol erance on the
PCG i terati on.
L fmincon, fminunc,
fsolve, lsqcurvefit,
lsqlin, lsqnonlin,
quadprog
TolX Termi nati on tol erance on x. B Al l but linprog and
lsqlin
TypicalX Typi cal x val ues. L fmincon, fminunc,
fsolve, lsqcurvefit,
lsqlin, lsqnonlin,
quadprog
Table 4-3: Optimization Options Parameters (Continued)
Parameter Name Description L, M, B Used by Functions
fgoalattain
4-14
4fgoal attai n
Purpose Sol ve mul ti objecti ve goal attai nment probl em
where x, weight, goal, b, beq, lb, and ub are vectors, A and Aeq are matri ces,
c(x), ceq(x), and F(x) are functi ons that return vectors. F(x), c(x), and ceq(x) can
be nonl i near functi ons.
Syntax x = fgoalattain(fun,x0,goal,weight)
x = fgoalattain(fun,x0,goal,weight,A,b)
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq)
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq,lb,ub)
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq,lb,ub,nonlcon)
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq,...
lb,ub,nonlcon,options)
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq,...
lb,ub,nonlcon,options,P1,P2,...)
[x,fval] = fgoalattain(...)
[x,fval,attainfactor] = fgoalattain(...)
[x,fval,attainfactor,exitflag] = fgoalattain(...)
[x,fval,attainfactor,exitflag,output] = fgoalattain(...)
[x,fval,attainfactor,exitflag,output,lambda] = fgoalattain(...)
Description fgoalattain sol ves the goal attai nment probl em, whi ch i s one formul ati on for
mi ni mi zi ng a mul ti objecti ve opti mi zati on probl em.
x = fgoalattain(fun,x0,goal,weight) tri es to make the objecti ve functi ons
suppl i ed by fun attai n the goal s speci fi ed by goal by varyi ng x, starti ng at x0,
wi th wei ght speci fi ed by weight.
x = fgoalattain(fun,x0,goal,weight,A,b) sol ves the goal attai nment
probl em subject to the l i near i nequal i ti es A*x <= b.

x,
minimize such that F x ( ) wei ght

goal
c x ( ) 0
ceq x ( ) 0 =
A x b
Aeq x beq =
l b x ub
fgoalattain
4-15
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq) sol ves the goal
attai nment probl em subject to the l i near equal i ti es Aeq*x = beq as wel l . Set
A=[] and b=[] i f no i nequal i ti es exi st.
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq,lb,ub) defi nes a set of
l ower and upper bounds on the desi gn vari abl es, x, so that the sol uti on i s
al ways i n the range lb <= x <= ub.
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq,lb,ub,nonlcon)
subjects the goal attai nment probl em to the nonl i near i nequal i ti es c(x) or
nonl i near equal i ty constrai nts ceq(x) defi ned i n nonlcon. fgoalattain
opti mi zes such that c(x) <= 0 and ceq(x) = 0. Set lb=[] and/or ub=[] i f no
bounds exi st.
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq,lb,ub,nonlcon,...
options) mi ni mi zes wi th the opti mi zati on parameters speci fi ed i n the
structure options.
x = fgoalattain(fun,x0,goal,weight,A,b,Aeq,beq,lb,ub,nonlcon,...
options,P1,P2,...) passes the probl em-dependent parameters P1, P2, etc.,
di rectl y to the functi ons fun and nonlcon. Pass empty matri ces as pl acehol ders
for A, b, Aeq, beq, lb, ub, nonlcon, and options i f these arguments are not
needed.
[x,fval] = fgoalattain(...) returns the val ues of the objecti ve functi ons
computed i n fun at the sol uti on x.
[x,fval,attainfactor] = fgoalattain(...) returns the attai nment factor
at the sol uti on x.
[x,fval,attainfactor,exitflag] = fgoalattain(...) returns a val ue
exitflag that descri bes the exi t condi ti on of fgoalattain.
[x,fval,attainfactor,exitflag,output] = fgoalattain(...) returns a
structure output that contai ns i nformati on about the opti mi zati on.
[x,fval,attainfactor,exitflag,output,lambda] = fgoalattain(...)
returns a structure lambda whose fi el ds contai n the Lagrange mul ti pl i ers at
the sol uti on x.
fgoalattain
4-16
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to fgoalattain are i ncl uded bel ow for fun, goal, nonlcon, options,
weight, attainfactor, exitflag, lambda, and output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
vector F of the objecti ve functi ons eval uated at x. You can speci fy
fun to be an i nl i ne object. For exampl e,
fun = inline('sin(x.*x)');
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function F = myfun(x)
F = ... % Compute function values at x
To make an objecti ve functi on as near as possi bl e to a goal val ue,
(i .e., nei ther greater than nor l ess than) set
options.GoalsExactAchieve to the number of objecti ves requi red
to be i n the nei ghborhood of the goal val ues. Such objecti ves must
be parti ti oned i nto the fi rst el ements of the vector F returned by
fun.
I f the gradi ent of the objecti ve functi on can al so be computed and
options.GradObj i s 'on', as set by
options = optimset('GradObj','on')
then the functi on fun must return, i n the second output argument,
the gradi ent val ue G, a matri x, at x. Note that by checki ng the val ue
of nargout the functi on can avoi d computi ng G when 'myfun' i s
cal l ed wi th onl y one output argument (i n the case where the
opti mi zati on al gori thm onl y needs the val ue of F but not G):
function [F,G] = myfun(x)
F = ... % compute the function values at x
if nargout > 1 % two output arguments
G = ... % gradients evaluated at x
end
fgoalattain
4-17
The gradi ent i s the parti al deri vati ves dF/ dxof each F at the poi nt
x. I f F i s a vector of l ength m and x has l ength n, then the gradi ent
G of F(x) i s an n-by-m matri x where G(i,j) i s the parti al deri vati ve
of F(j) wi th respect to x(i) (i .e., the jth col umn of G i s the gradi ent
of the jth objecti ve functi on F(j)).
goal Vector of val ues that the objecti ves attempt to attai n. The vector i s
the same l ength as the number of objecti ves F returned by fun.
fgoalattain attempts to mi ni mi ze the val ues i n the vector F to
attai n the goal val ues gi ven by goal.
nonlcon The functi on that computes the nonl i near i nequal i ty constrai nts
c(x) <=0 and nonl i near equal i ty constrai nts ceq(x)=0. nonlcon i s
a stri ng contai ni ng the name of a functi on (an M-fi l e, a bui l t-i n, or
a MEX-fi l e). nonlcon takes a vector x and returns two arguments,
a vector c of the nonl i near i nequal i ti es eval uated at x and a vector
ceq of the nonl i near equal i ti es eval uated at x. For exampl e, i f
nonlcon=mycon then the M-fi l e mycon.m woul d have the form
function [c,ceq] = mycon(x)
c = ... % Compute nonlinear inequalities at x
ceq = ... % Compute the nonlinear equalities at x
I f the gradi ents of the constrai nts can al so be computed and
options.GradConstr i s 'on', as set by
options = optimset('GradConstr','on')
then the functi on nonlcon must al so return, i n the thi rd and fourth
output arguments, GC, the gradi ent of c(x), and GCeq, the gradi ent
of ceq(x). Note that by checki ng the val ue of nargout the functi on
can avoi d computi ng GC and GCeq when nonlcon i s cal l ed wi th onl y
two output arguments (i n the case where the opti mi zati on
al gori thm onl y needs the val ues of c and ceq but not GC and GCeq):
function [c,ceq,GC,GCeq] = mycon(x)
c = ... % nonlinear inequalities at x
ceq = ... % nonlinear equalities at x
if nargout > 2 % nonlcon called with 4 outputs
GC = ... % gradients of the inequalities
GCeq = ... % gradients of the equalities
end
fgoalattain
4-18
I f nonlcon returns a vector c of m components and x has l ength n,
then the gradi ent GC of c(x) i s an n-by-m matri x, where GC(i,j) i s
the parti al deri vati ve of c(j) wi th respect to x(i) (i .e., the jth
col umn of GC i s the gradi ent of the jth i nequal i ty constrai nt c(j)).
Li kewi se, i f ceq has p components, the gradi ent GCeq of ceq(x) i s
an n-by-p matri x, where GCeq(i,j) i s the parti al deri vati ve of
ceq(j) wi th respect to x(i) (i .e., the jth col umn of GCeq i s the
gradi ent of the jth equal i ty constrai nt ceq(j)).
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on.
DerivativeCheck Compare user-suppl i ed deri vati ves
(gradi ents of objecti ve or constrai nts) to fi ni te-di fferenci ng
deri vati ves.
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed or sol ved.
DiffMaxChange Maxi mum change i n vari abl es for
fi ni te-di fference gradi ents.
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te-di fference gradi ents.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
GoalExactAchieve Speci fi es the number of goal s to just
achi eve, that i s, do not try to over- or underachi eve.
GradConstr Gradi ent for the constrai nts defi ned by user. See
the descri pti on of nonlcon under the Argumentssecti on above to
see how to defi ne the gradi ent i n nonlcon.
GradObj Gradi ent for the objecti ve functi on defi ned by user.
See the descri pti on of fun under the Arguments secti on above to
see how to defi ne the gradi ent i n fun. The gradi ent must be
provi ded to use the l arge-scal e method. I t i s opti onal for the
medi um-scal e method.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
fgoalattain
4-19
MaxIter Maxi mum number of i terati ons al l owed.
MeritFunction Use goal attai nment/mi ni max meri t functi on i f
set to 'multiobj'. Use fmincon meri t functi on i f set to
'singleobj'.
TolCon Termi nati on tol erance on the constrai nt vi ol ati on.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
weight A wei ghti ng vector to control the rel ati ve under-attai nment or
over-attai nment of the objecti ves i n fgoalattain. When the val ues
of goal are all nonzero, to ensure the same percentage of under- or
over-attai nment of the acti ve objecti ves, set the wei ghti ng functi on
to abs(goal). (The acti ve objecti ves are the set of objecti ves that
are barri ers to further i mprovement of the goal s at the sol uti on.)
Note: Setti ng weight=abs(goal) when any of the goal val ues
are zero wi l l cause that goal constrai nt to be treated l i ke a hard
constrai nt rather than as a goal constrai nt.
When the wei ghti ng functi on weight i s posi ti ve, fgoalattain
attempts to make the objecti ves l ess than the goal val ues. To make
the objecti ve functi ons greater than the goal val ues, set weight to
be negati ve rather than posi ti ve. To make an objecti ve functi on as
near as possi bl e to a goal val ue, use the GoalsExactAchieve
parameter and put that objecti ve as the fi rst el ement of the vector
returned by fun (see the descri pti on of fun and options above).
attain-
factor
attainfactor i s the amount of over- or underachi evement of the
goal s. I f attainfactor i s negati ve, the goal s have been
over -achi eved; i f attainfactor i s posi ti ve, the goal s have been
under-achi eved.
fgoalattain
4-20
Examples Consi der a l i near system of di fferenti al equati ons.
An output feedback control l er, K, i s desi gned produci ng a cl osed l oop system
The ei genval ues of the cl osed l oop system are determi ned from the matri ces A,
B, C, and K usi ng the command eig(A+B*K*C). Cl osed l oop ei genval ues must l i e
on the real axi s i n the compl ex pl ane to the l eft of the poi nts [5,3,1]. I n
order not to saturate the i nputs, no el ement i n K can be greater than 4 or be l ess
than 4.
The system i s a two-i nput, two-output, open l oop, unstabl e system, wi th
state-space matri ces.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb
lambda.upper for the upper bounds ub
lambda.ineqlin for the l i near i nequal i ti es
lambda.eqlin for the l i near equal i ti es
lambda.ineqnonlin for the nonl i near i nequal i ti es
lambda.eqnonlin for the nonl i near equal i ti es
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.funcCount The number of functi on eval uati ons.
output.algorithm The al gori thm used.
x

A BKC + ( )x Bu + =
y Cx =
fgoalattain
4-21
The set of goal values for the cl osed l oop ei genval ues ar e i ni ti al i zed as
goal = [5,3,1];
To ensure the same percentage of under- or over-attai nment i n the acti ve
objecti ves at the sol uti on, the wei ghti ng matri x, weight, i s set to abs(goal).
Starti ng wi th a control l er, K = [1,1; 1,1], fi rst wri te an M-fi l e, eigfun.m:
function F = eigfun(K,A,B,C)
F = sort(eig(A+B*K*C)); % Evaluate objectives
Next, enter system matri ces and i nvoke an opti mi zati on routi ne:
A = [0.5 0 0; 0 2 10; 0 1 2];
B = [1 0; 2 2; 0 1];
C = [1 0 0; 0 0 1];
K0 = [1 1; 1 1]; % Initialize controller matrix
goal = [5 3 1]; % Set goal values for the eigenvalues
weight = abs(goal) % Set weight for same percentage
lb = 4*ones(size(K0)); % Set lower bounds on the controller
ub = 4*ones(size(K0)); % Set upper bounds on the controller
options = optimset('Display','iter'); % Set display parameter
[K,fval,attainfactor] = fgoalattain('eigfun',K0,...
goal,weight,[],[],[],[],lb,ub,[],options,A,B,C)
A
0.5 0 0
0 2 10
0 1 2
= B
1 0
2 2
0 1
= C
1 0 0
0 0 1
=
fgoalattain
4-22
Thi s exampl e can be run by usi ng the demonstrati on scri pt goaldemo. After
about 12 i terati ons, a sol uti on i s
Active constraints:
1
2
4
9
10
K =
4.0000 0.2564
4.0000 4.0000
fval =
6.9313
4.1588
1.4099
attainfactor =
0.3863
Discussion The attai nment factor i ndi cates that each of the objecti ves has been
over-achi eved by at l east 38.63% over the ori gi nal desi gn goal s. The acti ve
constrai nts, i n thi s case constrai nts 1 and 2, are the objecti ves that are barri ers
to further i mprovement and for whi ch the percentage of over-attai nment i s met
exactl y. Three of the l ower bound constrai nts are al so acti ve.
I n the above desi gn, the opti mi zer tri es to make the objecti ves l ess than the
goal s. For a worst case probl em where the objecti ves must be as near to the
goal s as possi bl e, set options.GoalsExactAchieve to the number of objecti ves
for whi ch thi s i s requi red.
Consi der the above probl em when you want al l the ei genval ues to be equal to
the goal val ues. A sol uti on to thi s probl em i s found by i nvoki ng fgoalattain
wi th options.GoalsExactAchieve set to 3.
options = optimset('GoalsExactAchieve',3);
[K,fval,attainfactor] = fgoalattain(...
eigfun,K0,goal,weight,[],[],[],[],lb,ub,[],options,A,B,C)
fgoalattain
4-23
After about seven i terati ons, a sol uti on i s
K =
1.5954 1.2040
0.4201 2.9046
fval =
5.0000
3.0000
1.0000
attainfactor =
1.0859e20
I n thi s case the opti mi zer has tri ed to match the objecti ves to the goal s. The
attai nment factor (of 1.0859e20) i ndi cates that the goal s have been matched
al most exactl y.
Notes Thi s probl em has di sconti nui ti es when the ei genval ues become compl ex; thi s
expl ai ns why the convergence i s sl ow. Al though the underl yi ng methods
assume the functi ons are conti nuous, the method i s abl e to make steps toward
the sol uti on si nce the di sconti nui ti es do not occur at the sol uti on poi nt. When
the objecti ves and goal s are compl ex, fgoalattain tri es to achi eve the goal s i n
a l east-squares sense.
Algorithm Mul ti objecti ve opti mi zati on concerns the mi ni mi zati on of a set of objecti ves
si mul taneousl y. One formul ati on for thi s probl em, and i mpl emented i n
fgoalattain, i s the goal attai nment probl em of Gembi cki [1]. Thi s entai l s the
constructi on of a set of goal val ues for the objecti ve functi ons. Mul ti objecti ve
opti mi zati on i s di scussed ful l y i n the I ntroduction toAlgorithmschapter of thi s
tool box.
I n thi s i mpl ementati on, the sl ack vari abl e i s used as a dummy argument to
mi ni mi ze the vector of objecti ves F(x) si mul taneousl y; goal i s a set of val ues
that the objecti ves attai n. General l y, pri or to the opti mi zati on, i t i s unknown
whether the objecti ves wi l l even reach the goal s (under attai nment) or be
mi ni mi zed l ess than the goal s (over attai nment). A wei ghti ng vector, weight,
control s the rel ati ve under-attai nment or over-attai nment of the objecti ves.

fgoalattain
4-24
fgoalattain uses a Sequenti al Quadrati c Programmi ng (SQP) method, whi ch
i s descri bed ful l y i n the I ntroduction toAlgorithms chapter. Modi fi cati ons are
made to the l i ne sear ch and Hessi an. I n the l i ne sear ch an exact mer i t functi on
(see [5] and [6]) i s used together wi th the meri t functi on pr oposed by [2, 3]. The
l i ne search i s termi nated when ei ther meri t functi on shows i mprovement. A
modi fi ed Hessi an, whi ch takes advantage of speci al structure of thi s probl em,
i s al so used (see [5] and [6]). A ful l descri pti on of the modi fi cati ons used i s found
i n the Goal Attai nment Method secti on of the I ntroduction toAlgorithms
chapter. Setti ng options.MeritFunction = 'singleobj' uses the meri t
functi on and Hessi an used i n fmincon.
attainfactor contai ns the val ue of at the sol uti on. A negati ve val ue of
i ndi cates over-attai nment i n the goal s.
See al so the SQP I mpl ementati on secti on i n the I ntroduction toAlgorithms
chapter for more detai l s on the al gori thm used and the types of procedures
pri nted under the Procedures headi ng for the options.Display = 'iter'
setti ng.
Limitations The objecti ves must be conti nuous. fgoalattain may gi ve onl y l ocal sol uti ons.
See Also fmincon, fminimax, optimset
References [1] Gembi cki , F.W., Vector Opti mi zati on for Control wi th Performance and
Parameter Sensi ti vi ty I ndi ces, Ph.D. Di ssertati on, Case Western Reserve
Uni v., Cl evel and, OH, 1974.
[2] Han, S.P., A Gl obal l y Conver gent Method For Nonl i near Pr ogr ammi ng,
J ournal of Optimization Theory and Applications, Vol . 22, p. 297, 1977.
[3] Powel l , M.J.D., A Fast Al gori thm for Nonl i neary Constrai ned
Opti mi zati on Cal cul ati ons, Numerical Analysis, ed. G.A. Watson, Lecture
Notes in Mathematics, Spri nger Verl ag, Vol . 630, 1978.
[4] Fl emi ng, P.J. and A.P. Pashkevi ch, Computer Aided Control SystemDesign
Usinga Multi-ObjectiveOptimisation Approach, Control 1985 Conference,
Cambri dge, UK, pp. 174-179.
[5] Brayton, R.K., S.W. Di rector, G.D. Hachtel , and L.Vi di gal , A New
Al gori thm for Stati sti cal Ci rcui t Desi gn Based on Quasi Newton Methods and
fgoalattain
4-25
Functi on Spl i tti ng, I EEE Transactionson Circuitsand Systems, Vol . CAS-26,
pp. 784794, Sept. 1979.
[6] Grace, A.C.W., ComputerAi ded Control System Desi gn Usi ng
Opti mi zati on Techni ques, Ph.D. Thesi s, Uni versi ty of Wal es, Bangor,
Gwynedd, UK, 1989.
fminbnd
4-26
4fmi nbnd
Purpose Fi nd the mi ni mum of a functi on of one vari abl e on a fi xed i nterval
where x, x
1
, and x
2
are scal ars and f(x) i s a functi on that returns a scal ar.
Syntax x = fminbnd(fun,x1,x2)
x = fminbnd(fun,x1,x2,options)
x = fminbnd(fun,x1,x2,options,P1,P2,...)
[x,fval] = fminbnd(...)
[x,fval,exitflag] = fminbnd(...)
[x,fval,exitflag,output] = fminbnd(...)
Description fminbnd fi nds the mi ni mum of a functi on of one vari abl e wi thi n a fi xed
i nterval .
x = fminbnd(fun,x1,x2) returns a val ue x that i s a l ocal mi ni mi zer of the
scal ar val ued functi on that i s descri bed i n fun i n the i nterval x1 < x < x2.
x = fminbnd(fun,x1,x2,options) mi ni mi zes wi th the opti mi zati on
parameters speci fi ed i n the structure options.
x = fminbnd(fun,x1,x2,options,P1,P2,...) provi des for addi ti onal
arguments, P1, P2, etc., whi ch are passed to the objecti ve functi on, fun. Use
options=[] as a pl acehol der i f no opti ons are set.
[x,fval] = fminbnd(...) returns the val ue of the objecti ve functi on
computed i n fun at the sol uti on x.
[x,fval,exitflag] = fminbnd(...) returns a val ue exitflag that descri bes
the exi t condi ti on of fminbnd.
[x,fval,exitflag,output] = fminbnd(...) returns a structure output that
contai ns i nformati on about the opti mi zati on.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
f x ( )
x
min such that x
1
x x
2
< <
fminbnd
4-27
rel evant to fminbnd are i ncl uded bel ow for fun, options, exitflag, and
output.
fun The functi on to be mi ni mi zed. fun takes a scal ar x and returns a
scal ar val ue f of the objecti ve functi on eval uated at x. You can
speci fy fun to be an i nl i ne object. For exampl e,
x = fminbnd(inline('sin(x*x)'),x0)
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function f = myfun(x)
f = ... % Compute function value at x
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on. fminbnd uses
these options structure fi el ds:
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
TolX Termi nati on tol erance on x.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.algorithm The al gori thm used.
output.funcCount The number of functi on eval uati ons.
fminbnd
4-28
Examples A mi ni mum of si n(x) occurs at
x = fminbnd('sin',0,2*pi)
x =
4.7124
The val ue of the functi on at the mi ni mum i s
y = sin(x)
y =
1.0000
To fi nd the mi ni mum of the functi on
on the i nterval (0,5), fi rst wri te an M-fi l e:
function f = myfun(x)
f = (x3).^2 1;
Next, cal l an opti mi zati on routi ne:
x = fminbnd('myfun',0,5)
Thi s generates the sol uti on
x =
3
The val ue at the mi ni mum i s
y = f(x)
y =
1
Algorithm fminbnd i s an M-fi l e. The al gori thm i s based on gol den secti on search and
parabol i c i nterpol ati on. A Fortran program i mpl ementi ng the same al gori thm
i s gi ven i n [1].
Limitations The functi on to be mi ni mi zed must be conti nuous. fminbnd may onl y gi ve l ocal
sol uti ons.
f x ( ) x 3 ( )
2
1 =
fminbnd
4-29
fminbnd often exhi bi ts sl ow convergence when the sol uti on i s on a boundary of
the i nterval . I n such a case, fmincon often gi ves faster and more accurate
sol uti ons.
fminbnd onl y handl es real vari abl es.
See Also fminsearch, fmincon, fminunc, optimset, inline
References [1] Forsythe, G.E., M.A. Mal col m, and C.B. Mol er, Computer Methods for
Mathematical Computations, Prenti ce Hal l , 1976.
fmincon
4-30
4fmi ncon
Purpose Fi nd the mi ni mum of a constrai ned nonl i near mul ti vari abl e functi on
where x, b, beq, lb, and ub are vectors, A and Aeq are matri ces, c(x) and ceq(x)
are functi ons that return vectors, and f(x) i s a functi on that returns a scal ar.
f(x), c(x), and ceq(x) can be nonl i near functi ons.
Syntax x = fmincon(fun,x0,A,b)
x = fmincon(fun,x0,A,b,Aeq,beq)
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub)
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon)
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options)
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options,P1,P2, ...)
[x,fval] = fmincon(...)
[x,fval,exitflag] = fmincon(...)
[x,fval,exitflag,output] = fmincon(...)
[x,fval,exitflag,output,lambda] = fmincon(...)
[x,fval,exitflag,output,lambda,grad] = fmincon(...)
[x,fval,exitflag,output,lambda,grad,hessian] = fmincon(...)
Description fmincon fi nds the constrai ned mi ni mum of a scal ar functi on of several
vari abl es starti ng at an i ni ti al esti mate. Thi s i s general l y referred to as
constrained nonlinear optimization or nonlinear programming.
x = fmincon(fun,x0,A,b) starts at x0 and fi nds a mi ni mum x to the functi on
descri bed i n fun subject to the l i near i nequal i ti es A*x <= b. x0 can be a scal ar,
vector, or matri x.
x = fmincon(fun,x0,A,b,Aeq,beq) mi ni mi zes fun subject to the l i near
equal i ti es Aeq*x = beq as wel l as A*x <= b. Set A=[] and b=[] i f no
i nequal i ti es exi st.
f x ( )
x
min subject to c x ( ) 0
ceq x ( ) 0 =
A x b
Aeq x beq
l b x ub
fmincon
4-31
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub) defi nes a set of l ower and upper
bounds on the desi gn vari abl es, x, so that the sol uti on i s al ways i n the range
lb <= x <= ub. Set Aeq=[] and beq=[] i f no equal i ti es exi st.
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon) subjects the
mi ni mi zati on to the nonl i near i nequal i ti es c(x) or equal i ti es ceq(x) defi ned i n
nonlcon. fmincon opti mi zes such that c(x) <= 0 and ceq(x) = 0. Set lb=[]
and/or ub=[] i f no bounds exi st.
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options) mi ni mi zes
wi th the opti mi zati on parameters speci fi ed i n the structure options.
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options,P1,P2,...)
passes the probl em-dependent parameters P1, P2, etc., di rectl y to the functi ons
fun and nonlcon. Pass empty matri ces as pl acehol ders for A, b, Aeq, beq, lb, ub,
nonlcon, and options i f these arguments are not needed.
[x,fval] = fmincon(...) returns the val ue of the objecti ve functi on fun at
the sol uti on x.
[x,fval,exitflag] = fmincon(...) returns a val ue exitflag that descri bes
the exi t condi ti on of fmincon.
[x,fval,exitflag,output] = fmincon(...) returns a structure output wi th
i nformati on about the opti mi zati on.
[x,fval,exitflag,output,lambda] = fmincon(...) returns a structure
lambda whose fi el ds contai n the Lagrange mul ti pl i ers at the sol uti on x.
[x,fval,exitflag,output,lambda,grad] = fmincon(...) returns the val ue
of the gr adi ent of fun at the sol uti on x.
[x,fval,exitflag,output,lambda,grad,hessian] = fmincon(...) returns
the val ue of the Hessi an of fun at the sol uti on x.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
fmincon
4-32
rel evant to fmincon are i ncl uded bel ow for fun, nonlcon, options, exitflag,
lambda, and output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
scal ar val ue f of the objecti ve functi on eval uated at x. You can
speci fy fun to be an i nl i ne object. For exampl e,
fun = inline('sin(x''*x)');
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function f = myfun(x)
f = ... % Compute function value at x
I f the gradi ent of fun can al so be computed and options.GradObj
i s 'on', as set by
options = optimset('GradObj','on')
then the functi on fun must return, i n the second output argument,
the gradi ent val ue g, a vector, at x. Note that by checki ng the val ue
of nargout the functi on can avoi d computi ng g when fun i s cal l ed
wi th onl y one output argument (i n the case where the opti mi zati on
al gor i thm onl y needs the val ue of f but not g):
function [f,g] = myfun(x)
f = ... % compute the function value at x
if nargout > 1 % fun called with two output arguments
g = ... % compute the gradient evaluated at x
end
The gradi ent i s the parti al deri vati ves of f at the poi nt x. That i s,
the ith component of g i s the parti al deri vati ve of f wi th respect to
the ith component of x.
fmincon
4-33
I f the Hessi an matri x can al so be computed and options.Hessian
i s 'on', i .e., options = optimset('Hessian','on'), then the
functi on fun must return the Hessi an val ue H, a symmetri c matri x,
at x i n a thi rd output argument. Note that by checki ng the val ue of
nargout we can avoi d computi ng H when fun i s cal l ed wi th onl y one
or two output arguments (i n the case where the opti mi zati on
al gori thm onl y needs the val ues of f and g but not H):
function [f,g,H] = myfun(x)
f = ... % Compute the objective function value at x
if nargout > 1 % fun called with two output arguments
g = ... % gradient of the function evaluated at x
if nargout > 2
H = ... % Hessian evaluated at x
end
The Hessi an matri x i s the second parti al deri vati ves matri x of f at
the poi nt x. That i s, the (ith,jth) component of H i s the second
parti al deri vati ve of f wi th respect to x
i
and x
j
, . The
Hessi an i s by defi ni ti on a symmetri c matri x.
nonlcon The functi on that computes the nonl i near i nequal i ty constr ai nts
c(x)<=0 and nonl i near equal i ty constrai nts ceq(x)=0. nonlcon i s a
stri ng contai ni ng the name of a functi on (an M-fi l e, a bui l t-i n, or a
MEX-fi l e). nonlcon takes a vector x and returns two arguments, a
vector c of the nonl i near i nequal i ti es eval uated at x and a vector
ceq of the nonl i near equal i ti es eval uated at x. For exampl e, i f
nonlcon='mycon' then the M-fi l e mycon.m woul d have the form
function [c,ceq] = mycon(x)
c = ... % Compute nonlinear inequalities at x
ceq = ... % Compute the nonlinear equalities at x

2
f x
i
x
j

fmincon
4-34
I f the gradi ents of the constrai nts can al so be computed and
options.GradConstr i s 'on', as set by
options = optimset('GradConstr','on')
then the functi on nonlcon must al so return, i n the thi rd and fourth
output arguments, GC, the gradi ent of c(x), and GCeq, the gradi ent
of ceq(x). Note that by checki ng the val ue of nargout the functi on
can avoi d computi ng GC and GCeq when nonlcon i s cal l ed wi th onl y
two output arguments (i n the case where the opti mi zati on
al gori thm onl y needs the val ues of c and ceq but not GC and GCeq):
function [c,ceq,GC,GCeq] = mycon(x)
c = ... % nonlinear inequalities at x
ceq = ... % nonlinear equalities at x
if nargout > 2 % nonlcon called with 4 outputs
GC = ... % gradients of the inequalities
GCeq = ... % gradients of the equalities
end
I f nonlcon returns a vector c of m components and x has l ength n,
then the gradi ent GC of c(x) i s an n-by-m matri x, where GC(i,j) i s
the parti al deri vati ve of c(j) wi th respect to x(i) (i .e., the jth
col umn of GC i s the gradi ent of the jth i nequal i ty constrai nt c(j)).
Li kewi se, i f ceq has p components, the gradi ent GCeq of ceq(x) i s
an n-by-p matri x, where GCeq(i,j) i s the parti al deri vati ve of
ceq(j) wi th respect to x(i) (i .e., the jth col umn of GCeq i s the
gradi ent of the jth equal i ty constrai nt ceq(j)).
fmincon
4-35
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on. Some parameters
appl y to al l al gori thms, some are onl y rel evant when usi ng the
l arge-scal e al gori thm, and others are onl y rel evant when usi ng the
medi um-scal e al gori thm.
We start by descri bi ng the LargeScale opti on si nce i t states a
preferencefor whi ch al gori thm to use. I t i s onl y a preference si nce
certai n condi ti ons must be met to use the l arge-scal e al gori thm.
For fmincon, the gradient must beprovided (see the descri pti on of
fun above to see how) or el se the medi um-scal e al gori thm wi l l be
used.
LargeScale Use l arge-scal e al gori thm i f possi bl e when set to
'on'. Use medi um-scal e al gori thm when set to 'off'.
Parameters used by both the l arge-scal e and medi um-scal e
al gori thms:
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
GradObj Gradi ent for the objecti ve functi on defi ned by user.
See the descri pti on of fun under the Arguments secti on above to
see how to defi ne the gradi ent i n fun. The gradi ent must be
provi ded to use the l arge-scal e method. I t i s opti onal for the
medi um-scal e method.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
TolCon Termi nati on tol erance on the constrai nt vi ol ati on.
TolX Termi nati on tol erance on x.
fmincon
4-36
Parameters used by the l arge-scal e al gori thm onl y:
Hessian Hessi an for the objecti ve functi on defi ned by user. See
the descri pti on of fun under the Arguments secti on above to see
how to defi ne the Hessi an i n fun.
HessPattern Sparsi ty pattern of the Hessi an for
fi ni te-di fferenci ng. I f i t i s not conveni ent to compute the sparse
Hessi an matri x H i n fun, the l arge-scal e method i n fmincon can
approxi mate H vi a sparse fi ni te-di fferences (of the gradi ent)
provi ded the sparsity structureof H i .e., l ocati ons of the
nonzeros i s suppl i ed as the val ue for HessPattern. I n the
worst case, i f the structure i s unknown, you can set HessPattern
to be a dense matri x and a ful l fi ni te-di fference approxi mati on
wi l l be computed at each i terati on (thi s i s the defaul t). Thi s can
be very expensi ve for l arge probl ems so i t i s usual l y worth the
effort to determi ne the sparsi ty structure.
MaxPCGIter Maxi mum number of PCG (precondi ti oned
conjugate gradi ent) i terati ons (see the Algorithmsecti on bel ow).
PrecondBandWidth Upper bandwi dth of precondi ti oner for
PCG. By defaul t, di agonal precondi ti oni ng i s used (upper
bandwi dth of 0). For some probl ems, i ncreasi ng the bandwi dth
reduces the number of PCG i terati ons.
TolPCG Termi nati on tol erance on the PCG i terati on.
TypicalX Typi cal x val ues.
Parameters used by the medi um-scal e al gori thm onl y:
DerivativeCheck Compare user-suppl i ed deri vati ves
(gr adi ents of the objecti ve and constr ai nts) to fi ni te-di ffer enci ng
der i vati ves.
DiffMaxChange Maxi mum change i n vari abl es for
fi ni te-di fference gradi ents.
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te-di fference gradi ents.
LineSearchType Li ne search al gori thm choi ce.
fmincon
4-37
Examples Fi nd val ues of x that mi ni mi ze , starti ng at the poi nt
x = [10; 10; 10] and subject to the constrai nts
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb.
lambda.upper for the upper bounds ub.
lambda.ineqlin for the l i near i nequal i ti es.
lambda.eqlin for the l i near equal i ti es.
lambda.ineqnonlin for the nonl i near i nequal i ti es.
lambda.eqnonlin for the nonl i near equal i ti es.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.funcCount The number of functi on eval uati ons.
output.algorithm The al gori thm used.
output.cgiterations The number of PCG i terati ons
(l arge-scal e al gori thm onl y).
output.stepsize The fi nal step si ze taken (medi um-scal e
al gori thm onl y).
output.firstorderopt A measure of fi rst-order opti mal i ty
(l arge-scal e al gori thm onl y).
f x ( ) x
1
x
2
x
3
=
0 x
1
2x
2
2x
3
+ + 72
fmincon
4-38
Fi rst, wri te an Mfi l e that returns a scal ar val ue f of the functi on eval uated at
x:
function f = myfun(x)
f = x(1) * x(2) * x(3);
Then rewri te the constrai nts as both l ess than or equal to a constant,
Si nce both constrai nts are l i near, formul ate them as the matri x i nequal i ty
where
Next, suppl y a starti ng poi nt and i nvoke an opti mi zati on routi ne:
x0 = [10; 10; 10]; % Starting guess at the solution
[x,fval] = fmincon('myfun',x0,A,b)
After 66 functi on eval uati ons, the sol uti on i s
x =
24.0000
12.0000
12.0000
where the functi on val ue i s
fval =
3.4560e+03
and l i near i nequal i ty constrai nts eval uate to be <= 0
A*xb=
72
0
Notes Large-scale optimization. To use the l arge-scal e method, the gradi ent must be
pr ovi ded i n fun (and options.GradObj set to 'on'). A warni ng i s gi ven i f no
x
1
2x
2
2x
3
0
x
1
2x
2
2x
3
+ + 72
A x b
A
1 2 2
1 2 2
= b
0
72
=
fmincon
4-39
gradi ent i s provi ded and options.LargeScale i s not 'off'. fmincon permi ts
g(x) to be an approxi mate gradi ent but thi s opti on i s not recommended: the
numeri cal behavi or of most opti mi zati on codes i s consi derabl y more robust
when the true gradi ent i s used.
The l arge-scal e method i n fmincon i s most effecti ve when the matri x of second
deri vati ves, i .e., the Hessi an matri x H(x), i s al so computed. However,
eval uati on of the true Hessi an matri x i s not requi red. For exampl e, i f you can
suppl y the Hessi an sparsi ty structure (usi ng the HessPattern parameter i n
options), then fmincon wi l l compute a sparse fi ni te-di fference approxi mati on
to H(x).
I f x0 i s not stri ctl y feasi bl e, fmincon chooses a new stri ctl y feasi bl e (centered)
starti ng poi nt.
I f components of xhave no upper (or l ower) bounds, then fmincon prefers that
the correspondi ng components of ub (or lb) be set to Inf (or Inf for lb) as
opposed to an arbi trary but very l arge posi ti ve (or negati ve i n the case of l ower
bounds) number.
Several aspects of l i nearl y constrai ned mi ni mi zati on shoul d be noted:
A dense (or fai r l y dense) col umn of matri x Aeq can resul t i n consi derabl e fi l l
and computati onal cost.
fmincon removes (numeri cal l y) l i nearl y dependent rows i n Aeq; however,
thi s process i nvol ves repeated matri x factori zati ons and therefore can be
costl y i f there are many dependenci es.
Each i terati on i nvol ves a sparse l east-squares sol ve wi th matri x
where R
T
i s the Chol esky factor of the precondi ti oner. Therefore, there i s a
potenti al confl i ct between choosi ng an effecti ve precondi ti oner and
mi ni mi zi ng fi l l i n .
Medium-scale optimization. Better numeri cal resul ts are l i kel y i f you speci fy
equal i ti es expl i ci tl y usi ng Aeq and beq, i nstead of i mpl i ci tl y usi ng lb and ub.
I f equal i ty constrai nts are present and dependent equal i ti es are detected and
removed i n the quadrati c subprobl em, 'dependent' i s pri nted under the
Procedures headi ng (when output i s asked for usi ng
options.Display = 'iter'). The dependent equal i ti es are onl y removed
B Aeq
T
R
T
=
B
fmincon
4-40
when the equal i ti es are consi stent. I f the system of equal i ti es i s not consi stent,
the subprobl em i s i nfeasi bl e and 'infeasible' i s pri nted under the
Procedures headi ng.
Algorithm Large-scale optimization. By defaul t fmincon wi l l choose the l arge-scal e al gori thm
if the user suppl i es the gradi ent i n fun (and GradObj i s 'on' i n options) and i f
only upper and l ower bounds exi sts or only l i near equal i ty constrai nts exi st.
Thi s al gori thm i s a subspace trust regi on method and i s based on the
i nteri or-refl ecti ve Newton method descri bed i n [5],[6]. Each i terati on i nvol ves
the approxi mate sol uti on of a l arge l i near system usi ng the method of
precondi ti oned conjugate gradi ents (PCG). See the trust-regi on and
precondi ti oned conjugate gradi ent method descri pti ons i n the Large-Scale
Algorithms chapter.
Medium-scale optimization. fmincon uses a Sequenti al Quadrati c Programmi ng
(SQP) method. I n thi s method, a Quadrati c Programmi ng (QP) subprobl em i s
sol ved at each i terati on. An esti mate of the Hessi an of the Lagrangi an i s
updated at each i terati on usi ng the BFGS formul a (see fminunc, references [3,
6]).
A l i ne search i s performed usi ng a meri t functi on si mi l ar to that proposed by
[1] and [2, 3]. The QP subprobl em i s sol ved usi ng an acti ve set strategy si mi l ar
to that descri bed i n [4]. A ful l descri pti on of thi s al gori thm i s found i n the
Constrai ned Opti mi zati on secti on of the I ntroduction toAlgorithms chapter
of the tool box manual .
See al so the SQP i mpl ementati on secti on i n the I ntroduction toAlgorithms
chapter for more detai l s on the al gori thm used.
Diagnostics Large-scale optimization. The l arge-scal e code wi l l not al l ow equal upper and
l ower bounds. For exampl e i f lb(2)==ub(2), then fmincon gi ves the error:
Equal upper and lower bounds not permitted in this large-scale
method.
Use equality constraints and the medium-scale method instead.
I f you onl y have equal i ty constrai nts you can sti l l use the l arge-scal e method.
But i f you have both equal i ti es and bounds, you must use the medi um-scal e
method.
fmincon
4-41
Limitations The functi on to be mi ni mi zed and the constrai nts must both be conti nuous.
fmincon may onl y gi ve l ocal sol uti ons.
When the probl em i s i nfeasi bl e, fmincon attempts to mi ni mi ze the maxi mum
constrai nt val ue.
The objecti ve functi on and constrai nt functi on must be real -val ued, that i s they
cannot return compl ex val ues.
Large-scale optimization. To use the l arge-scal e al gori thm, the user must suppl y
the gradi ent i n fun (and GradObj must be set 'on' i n options) , and onl y upper
and l ower bounds constrai nts may be speci fi ed, or only l i near equal i ty
constrai nts must exi st and Aeq cannot have more rows than col umns. Aeq i s
typi cal l y sparse. See Tabl e 1-4 for more i nformati on on what probl em
formul ati ons are covered and what i nformati on must be provi ded.
Currentl y, i f the anal yti cal gradi ent i s provi ded i n fun, the options parameter
DerivativeCheck cannot be used wi th the l arge-scal e method to compare the
anal yti c gradi ent to the fi ni te-di fference gradi ent. I nstead, use the
medi um-scal e method to check the deri vati ve wi th options parameter MaxIter
set to 0 i terati ons. Then run the probl em wi th the l arge-scal e method.
References [1] Han, S.P., A Gl obal l y Convergent Method for Nonl i near Programmi ng,
J ournal of Optimization Theoryand Applications, Vol . 22, p. 297, 1977.
[2] Powel l , M.J.D., The Convergence of Vari abl e Metri c Methods For
Nonl i nearl y Constrai ned Opti mi zati on Cal cul ati ons, Nonlinear Programming
3, (O.L. Mangasari an, R.R. Meyer, and S.M. Robi nson, eds.) Academi c Press,
1978.
[3] Powel l , M.J.D., A Fast Al gori thm for Nonl i neary Constrai ned
Opti mi zati on Cal cul ati ons, Numerical Analysis, ed. G.A. Watson, Lecture
Notes in Mathematics, Spri nger Verl ag, Vol . 630, 1978.
[4] Gi l l , P.E., W. Murray, and M.H. Wri ght, Practical Optimization, Academi c
Press, London, 1981.
[5] Col eman, T.F. and Y. Li , On the Convergence of Refl ecti ve Newton
Methods for Large-Scal e Nonl i near Mi ni mi zati on Subject to Bounds,
Mathematical Programming, Vol . 67, Number 2, pp. 189-224, 1994.
fmincon
4-42
[6] Col eman, T.F. and Y. Li , An I nteri or, Trust Regi on Approach for Nonl i near
Mi ni mi zati on Subject to Bounds, SI AM J ournal on Optimization, Vol . 6, pp.
418-445, 1996.
See Also fminbnd, fminsearch, fminunc, optimset
fminimax
4-43
4fmi ni max
Purpose Sol ve the mi ni max probl em
where x, b, beq, lb, and ubare vectors, A and Aeqare matri ces, c(x), ceq(x), and
F(x) are functi ons that return vectors. F(x), c(x), and ceq(x) can be nonl i near
functi ons.
Syntax x = fminimax(fun,x0)
x = fminimax(fun,x0,A,b)
x = fminimax(fun,x0,A,b,Aeq,beq)
x = fminimax(fun,x0,A,b,Aeq,beq,lb,ub)
x = fminimax(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon)
x = fminimax(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options)
x = fminimax(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options,P1,P2,...)
[x,fval] = fminimax(...)
[x,fval,maxfval] = fminimax(...)
[x,fval,maxfval,exitflag] = fminimax(...)
[x,fval,maxfval,exitflag,output] = fminimax(...)
[x,fval,maxfval,exitflag,output,lambda] = fminimax(...)
Description fminimax mi ni mi zes the worst-case val ue of a set of mul ti vari abl e functi ons,
starti ng at an i ni ti al esti mate. The val ues may be subject to constrai nts. Thi s
i s general l y referred to as the minimaxprobl em.
x = fminimax(fun,x0) starts at x0 and fi nds a mi ni max sol uti on x to the
functi ons descri bed i n fun.
x = fminimax(fun,x0,A,b) sol ves the mi ni max probl em subject to the l i near
i nequal i ti es A*x <= b.
F
i
x ( ) { }
F
i
{ }
max
x
min such that c x ( ) 0
ceq x ( ) 0 =
A x b
Aeq x beq =
l b x ub
fminimax
4-44
x = fminimax(fun,x,A,b,Aeq,beq) sol ves the mi ni max probl em subject to
the l i near equal i ti es Aeq*x = beq as wel l . Set A=[] and b=[] i f no i nequal i ti es
exi st.
x = fminimax(fun,x,A,b,Aeq,beq,lb,ub) defi nes a set of l ower and upper
bounds on the desi gn vari abl es, x, so that the sol uti on i s al ways i n the range
lb <= x <= ub.
x = fminimax(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon) subjects the mi ni max
probl em to the nonl i near i nequal i ti es c(x) or equal i ty constrai nts ceq(x)
defi ned i n nonlcon. fminimax opti mi zes such that c(x) <= 0 and ceq(x) = 0.
Set lb=[] and/or ub=[] i f no bounds exi st.
x = fminimax(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options) mi ni mi zes
wi th the opti mi zati on parameters speci fi ed i n the structure options.
x = fminimax(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options,P1,P2,...)
passes the probl em-dependent parameters P1, P2, etc., di rectl y to the functi ons
fun and nonlcon. Pass empty matri ces as pl acehol ders for A, b, Aeq, beq, lb, ub,
nonlcon, and options i f these arguments are not needed.
[x,fval] = fminimax(...) returns the val ue of the objecti ve functi on fun at
the sol uti on x.
[x,fval,maxfval] = fminimax(...) returns the maxi mum functi on val ue at
the sol uti on x.
[x,fval,maxfval,exitflag] = fminimax(...) returns a val ue exitflag
that descri bes the exi t condi ti on of fminimax.
[x,fval,maxfval,exitflag,output] = fminimax(...) returns a structure
output wi th i nformati on about the opti mi zati on.
[x,fval,maxfval,exitflag,output,lambda] = fminimax(...) returns a
structure lambda whose fi el ds contai n the Lagrange mul ti pl i ers at the sol uti on
x.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
fminimax
4-45
rel evant to fminimax are i ncl uded bel ow for fun, nonlcon, options, exitflag,
lambda, maxfval, and output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
vector F of the objecti ve functi ons eval uated at x. You can speci fy
fun to be an i nl i ne object. For exampl e,
fun = inline('sin(x.*x)');
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function F = myfun(x)
F = ... % Compute function values at x
To mi ni mi ze the worst case absol ute val ues of any of the el ements
of the vector F(x) (i .e., mi n{max abs{F(x)}}), parti ti on those
objecti ves i nto the fi rst el ements of F and set options.MinAbsMax
to be the number of such objecti ves.
I f the gradi ent of the objecti ve functi on can al so be computed and
options.GradObj i s 'on', as set by
options = optimset('GradObj','on')
then the functi on fun must return, i n the second output argument,
the gradi ent val ue G, a matri x, at x. Note that by checki ng the val ue
of nargout the functi on can avoi d computi ng G when 'myfun' i s
cal l ed wi th onl y one output argument (i n the case where the
opti mi zati on al gori thm onl y needs the val ue of F but not G):
function [F,G] = myfun(x)
F = ... % compute the function values at x
if nargout > 1 % two output arguments
G = ... % gradients evaluated at x
end
The gradi ent i s the parti al deri vati ves dF/ dxof each F at the poi nt
x. I f F i s a vector of l ength m and x has l ength n, then the gradi ent
G of F(x) i s an n-by-m matri x where G(i,j) i s the parti al deri vati ve
of F(j) wi th respect to x(i) (i .e., the jth col umn of G i s the gradi ent
of the jth objecti ve functi on F(j)).
fminimax
4-46
nonlcon The functi on that computes the nonl i near i nequal i ty constrai nts
c(x) <=0 and nonl i near equal i ty constrai nts ceq(x)=0. nonlcon i s
a stri ng contai ni ng the name of a functi on (an M-fi l e, a bui l t-i n, or
a MEX-fi l e). nonlcon takes a vector x and returns two arguments,
a vector c of the nonl i near i nequal i ti es eval uated at x and a vector
ceq of the nonl i near equal i ti es eval uated at x. For exampl e, i f
nonlcon='mycon' then the M-fi l e mycon.m woul d have the form
function [c,ceq] = mycon(x)
c = ... % Compute nonlinear inequalities at x
ceq = ... % Compute the nonlinear equalities at x
I f the gradi ents of the constrai nts can al so be computed and
options.GradConstr i s 'on', as set by
options = optimset('GradConstr','on')
then the functi on nonlcon must al so return, i n the thi rd and fourth
output arguments, GC, the gradi ent of c(x), and GCeq, the gradi ent
of ceq(x). Note that by checki ng the val ue of nargout the functi on
can avoi d computi ng GC and GCeq when nonlcon i s cal l ed wi th onl y
two output arguments (i n the case where the opti mi zati on
al gori thm onl y needs the val ues of c and ceq but not GC and GCeq):
function [c,ceq,GC,GCeq] = mycon(x)
c = ... % nonlinear inequalities at x
ceq = ... % nonlinear equalities at x
if nargout > 2 % nonlcon called with 4 outputs
GC = ... % gradients of the inequalities
GCeq = ... % gradients of the equalities
end
I f nonlcon returns a vector c of m components and x has l ength n,
then the gradi ent GC of c(x) i s an n-by-m matri x, where GC(i,j) i s
the parti al deri vati ve of c(j) wi th respect to x(i) (i .e., the jth
col umn of GC i s the gradi ent of the jth i nequal i ty constrai nt c(j)).
Li kewi se, i f ceq has p components, the gradi ent GCeq of ceq(x) i s
an n-by-p matri x, where GCeq(i,j) i s the parti al deri vati ve of
ceq(j) wi th respect to x(i) (i .e., the jth col umn of GCeq i s the
gradi ent of the jth equal i ty constrai nt ceq(j)).
fminimax
4-47
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameter s usi ng the optimset functi on.
DerivativeCheck Compare user-suppl i ed deri vati ves
(gradi ents of the objecti ve or constrai nts) to fi ni te-di fferenci ng
deri vati ves.
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed or sol ved.
DiffMaxChange Maxi mum change i n vari abl es for
fi ni te-di fference gradi ents.
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te-di fference gradi ents.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
GradConstr Gradi ent for the constrai nts defi ned by user. See
the descri pti on of nonlcon under the Argumentssecti on above to
see how to defi ne the gradi ent i n nonlcon.
GradObj Gradi ent for the objecti ve functi on defi ned by user.
See the descri pti on of fun under the Arguments secti on above to
see how to defi ne the gradi ent i n fun. The gradi ent must be
provi ded to use the l arge-scal e method. I t i s opti onal for the
medi um-scal e method.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
MeritFunction Use goal attai nment/mi ni max meri t functi on i f
set to 'multiobj'. Use fmincon meri t functi on i f set to
'singleobj'.
MinAbsMax Number of F(x) to mi ni mi ze the worst case absol ute
val ues.
TolCon Termi nati on tol erance on the constrai nt vi ol ati on.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
fminimax
4-48
Examples Fi nd val ues of x that mi ni mi ze the maxi mum val ue of
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb
lambda.upper for the upper bounds ub
lambda.ineqlin for the l i near i nequal i ti es
lambda.eqlin for the l i near equal i ti es
lambda.ineqnonlin for the nonl i near i nequal i ti es
lambda.eqnonlin for the nonl i near equal i ti es
maxfval Maxi mum of the functi on val ues eval uated at the sol uti on x, that
i s, maxfval = max{fun(x)}.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.funcCount The number of functi on eval uati ons.
output.algorithm The al gori thm used.
f
1
x ( ) , f
2
x ( ) , f
3
x ( ) , f
4
x ( ) , f
5
x ( )
fminimax
4-49
where
Fi rst, wri te an M-fi l e that computes the fi ve functi ons at x:
function f = myfun(x)
f(1)= 2*x(1)^2+x(2)^248*x(1)40*x(2)+304; %Objectives
f(2)= x(1)^2 3*x(2)^2;
f(3)= x(1) + 3*x(2) 18;
f(4)= x(1) x(2);
f(5)= x(1) + x(2) 8;
Next, i nvoke an opti mi zati on routi ne:
x0 = [0.1; 0.1]; % Make a starting guess at solution
[x,fval] = fminimax('myfun',x0)
After seven i terati ons, the sol uti on i s
x =
4.0000
4.0000
fval =
0.0000 64.0000 2.0000 8.0000 0.0000
Notes The number of objecti ves for whi ch the worst case absol ute val ues of F are
mi ni mi zed i s set i n options.MinAbsMax. Such objecti ves shoul d be parti ti oned
i nto the fi rst el ements of F.
For exampl e, consi der the above probl em, whi ch requi res fi ndi ng val ues of x
that mi ni mi ze the maxi mum absol ute val ue of
f
1
x ( ) 2x
1
2
x
2
2
48x
1
40x
2
304 + + =
f
2
x ( ) x
2
2
3x
2
2
=
f
3
x ( ) x
1
3x
2
18 + =
f
4
x ( ) x
1
x
2
=
f
5
x ( ) x
1
x
2
8. + =
f
1
x ( ) , f
2
x ( ) , f
3
x ( ) , f
4
x ( ) , f
5
x ( )
fminimax
4-50
Thi s i s sol ved by i nvoki ng fminimax wi th the commands
x0 = [0.1; 0.1]; % Make a starting guess at the solution
options = optimset('MinAbsMax',5); % Minimize absolute values
[x,fval] = fminimax(fun,x0,[],[],[],[],[],[],[],options);
After seven i terati ons, the sol uti on i s
x =
4.9256
2.0796
fval =
37.2356 37.2356 6.8357 7.0052 0.9948
I f equal i ty constrai nts are present and dependent equal i ti es are detected and
removed i n the quadrati c subprobl em, 'dependent' i s pri nted under the
Procedures headi ng (when the output opti on i s options.Display='iter').
The dependent equal i ti es are onl y removed when the equal i ti es are consi stent.
I f the system of equal i ti es i s not consi stent, the subprobl em i s i nfeasi bl e and
'infeasible' i s pri nted under the Procedures headi ng.
Algorithm fminimax uses a Sequenti al Quadrati c Programmi ng (SQP) method [3].
Modi fi cati ons are made to the l i ne search and Hessi an. I n the l i ne search an
exact meri t functi on (see [4] and [5]) i s used together wi th the meri t functi on
proposed by [1] and [2]. The l i ne search i s termi nated when ei ther meri t
functi on shows i mprovement. A modi fi ed Hessi an that takes advantage of
speci al structure of thi s probl em i s al so used. Setti ng
options.MeritFunction = 'singleobj' uses the meri t functi on and Hessi an
used i n fmincon.
See al so the SQP i mpl ementati on secti on i n the I ntroduction toAlgorithms
chapter for more detai l s on the al gori thm used and the types of procedures
pri nted under the Procedures headi ng for options.Display='iter' setti ng.
Limitations The functi on to be mi ni mi zed must be conti nuous. fminimax may onl y gi ve
l ocal sol uti ons.
See Also optimset, fgoalattain, lsqnonlin
References [1] Han, S.P., A Gl obal l y Convergent Method For Nonl i near Programmi ng, J .
of Optimization Theory and Applications, Vol . 22, p. 297, 1977.
fminimax
4-51
[2] Powel l , M.J.D., A Fast Al gori thm for Nonl i neary Constrai ned
Opti mi zati on Cal cul ati ons, Numerical Analysis, ed. G.A. Watson, Lecture
Notes in Mathematics, Spri nger Verl ag, Vol . 630, 1978.
[3] Brayton, R.K., S.W. Di rector, G.D. Hachtel , and L.Vi di gal , A New
Al gori thm for Stati sti cal Ci rcui t Desi gn Based on Quasi Newton Methods and
Functi on Spl i tti ng, I EEE Trans. Circuits and Systems, Vol . CAS-26, pp.
784-794, Sept. 1979.
[4] Grace, A.C.W., ComputerAi ded Control System Desi gn Usi ng
Opti mi zati on Techni ques, Ph.D. Thesi s, Uni versi ty of Wal es, Bangor,
Gwynedd, UK, 1989.
[5] Madsen, K. and H. Schjaer-Jacobsen, Al gori thms for Worst Case Tol erance
Opti mi zati on, I EEE Transactionsof Circuits and Systems, Vol . CAS-26, Sept.
1979.
fminsearch
4-52
4fmi nsear ch
Purpose Fi nd the mi ni mum of an unconstrai ned mul ti vari abl e functi on
where x i s a vector and f(x) i s a functi on that returns a scal ar.
Syntax x = fminsearch(fun,x0)
x = fminsearch(fun,x0,options)
x = fminsearch(fun,x0,options,P1,P2,...)
[x,fval] = fminsearch(...)
[x,fval,exitflag] = fminsearch(...)
[x,fval,exitflag,output] = fminsearch(...)
Description fminsearch fi nds the mi ni mum of a scal ar functi on of several vari abl es,
starti ng at an i ni ti al esti mate. Thi s i s general l y referred to as unconstrained
nonlinear optimization.
x = fminsearch(fun,x0) starts at the poi nt x0 and fi nds a l ocal mi ni mum x of
the functi on descri bed i n fun. x0 can be a scal ar, vector, or matri x.
x = fminsearch(fun,x0,options) mi ni mi zes wi th the opti mi zati on
parameters speci fi ed i n the structure options.
x = fminsearch(fun,x0,options,P1,P2,...) passes the probl em-dependent
parameters P1, P2, etc., di rectl y to the functi on fun. Pass an empty matri x for
options to use the defaul t val ues for options.
[x,fval] = fminsearch(...) returns i n fval the val ue of the objecti ve
functi on fun at the sol uti on x.
[x,fval,exitflag] = fminsearch(...) returns a val ue exitflag that
descri bes the exi t condi ti on of fminsearch.
[x,fval,exitflag,output] = fminsearch(...) returns a structure output
that contai ns i nformati on about the opti mi zati on.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
f x ( )
x
min
fminsearch
4-53
rel evant to fminsearch are i ncl uded bel ow for fun, options, exitflag, and
output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
scal ar val ue f of the objecti ve functi on eval uated at x. You can
speci fy fun to be an i nl i ne object. For exampl e,
x = fminsearch(inline('sin(x''*x)'),x0)
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function f = myfun(x)
f = ... % Compute function value at x
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameter s usi ng the optimset functi on.fminsearch uses
these options parameters:
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
fminsearch
4-54
Examples Mi ni mi ze the one-di mensi onal functi on f(x) = sin(x) + 3.
To use an M-fi l e, i .e., fun = 'myfun', create a fi l e myfun.m:
function f = myfun(x)
f = sin(x) + 3;
Then cal l fminsearch to fi nd a mi ni mum of fun near 2:
x = fminsearch('myfun',2)
To mi ni mi ze the functi on f(x) = sin(x) + 3 usi ng an i nl i ne object:
f = inline('sin(x)+3');
x = fminsearch(f,2);
Algorithms fminsearch uses the si mpl ex search method of [1]. Thi s i s a di rect search
method that does not use numeri cal or anal yti c gradi ents as i n fminunc.
I f n i s the l ength of x, a si mpl ex i n n-di mensi onal space i s characteri zed by the
n+1 di sti nct vectors that are i ts verti ces. I n two-space, a si mpl ex i s a tri angl e;
i n three-space, i t i s a pyrami d. At each step of the search, a new poi nt i n or near
the current si mpl ex i s generated. The functi on val ue at the new poi nt i s
compared wi th the functi ons val ues at the verti ces of the si mpl ex and, usual l y,
one of the verti ces i s repl aced by the new poi nt, gi vi ng a new si mpl ex. Thi s step
i s repeated unti l the di ameter of the si mpl ex i s l ess than the speci fi ed
tol erance.
fminsearch i s general l y l ess effi ci ent than fminunc for probl ems of order
greater than two. However, when the probl em i s hi ghl y di sconti nuous,
fminsearch may be more robust.
Limitations fminsearch can often handl e di sconti nui ty, parti cul arl y i f i t does not occur
near the sol uti on. fminsearch may onl y gi ve l ocal sol uti ons.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.algorithm The al gori thm used.
output.funcCount The number of functi on eval uati ons.
fminsearch
4-55
fminsearch onl y mi ni mi zes over the real numbers, that i s, xmust onl y consi st
of real numbers and f(x) must onl y return real numbers. When xhas compl ex
vari abl es, they must be spl i t i nto real and i magi nary parts.
Note: fminsearch i s not the preferred choi ce for sol vi ng probl ems that are
sums-of-squares, that i s, of the form: .
I nstead use the lsqnonlin functi on, whi ch has been opti mi zed for probl ems of
thi s form.
See Also fminbnd, fminunc, optimset, inline
References [1] Lagari as, J.C., J.A. Reeds, M.H. Wri ght, P.E. Wri ght, Convergence
Pr oper ti es of the Nel der -Mead Si mpl ex Al gor i thm i n Low Di mensi ons, to
appear i n the SI AM J ournal of Optimization.
f x ( ) min f
1
x ( )
2
f
2
x ( )
2
f
3
x ( )
2
L + + + =
fminunc
4-56
4fmi nunc
Purpose Fi nd the mi ni mum of an unconstrai ned mul ti vari abl e functi on
where x i s a vector and f(x) i s a functi on that returns a scal ar.
Syntax x = fminunc(fun,x0)
x = fminunc(fun,x0,options)
x = fminunc(fun,x0,options,P1,P2,...)
[x,fval] = fminunc(...)
[x,fval,exitflag] = fminunc(...)
[x,fval,exitflag,output] = fminunc(...)
[x,fval,exitflag,output,grad] = fminunc(...)
[x,fval,exitflag,output,grad,hessian] = fminunc(...)
Description fminunc fi nds the mi ni mum of a scal ar functi on of several vari abl es, starti ng
at an i ni ti al esti mate. Thi s i s general l y referred to as unconstrained nonlinear
optimization.
x = fminunc(fun,x0) starts at the poi nt x0 and fi nds a l ocal mi ni mum x of the
functi on descri bed i n fun. x0 can be a scal ar, vector, or matri x.
x = fminunc(fun,x0,options) mi ni mi zes wi th the opti mi zati on parameters
speci fi ed i n the structure options.
x = fminunc(fun,x0,options,P1,P2,...) passes the probl em-dependent
parameters P1, P2, etc., di rectl y to the functi on fun. Pass an empty matri x for
options to use the defaul t val ues for options.
[x,fval] = fminunc(...) returns i n fval the val ue of the objecti ve functi on
fun at the sol uti on x.
[x,fval,exitflag] = fminunc(...) returns a val ue exitflag that descri bes
the exi t condi ti on.
[x,fval,exitflag,output] = fminunc(...) returns a structure output that
contai ns i nformati on about the opti mi zati on.
f x ( )
x
min
fminunc
4-57
[x,fval,exitflag,output,grad] = fminunc(...) returns i n grad the val ue
of the gr adi ent of fun at the sol uti on x.
[x,fval,exitflag,output,grad,hessian] = fminunc(...) returns i n
hessian the val ue of the Hessi an of the objecti ve functi on fun at the sol uti on x.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to fminunc are i ncl uded bel ow for fun, options, exitflag, and
output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
scal ar val ue f of the objecti ve functi on eval uated at x. You can
speci fy fun to be an i nl i ne object. For exampl e,
x = fminunc(inline('sin(x''*x)'),x0)
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function f = myfun(x)
f = ... % Compute function value at x
I f the gradi ent of fun can al so be computed and options.GradObj
i s 'on', as set by
options = optimset('GradObj','on')
then the functi on fun must return, i n the second output argument,
the gradi ent val ue g, a vector, at x. Note that by checki ng the val ue
of nargout the functi on can avoi d computi ng g when fun i s cal l ed
wi th onl y one output argument (i n the case where the opti mi zati on
al gori thm onl y needs the val ue of f but not g):
function [f,g] = myfun(x)
f = ... % compute the function value at x
if nargout > 1 % fun called with 2 output arguments
g = ... % compute the gradient evaluated at x
end
fminunc
4-58
The gradi ent i s the parti al deri vati ves of f at the poi nt x.
That i s, the ith component of g i s the parti al deri vati ve of f wi th
respect to the ith component of x.
I f the Hessi an matri x can al so be computed and options.Hessian
i s 'on', i .e., options = optimset('Hessian','on'), then the
functi on fun must return the Hessi an val ue H, a symmetri c matri x,
at x i n a thi rd output argument. Note that by checki ng the val ue of
nargout we can avoi d computi ng H when fun i s cal l ed wi th onl y one
or two output arguments (i n the case where the opti mi zati on
al gor i thm onl y needs the val ues of f and g but not H):
function [f,g,H] = myfun(x)
f = ... % Compute the objective function value at x
if nargout > 1 % fun called with two output arguments
g = ... % gradient of the function evaluated at x
if nargout > 2
H = ... % Hessian evaluated at x
end
The Hessi an matri x i s the second parti al deri vati ves matri x of f at
the poi nt x. That i s, the (ith,jth) component of H i s the second
parti al deri vati ve of f wi th respect to x
i
and x
j
, . The
Hessi an i s by defi ni ti on a symmetri c matri x.
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on. Some
parameters appl y to al l al gori thms, some are onl y rel evant when
usi ng the l arge-scal e al gori thm, and others are onl y rel evant when
usi ng the medi um-scal e al gori thm.
We start by descri bi ng the LargeScale opti on si nce i t states a
preferencefor whi ch al gori thm to use. I t i s onl y a preference si nce
certai n condi ti ons must be met to use the l arge-scal e al gori thm.
For fminunc, the gradient must beprovided (see the descri pti on of
fun above to see how) or el se the medi um-scal e al gori thm wi l l be
used.
LargeScale Use l arge-scal e al gori thm i f possi bl e when set to
'on'. Use medi um-scal e al gori thm when set to 'off'.
f x

2
f x
i
x
j

fminunc
4-59
Parameters used by both the l arge-scal e and medi um-scal e
al gori thms:
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
GradObj Gradi ent for the objecti ve functi on defi ned by user.
See the descri pti on of fun under the Arguments secti on above to
see how to defi ne the gradi ent i n fun. The gradi ent must be
provi ded to use the l arge-scal e method. I t i s opti onal for the
medi um-scal e method.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
Parameters used by the l arge-scal e al gori thm onl y:
Hessian Hessi an for the objecti ve functi on defi ned by user. See
the descri pti on of fun under the Arguments secti on above to see
how to defi ne the Hessi an i n fun.
HessPattern Sparsi ty pattern of the Hessi an for
fi ni te-di fferenci ng. I f i t i s not conveni ent to compute the sparse
Hessi an matri x H i n fun, the l arge-scal e method i n fminunc can
approxi mate H vi a sparse fi ni te-di fferences (of the gradi ent)
provi ded the sparsity structureof H i .e., l ocati ons of the
nonzeros i s suppl i ed as the val ue for HessPattern. I n the
worst case, i f the structure i s unknown, you can set HessPattern
to be a dense matri x and a ful l fi ni te-di fference approxi mati on
wi l l be computed at each i terati on (thi s i s the defaul t). Thi s can
be very expensi ve for l arge probl ems so i t i s usual l y worth the
effort to determi ne the sparsi ty structure.
fminunc
4-60
MaxPCGIter Maxi mum number of PCG (precondi ti oned
conjugate gradi ent) i terati ons (see the Algorithmsecti on bel ow).
PrecondBandWidth Upper bandwi dth of precondi ti oner for
PCG. By defaul t, di agonal precondi ti oni ng i s used (upper
bandwi dth of 0). For some probl ems, i ncreasi ng the bandwi dth
reduces the number of PCG i terati ons.
TolPCG Termi nati on tol erance on the PCG i terati on.
TypicalX Typi cal x val ues.
Parameters used by the medi um-scal e al gori thm onl y:
DerivativeCheck Compare user-suppl i ed deri vati ves
(gradi ent) to fi ni te-di fferenci ng deri vati ves.
DiffMaxChange Maxi mum change i n vari abl es for
fi ni te-di fference gradi ents.
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te-di fference gradi ents.
LineSearchType Li ne search al gori thm choi ce.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
fminunc
4-61
Examples Mi ni mi ze the functi on f(x) = 3*x
1
2
+ 2*x1*x2 + x
2
2
.
To use an M-fi l e, i .e., fun = 'myfun', create a fi l e myfun.m:
function f = myfun(x)
f = 3*x(1)^2 + 2*x(1)*x(2) + x(2)^2; % cost function
Then cal l fminunc to fi nd a mi ni mum of 'myfun' near [1,1]:
x0 = [1,1];
[x,fval] = fminunc('myfun',x0)
After a coupl e of i terati ons, the sol uti on, x, and the val ue of the functi on at x,
fval, are returned:
x =
1.0e008 *
0.7914 0.2260
fval =
1.5722e016
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.funcCount The number of functi on eval uati ons.
output.algorithm The al gori thm used.
output.cgiterations The number of PCG i terati ons
(l arge-scal e al gori thm onl y).
output.stepsize The fi nal step si ze taken (medi um-scal e
al gor i thm onl y).
output.firstorderopt A measure of fi rst-order opti mal i ty:
the norm of the gradi ent at the sol uti on x.
fminunc
4-62
To mi ni mi ze thi s functi on wi th the gradi ent provi ded, modi fy the M-fi l e
myfun.m so the gradi ent i s the second output argument
function [f,g] = myfun(x)
f = 3*x(1)^2 + 2*x(1)*x(2) + x(2)^2; % cost function
if nargout > 1
g(1) = 6*x(1)+2*x(2);
g(2) = 2*x(1)+2*x(2);
end
and i ndi cate the gradi ent val ue i s avai l abl e by creati ng an opti mi zati on opti ons
structure wi th options.GradObj set to 'on' usi ng optimset:
options = optimset('GradObj','on');
x0 = [1,1];
[x,fval] = fminunc('myfun',x0,options)
After several i terati ons the sol uti on x and fval, the val ue of the functi on at x,
are returned:
x =
1.0e015 *
0.6661 0
fval2 =
1.3312e030
To mi ni mi ze the functi on f(x) = sin(x) + 3 usi ng an i nl i ne object
f = inline('sin(x)+3');
x = fminunc(f,4)
whi ch returns a sol uti on
x =
4.7124
Notes fminunc i s not the preferred choi ce for sol vi ng probl ems that are
sums-of-squares, that i s, of the form: .
I nstead use the lsqnonlin functi on, whi ch has been opti mi zed for probl ems of
thi s form.
f x ( ) min f
1
x ( )
2
f
2
x ( )
2
f
3
x ( )
2
L + + + =
fminunc
4-63
To use the l arge-scal e method, the gradi ent must be provi ded i n fun (and
options.GradObj set to 'on'). A warni ng i s gi ven i f no gradi ent i s provi ded and
options.LargeScale i s not 'off'.
Algorithms Large-scale optimization. By defaul t fminunc wi l l choose the l arge-scal e al gori thm
i f the user suppl i es the gradi ent i n fun (and GradObj i s 'on' i n options). Thi s
al gori thm i s a subspace trust regi on method and i s based on the
i nteri or-refl ecti ve Newton method descri bed i n [8],[9]. Each i terati on i nvol ves
the approxi mate sol uti on of a l arge l i near system usi ng the method of
precondi ti oned conjugate gradi ents (PCG). See the trust-regi on and
precondi ti oned conjugate gradi ent method descri pti ons i n the Large-Scale
Algorithms chapter.
Medium-scale optimization. fminunc wi th options.LargeScale set to off uses
the BFGS Quasi -Newton method wi th a mi xed quadrati c and cubi c l i ne search
procedure. Thi s quasi -Newton method uses the BFGS [1-4] formul a for
updati ng the approxi mati on of the Hessi an matri x. The DFP [5,6,7] formul a,
whi ch approxi mates the i nverse Hessi an matri x, i s sel ected by setti ng
options. HessUpdate to 'dfp' (and options.LargeScale to 'off'). A steepest
descent method i s sel ected by setti ng options.HessUpdate to 'steepdesc'
(and options.LargeScale to 'off'), al though thi s i s not recommended.
The defaul t l i ne search al gori thm, i .e., when options.LineSearchType i s set to
'quadcubic', i s a safeguarded mi xed quadrati c and cubi c pol ynomi al
i nterpol ati on and extrapol ati on method. A safeguarded cubi c pol ynomi al
method can be sel ected by setti ng options.LineSearchType to 'cubicpoly'.
Thi s second method general l y requi res fewer functi on eval uati ons but more
gradi ent eval uati ons. Thus, i f gradi ents are bei ng suppl i ed and can be
cal cul ated i nexpensi vel y, the cubi c pol ynomi al l i ne search method i s
preferabl e. A ful l descri pti on of the al gori thms i s gi ven i n the I ntroduction to
Algorithms chapter.
Limitations The functi on to be mi ni mi zed must be conti nuous.fminunc may onl y gi ve l ocal
sol uti ons.
fminunc onl y mi ni mi zes over the real numbers, that i s, x must onl y consi st of
real numbers and f(x) must onl y return real numbers. When x has compl ex
vari abl es, they must be spl i t i nto real and i magi nary parts.
fminunc
4-64
Large-scale optimization. To use the l arge-scal e al gori thm, the user must suppl y
the gradi ent i n fun (and GradObj must be set 'on' i n options). See Tabl e 1-4
for more i nformati on on what probl em formul ati ons are covered and what
i nformati on must be provi ded.
Currentl y, i f the anal yti cal gradi ent i s provi ded i n fun, the options parameter
DerivativeCheck cannot be used wi th the l arge-scal e method to compare the
anal yti c gradi ent to the fi ni te-di fference gradi ent. I nstead, use the
medi um-scal e method to check the deri vati ve wi th options parameter MaxIter
set to 0 i terati ons. Then run the probl em agai n wi th the l arge-scal e method.
See Also fminsearch, optimset, inline
References [1] Broyden, C.G., The Convergence of a Cl ass of Doubl eRank Mi ni mi zati on
Al gori thms, J . I nst. Math. Applic., Vol . 6, pp. 7690, 1970.
[2] Fl etcher, R.,A New Approach to Vari abl e Metri c Al gori thms, Computer
J ournal, Vol . 13, pp. 317322, 1970.
[3] Gol dfarb, D., A Fami l y of Vari abl e Metri c Updates Deri ved by Vari ati onal
Means, Mathematics of Computing, Vol . 24, pp. 2326, 1970.
[4] Shanno, D.F., Condi ti oni ng of Quasi Newton Methods for Functi on
Mi ni mi zati on, Mathematics of Computing, Vol . 24, pp. 647656, 1970.
[5] Davi don, W.C., Vari abl e Metri c Method for Mi ni mi zati on, A.E.C. Research
and Development Report, ANL-5990, 1959.
[6] Fl etcher, R. and M.J.D. Powel l , A Rapi dl y Convergent Descent Method for
Mi ni mi zati on, Computer J ., Vol . 6, pp. 163168, 1963.
[7] Fl etcher, R., Practi cal Methods of Opti mi zati on, Vol . 1, Unconstrained
Optimization, John Wi l ey and Sons, 1980.
[8] Col eman, T.F. and Y. Li , On the Convergence of Refl ecti ve Newton
Methods for Large-Scal e Nonl i near Mi ni mi zati on Subject to Bounds,
Mathematical Programming, Vol . 67, Number 2, pp. 189224, 1994.
[9] Col eman, T.F. and Y. Li , An I nteri or, Trust Regi on Approach for Nonl i near
Mi ni mi zati on Subject to Bounds, SI AM J ournal on Optimization, Vol . 6, pp.
418445, 1996.
fseminf
4-65
4fsemi nf
Purpose Fi nd mi ni mum of a semi -i nfi ni tel y constrai ned mul ti vari abl e nonl i near
functi on
where x, b, beq, lb, and ubare vectors, A and Aeqare matri ces, c(x), ceq(x), and
K
i
(x,w
i
) are functi ons that return vectors, and f(x) i s a functi on that returns a
scal ar. f(x), c(x), and ceq(x) can be nonl i near functi ons. The vectors (or matri ces)
are conti nuous functi ons of both x and an addi ti onal set of
vari abl es . The vari abl es are vectors of, at most,
l ength two.
Syntax x = fseminf(fun,x0,ntheta,seminfcon)
x = fseminf(fun,x0,ntheta,seminfcon,A,b)
x = fseminf(fun,x0,ntheta,seminfcon,A,b,Aeq,beq)
x = fseminf(fun,x0,ntheta,seminfcon,A,b,Aeq,beq,lb,ub)
x = fseminf(fun,x0,ntheta,seminfcon,A,b,Aeq,beq,lb,ub,options)
x = fseminf(fun,x0,ntheta,seminfcon,A,b,Aeq,beq,...
lb,ub,options,P1,P2,...)
[x,fval] = fseminf(...)
[x,fval,exitflag] = fseminf(...)
[x,fval,exitflag,output] = fseminf(...)
[x,fval,exitflag,output,lambda] = fseminf(...)
Description fseminf fi nds the mi ni mum of a semi -i nfi ni tel y constrai ned scal ar functi on of
several vari abl es, starti ng at an i ni ti al esti mate. The ai m i s to mi ni mi ze f(x) so
f x ( )
x
min subject to c x ( ) 0,
ceq x ( ) 0 =
A x b
Aeq x beq =
l b x ub
K
1
x w
1
, ( ) 0
K
2
x w
2
, ( ) 0

K
n
x w
n
, ( ) 0
K
i
x w
i
, ( ) 0
w
1
w
2
w
n
, , , w
1
w
2
w
n
, , ,
fseminf
4-66
the constrai nts hol d for al l possi bl e val ues of (or ). Si nce i t i s
i mpossi bl e to cal cul ate al l possi bl e val ues of , a regi on must be chosen
for over whi ch to cal cul ate an appropri atel y sampl ed set of val ues.
x = fseminf(fun,x0,ntheta,seminfcon) starts at x0 and fi nds a mi ni mum of
the functi on fun constrai ned by ntheta semi -i nfi ni te constrai nts defi ned i n
seminfcon.
x = fseminf(fun,x0,ntheta,seminfcon,A,b) al so tri es to sati sfy the l i near
i nequal i ti es A*x <= b.
x = fseminf(fun,x0,ntheta,seminfcon,A,b,Aeq,beq) mi ni mi zes subject to
the l i near equal i ti es Aeq*x = beq as wel l . Set A=[] and b=[] i f no i nequal i ti es
exi st.
x = fseminf(fun,x0,ntheta,seminfcon,A,b,Aeq,beq,lb,ub) defi nes a set
of l ower and upper bounds on the desi gn vari abl es, x, so that the sol uti on i s
al ways i n the range lb <= x <= ub.
x = fseminf(fun,x0,ntheta,seminfcon,A,b,Aeq,beq,lb,ub,options)
mi ni mi zes wi th the opti mi zati on parameters speci fi ed i n the structure
options.
x = fseminf(fun,x0,ntheta,seminfcon,A,b,Aeq,beq,lb,ub,options,
P1,P2,...) passes the probl em-dependent parameters P1, P2, etc., di rectl y to
the functi ons fun and seminfcon. Pass empty matri ces as pl acehol ders for A, b,
Aeq, beq, lb, ub, and options i f these arguments are not needed.
[x,fval] = fseminf(...) returns the val ue of the objecti ve functi on fun at
the sol uti on x.
[x,fval,exitflag] = fseminf(...) returns a val ue exitflag that descri bes
the exi t condi ti on.
[x,fval,exitflag,output] = fseminf(...) returns a structure output that
contai ns i nformati on about the opti mi zati on.
[x,fval,exitflag,output,lambda] = fseminf(...) returns a structure
lambda whose fi el ds contai n the Lagrange mul ti pl i ers at the sol uti on x.
w
i

1
w
i

2

K
i
x w
i
, ( )
w
i
fseminf
4-67
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to fseminf are i ncl uded bel ow for fun, ntheta, options, seminfcon,
exitflag, lambda, and output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
scal ar val ue f of the objecti ve functi on eval uated at x. You can
speci fy fun to be an i nl i ne object. For exampl e,
fun = inline('sin(x''*x)');
Al ternati vel y, fun can be a stri ng contai ni ng the name of a
functi on (an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f
fun='myfun' then the M-fi l e functi on myfun.m woul d have the
form
function f = myfun(x)
f = ... % Compute function value at x
I f the gradi ent of fun can al so be computed and options.GradObj
i s 'on', as set by
options = optimset('GradObj','on')
then the functi on fun must return, i n the second output argument,
the gradi ent val ue g, a vector, at x. Note that by checki ng the val ue
of nargout the functi on can avoi d computi ng g when fun i s cal l ed
wi th onl y one output argument (i n the case where the
opti mi zati on al gori thm onl y needs the val ue of f but not g):
function [f,g] = myfun(x)
f = ... % compute the function value at x
if nargout > 1 % fun called with 2 output arguments
g = ... % compute the gradient evaluated at x
end
The gradi ent i s the parti al deri vati ves of f at the poi nt x. That i s,
the ith component of g i s the parti al deri vati ve of f wi th respect
to the ith component of x.
ntheta The number of semi -i nfi ni te constrai nts.
fseminf
4-68
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on.
DerivativeCheck Compare user-suppl i ed deri vati ves
(gradi ents) to fi ni te-di fferenci ng deri vati ves.
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed or sol ved.
DiffMaxChange Maxi mum change i n vari abl es for
fi ni te-di fference gradi ents.
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te-di fference gradi ents.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
GradObj Gradi ent for the objecti ve functi on defi ned by user.
See the descri pti on of fun under the Argumentssecti on above to
see how to defi ne the gradi ent i n fun. The gradi ent must be
provi ded to use the l arge-scal e method. I t i s opti onal for the
medi um-scal e method.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
TolCon Termi nati on tol erance on the constrai nt vi ol ati on.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
seminfcon The functi on that computes the vector of nonl i near i nequal i ty
constrai nts, c, a vector of nonl i near equal i ty constrai nts, ceq, and
ntheta semi -i nfi ni te constrai nts (vectors or matri ces) K1, K2,...,
Kntheta eval uated over an i nterval S at the poi nt x. seminfcon i s
a stri ng contai ni ng the name of the functi on (an M-fi l e, a bui l t-i n,
or a MEX-fi l e). For exampl e, i f seminfcon='myinfcon' then the
M-fi l e myinfcon.m woul d have the form
fseminf
4-69
function [c,ceq,K1,K2,...,Kntheta,S] = myinfcon(x,S)
% Initial sampling interval
if isnan(S(1,1)),
S = ...% S has ntheta rows and 2 columns
end
w1 = ...% Compute sample set
w2 = ...% Compute sample set
...
wntheta = ... % Compute sample set
K1 = ... % 1st semi-infinite constraint at x and w
K2 = ... % 2nd semi-infinite constraint at x and w
...
Kntheta = ...% last semi-infinite constraint at x and w
c = ... % Compute nonlinear inequalities at x
ceq = ... % Compute the nonlinear equalities at x
S i s a recommended sampl i ng i nterval , whi ch may or may not be
used. Return [] for c and ceq i f no such constrai nts exi st.
The vectors or matri ces, K1, K2, ..., Kntheta, contai n the
semi -i nfi ni te constrai nts eval uated for a sampl ed set of val ues for
the i ndependent vari abl es, w1, w2, ... wntheta, respecti vel y. The
two col umn matri x, S, contai ns a recommended sampl i ng i nterval
for val ues of w1, w2, ... wntheta, whi ch are used to eval uate K1,
K2, ... Kntheta. The ith row of S contai ns the recommended
sampl i ng i nterval for eval uati ng Ki. When Ki i s a vector, use onl y
S(i,1) (the second col umn can be al l zeros). When Ki i s a matri x,
s(i,2) i s used for the sampl i ng of the rows i n Ki, S(i,1) i s used
for the sampl i ng i nterval of the col umns of Ki (see
Two-Di mensi onal Exampl e i n the Examples secti on). On the
fi rst i terati on S i s NaN, so that some i ni ti al sampl i ng i nterval must
be determi ned by seminfcon.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons
or i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
fseminf
4-70
Notes The recommended sampl i ng i nterval , S, set i n seminfcon, may be vari ed by the
opti mi zati on routi ne fseminf duri ng the computati on because val ues other
than the recommended i nterval may be more appropri ate for effi ci ency or
robustness. Al so, the fi ni te regi on , over whi ch i s cal cul ated, i s
al l owed to vary duri ng the opti mi zati on provi ded that i t does not resul t i n
si gni fi cant changes i n the number of l ocal mi ni ma i n .
Examples One-Dimensional Example
Fi nd val ues of xthat mi ni mi ze
for al l val ues of and over the ranges
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb.
lambda.upper for the upper bounds ub.
lambda.ineqlin for the l i near i nequal i ti es.
lambda.eqlin for the l i near equal i ti es.
lambda.ineqnonlin for the nonl i near i nequal i ti es.
lambda.eqnonlin for the nonl i near equal i ti es.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.funcCount The number of functi on eval uati ons.
output.algorithm The al gori thm used.
output.stepsize The fi nal step si ze taken.
w
i
K
i
x w
i
, ( )
K
i
x w
i
, ( )
f x ( ) x
1
0.5 ( )
2
x
2
0.5 ( )
2
x
3
0.5 ( )
2
+ + =
where
K
1
x w
1
, ( ) w
1
x
1
( ) si n w
1
x
2
( ) cos
1
1000
------------- w
1
50 ( )
2
w
1
x
3
( ) si n x
3
1 =
K
2
x w
2
, ( ) w
2
x
2
( ) si n w
2
x
1
( ) cos
1
1000
------------- w
2
50 ( )
2
w
2
x
3
( ) si n x
3
1 =
w
1
w
2
fseminf
4-71
Note that the semi -i nfi ni te constrai nts are one-di mensi onal , that i s, vectors.
Si nce the constrai nts must be i n the form you wi l l need to
compute the constrai nts as
Fi rst, wri te an M-fi l e that computes the objecti ve functi on:
function f = myfun(x,s)
% Objective function
f = sum((x0.2).^2);
Second, wri te an M-fi l e, mycon.m, that computes the nonl i near equal i ty and
i nequal i ty constrai nts and the semi -i nfi ni te constrai nts:
function [c,ceq,K1,K2,s] = mycon(x,s)
% Initial sampling interval
if isnan(s(1,1)),
s = [0.2 0; 0.2 0];
end
% Sample set
w1 = 1:s(1,1):100;
w2 = 1:s(2,1):100;
% Semiinfinite constraints
K1 = sin(w1*X(1)).*cos(w1*X(2)) 1/1000*(w150).^2 ...
sin(w1*X(3))X(3)1;
K2 = sin(w2*X(2)).*cos(w2*X(1)) 1/1000*(w250).^2 ...
sin(w2*X(3))X(3)1;
% No constraints
c = []; ceq=[];
1 w
1
100
1 w
2
100
K
i
x w
i
, ( ) 0
K
1
x w
1
, ( ) w
1
x
1
( ) si n w
1
x
2
( ) cos
1
1000
------------- w
1
50 ( )
2
w
1
x
3
( ) si n x
3
1 0 =
K
2
x w
2
, ( ) w
2
x
2
( ) si n w
2
x
1
( ) cos
1
1000
------------- w
2
50 ( )
2
w
2
x
3
( ) si n x
3
1 0 =
fseminf
4-72
% Plot a graph of semi-infinite constraints
plot(w1,K1,'-',w2,K2,':'),title('Semiinfinite constraints')
drawnow
Then, i nvoke an opti mi zati on routi ne:
x0 = [0.5; 0.2; 0.3]; % Starting guess at the solution
[x,fval] = fseminf('myfun',x0,2,'mycon')
After ei ght i terati ons, the sol uti on i s
x =
0.6673
0.3013
0.4023
The functi on val ue and the maxi mum val ues of the semi -i nfi ni te constrai nts at
the sol uti on x are
fval =
0.0770
[c,ceq,K1,K2] = mycon(x,NaN);
max(K1)
ans =
0.0017
max(K2)
ans =
0.0845
fseminf
4-73
A pl ot of the semi -i nfi ni te constrai nts i s produced.
Thi s pl ot shows how peaks i n both functi ons ar e on the constr ai nt boundary.
The pl ot command i nsi de of 'mycon.m' wi l l sl ow down the computati on; remove
thi s l i ne to i mprove the speed.
Two-Dimensional Example
Fi nd val ues of x that mi ni mi ze
where
for al l val ues of and over the ranges:
starti ng at the poi nt .
Note that the semi -i nfi ni te constrai nt i s two-di mensi onal , that i s, a matri x.
-6
-5
-4
-3
-2
-1
0
0 10 20 30 40 50 60 70 80 90 100
Semi-infinite constraints
f x ( ) x
1
0.5 ( )
2
x
2
0.5 ( )
2
x
3
0.5 ( )
2
+ + =
K
1
x w , ( ) w
1
x
1
( ) si n 10w
2
x
2
( ) cos
1
1000
------------- w
1
50 ( )
2
10w
1
x
3
( ) si n x
3
+ =
w
2
x
2
( ) si n w
1
x
1
( ) cos
1
1000
------------- w
2
50 ( )
2
w
2
x
3
( ) si n x
3
1.5 +
w
1
w
2
1 w
1
100
1 w
2
100
x 0.2 0.2 0.2 , , [ ] =
fseminf
4-74
Fi rst, we reuse the fi l e myfun.m for the objecti ve functi on from the previ ous
exampl e.
Second, wri te an M-fi l e for the constrai nts, cal l ed mycon.m:
function [c,ceq,K1,s] = mycon(x,s)
% Initial sampling interval
if isnan(s(1,1)),
s = [2 2];
end
%
% Sampling set
w1x = 1:s(1,1):100;
w1y = 1:s(1,2):100;
[wx,wy] = meshgrid(w1x,w1y);
%
% Semiinfinite constraint
K1 = sin(wx*X(1)).*cos(wy*X(2))1/1000*(wx50).^2 ...
sin(wx*X(3))X(3)+sin(wy*X(2)).*cos(wx*X(1))...
1/1000*(wy50).^2sin(wy*X(3))X(3)1.5;
%
% No finite nonlinear constraints
c = []; ceq=[];
%
% Mesh plot
mesh(K1,title('Semiinfinite constraint')
drawnow
Next, i nvoke an opti mi zati on routi ne:
x0 = [0.25, 0.25, 0.25]; % Starting guess at the solution
[x,fval] = fseminf('myfun',x0,1,'mycon')
After seven i terati ons, the sol uti on i s
x =
0.2081 0.2066 0.1965
[c,ceq,K1] = mycon(x,NaN);
fseminf
4-75
The functi on val ue and the maxi mum val ue of the semi -i nfi ni te constrai nt at
the sol uti on x are
fval =
1.2066e04
max(max(K1))
ans =
0.0262
The fol l owi ng mesh pl ot i s produced.
Algorithm fseminf uses cubi c and quadrati c i nterpol ati on techni ques to esti mate peak
val ues i n the semi -i nfi ni te constrai nts. The peak val ues are used to form a set
of constrai nts that are suppl i ed to an SQP method as i n the functi on fmincon.
When the number of constrai nts changes, Lagrange mul ti pl i ers are real l ocated
to the new set of constr ai nts.
The recommended sampl i ng i nterval cal cul ati on uses the di fference between
the i nterpol ated peak val ues and peak val ues appeari ng i n the data set to
esti mate whether more or fewer poi nts need to be taken. The effecti veness of
the i nterpol ati on i s al so taken i nto consi derati on by extrapol ati ng the curve
and compari ng i t to other poi nts i n the curve. The recommended sampl i ng
i nterval i s decreased when the peak val ues are cl ose to constrai nt boundari es,
i .e., zero.
Semi-infinite constraint
fseminf
4-76
See al so the SQP i mpl ementati on secti on i n the I ntroduction toAlgorithms
chapter for more detai l s on the al gori thm used and the types of procedures
pri nted under the Procedures headi ng for options.Display = 'iter' setti ng.
Limitations The functi on to be mi ni mi zed, the constrai nts, and semi -i nfi ni te constrai nts,
must be conti nuous functi ons of x and w. fseminf may onl y gi ve l ocal sol uti ons.
When the probl em i s not feasi bl e, fseminf attempts to mi ni mi ze the maxi mum
constrai nt val ue.
See Also fmincon, optimset
fsolve
4-77
4fsol ve
Purpose Sol ve a system of nonl i near equati ons
for x, where xi s a vector and F(x) i s a functi on that returns a vector val ue.
Syntax x = fsolve(fun,x0)
x = fsolve(fun,x0,options)
x = fsolve(fun,x0,options,P1,P2, ... )
[x,fval] = fsolve(...)
[x,fval,exitflag] = fsolve(...)
[x,fval,exitflag,output] = fsolve(...)
[x,fval,exitflag,output,jacobian] = fsolve(...)
Description fsolve fi nds a r oot (zer o) of a system of nonl i near equati ons.
x = fsolve(fun,x0) star ts at x0 and tri es to sol ve the equati ons descri bed i n
fun.
x = fsolve(fun,x0,options) mi ni mi zes wi th the opti mi zati on parameters
speci fi ed i n the structure options.
x = fsolve(fun,x0,options,P1,P2,...) passes the probl em-dependent
parameters P1, P2, etc., di rectl y to the functi on fun. Pass an empty matri x for
options to use the defaul t val ues for options.
[x,fval] = fsolve(fun,x0) returns the val ue of the objecti ve functi on fun at
the sol uti on x.
[x,fval,exitflag] = fsolve(...) returns a val ue exitflag that descri bes
the exi t condi ti on.
[x,fval,exitflag,output] = fsolve(...) returns a structure output that
contai ns i nformati on about the opti mi zati on.
[x,fval,exitflag,output,jacobian] = fsolve(...) returns the Jacobi an
of fun at the sol uti on x.
F x ( ) 0 =
fsolve
4-78
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to fsolve are i ncl uded bel ow for fun, options, exitflag, and output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
vector F of the nonl i near equati ons eval uated at x. You can speci fy
fun to be an i nl i ne object. For exampl e,
x = fsolve(inline('sin(x.*x)'),x0)
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function F = myfun(x)
F = ... % Compute function values at x
I f the Jacobi an can al so be computed and options.Jacobian i s
'on', set by
options = optimset('Jacobian','on')
then the functi on fun must return, i n a second output argument,
the Jacobi an val ue J, a matri x, at x. Note that by checki ng the
val ue of nargout the functi on can avoi d computi ng J when fun i s
cal l ed wi th onl y one output argument (i n the case where the
opti mi zati on al gori thm onl y needs the val ue of F but not J):
function [F,J] = myfun(x)
F = ... % objective function values at x
if nargout > 1 % two output arguments
J = ... % Jacobian of the function evaluated at x
end
I f fun returns a vector (matri x) of m components and x has l ength
n, then the Jacobi an J i s an m-by-n matri x where J(i,j) i s the
parti al deri vati ve of F(i) wi th respect to x(j). (Note that the
Jacobi an J i s the transpose of the gradi ent of F.)
fsolve
4-79
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on. Some parameters
appl y to al l al gori thms, some are onl y rel evant when usi ng the
l arge-scal e al gori thm, and others are onl y rel evant when usi ng the
medi um-scal e al gori thm.
We start by descri bi ng the LargeScale opti on si nce i t states a
preferencefor whi ch al gori thm to use. I t i s onl y a preference si nce
certai n condi ti ons must be met to use the l arge-scal e al gori thm.
For lsqnonlin, the nonl i near system of equati ons cannot be
underdetermi ned; that i s, the number of equati ons (the number of
el ements of F returned by fun) must be at l east as many as the
l ength of x or el se the medi um-scal e al gori thm wi l l be used.
LargeScale Use l arge-scal e al gori thm i f possi bl e when set to
'on'. Use medi um-scal e al gori thm when set to 'off'.
Parameters used by both the l arge-scal e and medi um-scal e
al gori thms:
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
Jacobian Jacobi an for the objecti ve functi on defi ned by user.
See the descri pti on of fun above to see how to defi ne the Jacobi an
i n fun.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
fsolve
4-80
Parameters used by the l arge-scal e al gori thm onl y:
JacobPattern Sparsi ty pattern of the Jacobi an for
fi ni te-di fferenci ng. I f i t i s not conveni ent to compute the
Jacobi an matri x J i n fun, lsqnonlin can approxi mate J vi a
sparse fi ni te-di fferences provi ded the structure of J i .e.,
l ocati ons of the nonzeros i s suppl i ed as the val ue for
JacobPattern. I n the worst case, i f the structure i s unknown,
you can set JacobPattern to be a dense matri x and a ful l
fi ni te-di fference approxi mati on wi l l be computed i n each
i terati on (thi s i s the defaul t i f JacobPattern i s not set). Thi s can
be very expensi ve for l arge probl ems so i t i s usual l y worth the
effort to determi ne the sparsi ty structure.
MaxPCGIter Maxi mum number of PCG (precondi ti oned
conjugate gradi ent) i terati ons (see the Algorithmsecti on bel ow).
PrecondBandWidth Upper bandwi dth of precondi ti oner for
PCG. By defaul t, di agonal precondi ti oni ng i s used (upper
bandwi dth of 0). For some probl ems, i ncreasi ng the bandwi dth
reduces the number of PCG i terati ons.
TolPCG Termi nati on tol erance on the PCG i terati on.
TypicalX Typi cal x val ues.
Parameters used by the medi um-scal e al gori thm onl y:
DerivativeCheck Compare user-suppl i ed deri vati ves
(Jacobi an) to fi ni te-di fferenci ng deri vati ves.
DiffMaxChange Maxi mum change i n vari abl es for
fi ni te-di fferenci ng.
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te-di fferenci ng.
LevenbergMarquardt Choose Levenberg-Marquardt over
Gauss-Newton al gori thm.
LineSearchType Li ne search al gori thm choi ce.
fsolve
4-81
Examples Example 1: Fi nd a zero of the system of two equati ons and two unknowns
Thus we want to sol ve the fol l owi ng system for x
starti ng at x0 = [5 5].
Fi rst, wri te an M-fi l e that computes F, the val ues of the equati ons at x:
function F = myfun(x)
F = [2*x(1) x(2) exp(x(1));
x(1) + 2*x(2) exp(x(2))];
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.funcCount The number of functi on eval uati ons.
output.algorithm The al gori thm used.
output.cgiterations The number of PCG i terati ons
(l arge-scal e al gori thm onl y).
output.stepsize The fi nal step si ze taken (medi um-scal e
al gori thm onl y).
output.firstorderopt A measure of fi rst-order opti mal i ty
(l arge-scal e al gori thm onl y).
2x
1
x
2
e
x
1

=
x
1
2x
2
+ e
x
2
=
2x
1
x
2
e
x
1

0 =
x
1
2x
2
e
x
2
+ 0 =
fsolve
4-82
Next, cal l an opti mi zati on routi ne:
x0 = [5; 5]; % Make a starting guess at the solution
options=optimset('Display','iter'); % Option to display output
[x,fval] = fsolve('myfun',x0,options) % call optimizer
After 28 functi on eval uati ons, a zero i s found:
Optimization terminated successfully:
Relative function value changing by less than OPTIONS.TolFun
x =
0.5671
0.5671
fval =
1.0e08 *
0.5320
0.5320
Example 2: Fi nd a matri x x that sati sfi es the equati on
starti ng at the poi nt x= [1,1; 1,1].
Iteration Func-count f(x)
Norm of
step
First-order
optimality CG-iterations
1 4 47071.2 1 2.29e+004 0
2 7 6527.47 1.45207 3.09e+003 1
3 10 918.372 1.49186 418 1
4 13 127.74 1.55326 57.3 1
5 16 14.9153 1.57591 8.26 1
6 19 0.779051 1.27662 1.14 1
7 22 0.00372453 0.484658 0.0683 1
8 25 9.21617e-008 0.0385552 0.000336 1
9 28 5.66133e-017 0.000193707 8.34e-009 1
X X X
1 2
3 4
, =
fsolve
4-83
Fi rst, wri te an M-fi l e that computes the equati ons to be sol ved:
function F = myfun(x)
F = x*x*x[1,2;3,4];
Next, i nvoke an opti mi zati on routi ne:
x0 = ones(2,2); % Make a starting guess at the solution
options = optimset('Display','off'); % Turn off Display
[x,Fval,exitflag] = fsolve('myfun',x0,options)
The sol uti on i s
x =
0.1291 0.8602
1.2903 1.1612
Fval =
1.0e03 *
0.1541 0.1163
0.0109 0.0243
exitflag =
1
and the resi dual i s cl ose to zero
sum(sum(Fval.*Fval))
ans =
3.7974e008
Notes I f the system of equati ons i s l i near, then \ (the backsl ash operator: see
help slash) shoul d be used for better speed and accuracy. For exampl e, say we
want to fi nd the sol uti on to the fol l owi ng l i near system of equati ons:
3x
1
11x
2
2x
3
+ 7 =
x
1
x
2
2x
3
+ 4 =
x
1
x
2
x
3
+ 19 =
fsolve
4-84
Then the probl em i s formul ated and sol ved as
A = [ 3 11 2; 1 1 2; 1 1 1];
b = [ 7; 4; 19];
x = A\b
x =
13.2188
2.3438
3.4375
Algorithm The methods are based on the nonl i near l east-squares al gori thms al so used i n
lsqnonlin. The advantage of usi ng a l east-squares method i s that i f the system
of equati ons i s never zero due to smal l i naccuraci es, or because i t just does not
have a zero, the al gori thm sti l l returns a poi nt where the resi dual i s smal l .
However, i f the Jacobi an of the system i s si ngul ar, the al gori thm may converge
to a poi nt that i s not a sol uti on of the system of equati ons (see Limitations and
Diagnostics bel ow).
Large-scale optimization. By defaul t fsolve wi l l choose the l arge-scal e al gori thm.
The al gori thm i s a subspace trust regi on method and i s based on the
i nteri or-refl ecti ve Newton method descri bed i n [5],[6]. Each i terati on i nvol ves
the approxi mate sol uti on of a l arge l i near system usi ng the method of
precondi ti oned conjugate gradi ents (PCG). See the trust-regi on and
precondi ti oned conjugate gradi ent method descri pti ons i n the Large-Scale
Algorithms chapter.
Medium-scale optimization. fsolve wi th options.LargeScale set to 'off' uses
the Gauss-Newton method [4] wi th l i ne-search. Al ternati vel y, a
Levenberg-Marquardt method [1], [2], [3] wi th l i ne-search may be sel ected. The
choi ce of al gori thm i s made by setti ng options.LevenbergMarquardt. Setti ng
options.LevenbergMarquardt to 'on' (and options.LargeScale to 'off')
sel ects the Levenberg-Marquardt method.
The defaul t l i ne search al gori thm, i .e., options.LineSearchType set to
'quadcubic', i s a safeguarded mi xed quadrati c and cubi c pol ynomi al
i nterpol ati on and extrapol ati on method. A safeguarded cubi c pol ynomi al
method can be sel ected by setti ng options.LineSearchType to 'cubicpoly'.
Thi s method general l y requi res fewer functi on eval uati ons but more gradi ent
eval uati ons. Thus, i f gradi ents are bei ng suppl i ed and can be cal cul ated
fsolve
4-85
i nexpensi vel y, the cubi c pol ynomi al l i ne search method i s preferabl e. The
al gori thms used are descri bed ful l y i n the I ntroduction toAlgorithms chapter.
Diagnostics fsolve may converge to a nonzero poi nt and gi ve thi s message
Optimizer is stuck at a minimum that is not a root
Try again with a new starting guess
I n thi s case, run fsolve agai n wi th other starti ng val ues.
Limitations The functi on to be sol ved must be conti nuous. When successful , fsolve onl y
gi ves one root. fsolve may converge to a nonzero poi nt, i n whi ch case, try other
starti ng val ues.
fsolve onl y handl es real vari abl es. When x has compl ex vari abl es, the
vari abl es must be spl i t i nto real and i magi nary parts.
Large-scale optimization. Currentl y, i f the anal yti cal Jacobi an i s provi ded i n fun,
the options parameter DerivativeCheck cannot be used wi th the l arge-scal e
method to compare the anal yti c Jacobi an to the fi ni te-di fference Jacobi an.
I nstead, use the medi um-scal e method to check the deri vati ve wi th options
parameter MaxIter set to 0 i terati ons. Then run the probl em agai n wi th the
l arge-scal e method. See Tabl e 1-4 for more i nformati on on what probl em
formul ati ons are covered and what i nformati on must be provi ded.
The precondi ti oner computati on used i n the precondi ti oned conjugate gradi ent
part of the l arge-scal e method forms J
T
J (where J i s the Jacobi an matri x)
before computi ng the precondi ti oner; therefore, a row of J wi th many nonzeros,
whi ch resul ts i n a nearl y dense product J
T
J , may l ead to a costl y sol uti on
process for l arge probl ems.
See Also \, optimset, lsqnonlin, lsqcurvefit, inline
References [1] Levenber g, K., A Method for the Sol uti on of Cer tai n Pr obl ems i n Least
Squares, Quarterly Applied Mathematics 2, pp. 164-168, 1944.
[2] Marquardt, D., An Al gori thm for Leastsquares Esti mati on of Nonl i near
Parameters, SI AM J ournal Applied Mathematics, Vol . 11, pp. 431-441, 1963.
fsolve
4-86
[3] More, J. J., The LevenbergMarquardt Al gori thm: I mpl ementati on and
Theory, Numerical Analysis, ed. G. A. Watson, LectureNotes in Mathematics
630, Spri nger Verl ag, pp. 105-116, 1977.
[4] Denni s, J. E. Jr., Nonl i near Least Squares, Stateof theArt in Numerical
Analysis, ed. D. Jacobs, Academi c Press, pp. 269-312.
[5] Col eman, T.F. and Y. Li , On the Convergence of Refl ecti ve Newton
Methods for Large-Scal e Nonl i near Mi ni mi zati on Subject to Bounds,
Mathematical Programming, Vol . 67, Number 2, pp. 189-224, 1994.
[6] Col eman, T.F. and Y. Li , An I nteri or, Trust Regi on Approach for Nonl i near
Mi ni mi zati on Subject to Bounds, SI AM J ournal on Optimization, Vol . 6, pp.
418-445, 1996.
fzero
4-87
4fzer o
Purpose Zero of a functi on of one vari abl e
Syntax x = fzero(fun,x0)
x = fzero(fun,x0,options)
x = fzero(fun,x0,options,P1,P2,...)
[x,fval] = fzero(...)
[x,fval,exitflag] = fzero(...)
[x,fval,exitflag,output] = fzero(...)
Description x = fzero(fun,x0) tri es to fi nd a zero of fun near x0 i f x0 i s a scal ar. The
val ue x returned by fzero i s near a poi nt wher e fun changes si gn, or NaN i f the
search fai l s. I n thi s case, the search termi nates when the search i nterval i s
expanded unti l an Inf, NaN, or compl ex val ue i s found.
I f x0 i s a vector of l ength two, fzero assumes x0 i s an i nterval where the si gn
of fun(x0(1)) di ffers from the si gn of fun(x0(2)). An error occurs i f thi s i s not
true. Cal l i ng fzero wi th such an i nterval guarantees fzero wi l l return a val ue
near a poi nt where fun changes si gn.
x = fzero(fun,x0,options) mi ni mi zes wi th the opti mi zati on parameters
speci fi ed i n the structure options.
x = fzero(fun,x0,options,P1,P2,...) provi des for addi ti onal arguments,
P1, P2, etc., whi ch are passed to the objecti ve functi on, fun. Use options=[] as
a pl acehol der i f no opti ons are set.
[x,fval] = fzero(...) returns the val ue of the objecti ve functi on fun at the
sol uti on x.
[x,fval,exitflag] = fzero(...) returns a val ue exitflag that descri bes
the exi t condi ti on.
[x,fval,exitflag,output] = fzero(...) returns a structure output that
contai ns i nformati on about the opti mi zati on.
Note: For the purposes of thi s command, zeros are consi dered to be poi nts
where the functi on actual l y crosses, not just touches, the x-axi s.
fzero
4-88
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to fzero are i ncl uded bel ow for fun, options, exitflag, and output.
fun The functi on to be mi ni mi zed. fun takes a scal ar x and returns a
scal ar val ue f of the objecti ve functi on eval uated at x. You can
speci fy fun to be an i nl i ne object. For exampl e,
x = fzero(inline('sin(x*x)'),x0)
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function f = myfun(x)
f = ... % Compute function value at x
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on. fzero uses these
options structure fi el ds:
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
TolX Termi nati on tol erance on x.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that fzero found a zero x
< 0 then no i nterval was found wi th a si gn change, or a NaN or
Inf functi on val ue was encountered duri ng the search for an
i nterval contai ni ng a si gn change, or a compl ex functi on val ue
was encountered duri ng the search for an i nterval contai ni ng a
si gn change.
fzero
4-89
Examples Cal cul ate by fi ndi ng the zero of the si ne functi on near 3.
x = fzero('sin',3)
x =
3.1416
To fi nd the zero of cosi ne between 1 and 2:
x = fzero('cos',[1 2])
x =
1.5708
Note that cos(1) and cos(2) di ffer i n si gn.
To fi nd a zero of the functi on
wri te an M-fi l e cal l ed f.m.
function y = f(x)
y = x.^32*x5;
To fi nd the zero near 2
z = fzero('f',2)
z =
2.0946
Si nce thi s functi on i s a pol ynomi al , the statement roots([1 0 2 5]) fi nds
the same real zero, and a compl ex conjugate pai r of zeros.
2.0946
1.0473 + 1.1359i
1.0473 1.1359i
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken (for fzero,
thi s i s the same as the number of functi on eval uati ons).
output.algorithm The al gori thm used.
output.funcCount The number of functi on eval uati ons.
f x ( ) x
3
2x 5 =
fzero
4-90
Notes Cal l i ng fzero wi th an i nterval (x0 wi th two el ements) i s often faster than
cal l i ng i t wi th a scal ar x0.
Algorithm The fzero command i s an M-fi l e. The al gori thm, whi ch was ori gi nated by T.
Dekker, uses a combi nati on of bi secti on, secant, and i nverse quadrati c
i nterpol ati on methods. An Al gol 60 versi on, wi th some i mprovements, i s gi ven
i n [1]. A Fortran versi on, upon whi ch the fzero M-fi l e i s based, i s i n [2].
Limitations The fzero command defi nes a zeroas a poi nt where the functi on crosses the
x-axi s. Poi nts where the functi on touches, but does not cross, the x-axi s are not
val i d zeros. For exampl e, y = x.^2 i s a parabol a that touches the x-axi s at 0.
Si nce the functi on never crosses the x-axi s, however, no zero i s found. For
functi ons wi th no val i d zeros, fzero executes unti l Inf, NaN, or a compl ex val ue
i s detected.
See Also roots, fminbnd, fsolve, \, inline, optimset
References [1] Brent, R., Algorithms for Minimization Without Derivatives, Prenti ce-Hal l ,
1973.
[2] Forsythe, G. E., M. A. Mal col m, and C. B. Mol er, Computer Methods for
Mathematical Computations, Prenti ce-Hal l , 1976.
linprog
4-91
4l i npr og
Purpose Sol ve a l i near programmi ng probl em
where f, x, b, beq, lb, and ub are vectors and A and Aeq are matri ces.
Syntax x = linprog(f,A,b,Aeq,beq)
x = linprog(f,A,b,Aeq,beq,lb,ub)
x = linprog(f,A,b,Aeq,beq,lb,ub,x0)
x = linprog(f,A,b,Aeq,beq,lb,ub,x0,options)
[x,fval] = linprog(...)
[x,fval,exitflag] = linprog(...)
[x,fval,exitflag,output] = linprog(...)
[x,fval,exitflag,output,lambda] = linprog(...)
Description linprog sol ves l i near programmi ng probl ems.
x = linprog(f,A,b) sol ves mi n f'*x such that A*x <= b.
x = linprog(f,A,b,Aeq,beq) sol ves the probl em above whi l e addi ti onal l y
sati sfyi ng the equal i ty constrai nts Aeq*x = beq. Set A=[] and b=[] i f no
i nequal i ti es exi st.
x = linprog(f,A,b,Aeq,beq,lb,ub) defi nes a set of l ower and upper bounds
on the desi gn vari abl es, x, so that the sol uti on i s al ways i n the range
lb <= x <= ub. Set Aeq=[] and beq=[] i f no equal i ti es exi st.
x = linprog(f,A,b,Aeq,beq,lb,ub,x0) sets the starti ng poi nt to x0. Thi s
opti on i s onl y avai l abl e wi th the medi um-scal e al gori thm
(options.LargeScale i s 'off'). The defaul t l arge-scal e al gori thm wi l l i gnore
any starti ng poi nt.
x = linprog(f,A,b,Aeq,beq,lb,ub,x0,options) mi ni mi zes wi th the
opti mi zati on parameters speci fi ed i n the structure options.
f
T
x
x
min such that A x b
Aeq x beq =
l b x ub
linprog
4-92
[x,fval] = linprog(...) returns the val ue of the objecti ve functi on fun at
the sol uti on x: fval = f'*x.
[x,lambda,exitflag] = linprog(...) returns a val ue exitflag that
descri bes the exi t condi ti on.
[x,lambda,exitflag,output] = linprog(...) returns a structure output
that contai ns i nformati on about the opti mi zati on.
[x,fval,exitflag,output,lambda] = linprog(...) returns a structure
lambda whose fi el ds contai n the Lagrange mul ti pl i ers at the sol uti on x.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to linprog are i ncl uded bel ow for options, exitflag, lambda, and
output.
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on.
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output. At thi s ti me the 'iter' l evel onl y works wi th the
l arge-scal e al gori thm.
LargeScale Use l arge-scal e al gori thm i f possi bl e when set to
'on'. Use medi um-scal e al gori thm when set to 'off'.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
linprog
4-93
Examples Fi nd x that mi ni mi zes
subject to
Fi rst, enter the coeffi ci ents:
f = [5; 4; 6]
A = [1 1 1
3 2 4
3 2 0];
b = [20; 42; 30];
lb = zeros(3,1);
Next, cal l a l i near programmi ng routi ne:
[x,fval,exitflag,output,lambda] = linprog(f,A,b,[],[],lb);
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb.
lambda.upper for the upper bounds ub.
lambda.ineqlin for the l i near i nequal i ti es.
lambda.eqlin for the l i near equal i ti es.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.algorithm The al gori thm used.
output.cgiterations The number of PCG i terati ons
(l arge-scal e al gori thm onl y).
f x ( ) 5x
1
4x
2
6x
3
=
x
1
x
2
x
3
+ 20
3x
1
2x
2
4x
3
+ + 42
3x
1
2x
2
+ 30
0 x
1
0 x
2
0 x
3
, ,
linprog
4-94
Enteri ng x, lambda.ineqlin, and lambda.lower gets
x =
0.0000
15.0000
3.0000
lambda.ineqlin =
0
1.5000
0.5000
lambda.lower =
1.0000
0
0
Nonzero el ements of the vectors i n the fi el ds of lambda i ndi cate acti ve
constrai nts at the sol uti on. I n thi s case, the second and thi rd i nequal i ty
constrai nts (i n lambda.ineqlin) and the fi rst l ower bound constrai nt (i n
lambda.lower) are acti ve constrai nts (i .e., the sol uti on i s on thei r constrai nt
boundari es).
Algorithm Large-scale optimization. The l arge-scal e method i s based on LI PSOL ([3]), whi ch
i s a vari ant of Mehrotras predi ctor-corrector al gori thm ([2]), a pri mal -dual
i nteri or-poi nt method. A number of preprocessi ng steps occur before the
al gori thm begi ns to i terate. See the l i near programmi ng secti on i n the
Large-ScaleAlgorithms chapter.
Medium-scale optimization. linprog uses a projecti on method as used i n the
quadprog al gori thm. linprog i s an acti ve set method and i s thus a vari ati on of
the wel l -known simplex method for l i near programmi ng [1]. I t fi nds an i ni ti al
feasi bl e sol uti on by fi rst sol vi ng another l i near programmi ng probl em.
Diagnostics Large-scale optimization. The fi rst stage of the al gori thm may i nvol ve some
preprocessi ng of the constrai nts (see the Large-Scal e Li near Programmi ng
secti on of the Large-Scal e Al gori thms chapter). Several possi bl e condi ti ons
mi ght occur that cause linprog to exi t wi th an i nfeasi bi l i ty message. I n each
case, the exitflag argument returned by linprog wi l l be set to a negati ve
val ue to i ndi cate fai l ure.
linprog
4-95
I f a row of al l zer os i s detected i n Aeq but the correspondi ng el ement of beq i s
not zero, the exi t message i s
Exiting due to infeasibility: an all zero row in the constraint
matrix does not have a zero in corresponding right hand size
entry.
I f one of the el ements of x i s found to not be bounded bel ow, the exi t message i s
Exiting due to infeasibility: objective f'*x is unbounded below.
I f one of the rows of Aeq has onl y one nonzero el ement, the associ ated val ue i n
x i s cal l ed a singleton var i abl e. I n thi s case, the val ue of that component of x
can be computed from Aeq and beq. I f the val ue computed vi ol ates another
constrai nt, the exi t message i s
Exiting due to infeasibility: Singleton variables in equality
constraints are not feasible.
I f the si ngl eton vari abl e can be sol ved for but the sol uti on vi ol ates the upper or
l ower bounds, the exi t message i s
Exiting due to infeasibility: singleton variables in the equality
constraints are not within bounds.
Note: The preprocessi ng steps are cumul ati ve. For exampl e, even i f your
constrai nt matri x does not have a row of al l zeros to begi n wi th, other
preprocessi ng steps may cause such a row to occur.
Once the preprocessi ng has fi ni shed, the i terati ve part al gori thm begi ns unti l
the stoppi ng cri teri a i s met. (See the Large-Scal e Li near Programmi ng
secti on of the Large-ScaleAlgorithms chapter for more i nformati on about
resi dual s, the pri mal probl em, the dual probl em, and the rel ated stoppi ng
cri teri a.) I f the resi dual s are growi ng i nstead of getti ng smal l er, or the
resi dual s are nei ther growi ng nor shri nki ng, one of the two fol l owi ng
termi nati on messages wi l l di spl ay, respecti vel y,
One or more of the residuals, duality gap, or total relative error
has grown 100000 times greater than its minimum value so far:
linprog
4-96
or
One or more of the residuals, duality gap, or total relative error
has stalled:
After one of these messages di spl ays, i t wi l l be fol l owed by one of the fol l owi ng
si x messages i ndi cati ng i f i t appears the dual , the pri mal , or both are
i nfeasi bl e. The messages di ffer accordi ng to how the i nfeasi bi l i ty or
unboundedness was measured.
The dual appears to be infeasible (and the primal unbounded).(The
primal residual < TolFun)
The primal appears to be infeasible (and the dual unbounded). The
dual residual < TolFun)
The dual appears to be infeasible (and the primal unbounded) since
the dual residual > sqrt(TolFun).(The primal residual <
10*TolFun)
The primal appears to be infeasible (and the dual unbounded) since
the primal residual > sqrt(TolFun).(The dual residual <
10*TolFun)
The dual appears to be infeasible and the primal unbounded since
the primal objective < 1e+10 and the dual objective < 1e+6.
The primal appears to be infeasible and the dual unbounded since
the dual objective > 1e+10 and the primal objective > 1e+6.
Both the primal and the dual appear to be infeasible.
Note that, for exampl e, the pri mal (objecti ve) can be unbounded and the pri mal
resi dual , whi ch i s a measure of pri mal constrai nt sati sfacti on, can be smal l .
linprog
4-97
Medium-scale optimization. linprog gi ves a warni ng when the sol uti on i s
i nfeasi bl e:
Warning: The constraints are overly stringent;
there is no feasible solution.
I n thi s case, linprog produces a resul t that mi ni mi zes the worst case
constrai nt vi ol ati on.
When the equal i ty constrai nts are i nconsi stent, linprog gi ves
Warning: The equality constraints are overly
stringent; there is no feasible solution.
Unbounded sol uti ons resul t i n the warni ng
Warning: The solution is unbounded and at infinity;
the constraints are not restrictive enough
I n thi s case, linprog returns a val ue of x that sati sfi es the constrai nts.
Limitations Medium-scale optimization. At thi s ti me, the onl y l evel s of di spl ay, usi ng the
Display parameter i n options, are 'off' and 'final'; i terati ve output usi ng
'iter' i s not avai l abl e.
See Also quadprog
References [1] Dantzi g, G.B., A. Orden, and P. Wol fe, General i zed Si mpl ex Method for
Mi ni mi zi ng a Li near from Under Li near I nequal i ty Constrai nts, Pacific
J ournal Math. Vol . 5, pp. 183195.
[2] Mehrotra, S., On the I mpl ementati on of a Pri mal -Dual I nteri or Poi nt
Method, SI AM J ournal on Optimization, Vol . 2, pp. 575-601, 1992.
[3] Zhang, Y., Sol vi ng Large-Scal e Li near Programs by I nteri or-Poi nt Methods
Under the MATLAB Envi ronment, Technical Report TR96-01, Department of
Mathemati cs and Stati sti cs, Uni versi ty of Maryl and, Bal ti more County,
Bal ti more, MD, Jul y 1995.
lsqcurvefit
4-98
4l sqcur vefi t
Purpose Sol ve nonl i near curve-fi tti ng (data-fi tti ng)probl ems i n the l east-squares sense.
That i s, gi ven i nput data xdata, and the obser ved output ydata, fi nd coeffi ci ents
x that best-fi t the equati on F(x, xdata)
where xdata and ydata are vectors and F(x, xdata) i s a vector val ued functi on.
The functi on lsqcurvefit uses the same al gori thm as lsqnonlin. I ts purpose
i s to provi de an i nterface desi gned speci fi cal l y for data-fi tti ng probl ems.
Syntax x = lsqcurvefit(fun,x0,xdata,ydata)
x = lsqcurvefit(fun,x0,xdata,ydata,lb,ub)
x = lsqcurvefit(fun,x0,xdata,ydata,lb,ub,options)
x = lsqcurvefit(fun,x0,xdata,ydata,lb,ub,options,P1,P2,...)
[x,resnorm] = lsqcurvefit(...)
[x,resnorm,residual] = lsqcurvefit(...)
[x,resnorm,residual,exitflag] = lsqcurvefit(...)
[x,resnorm,residual,exitflag,output] = lsqcurvefit(...)
[x,resnorm,residual,exitflag,output,lambda] = lsqcurvefit(...)
[x,resnorm,residual,exitflag,output,lambda,jacobian] =
lsqcurvefit(...)
Description lsqcurvefit sol ves nonl i near data-fi tti ng probl ems. lsqcurvefit requi res a
user-defi ned functi on to compute the vector-val ued functi on F(x, xdata). The
si ze of the vector returned by the user-defi ned functi on must be the same as the
si ze of ydata.
x = lsqcurvefit(fun,x0,xdata,ydata) starts at x0 and fi nds coeffi ci ents x
to best fi t the nonl i near functi on fun(x,xdata) to the data ydata (i n the
l east-squares sense). ydata must be the same si ze as the vector (or matri x) F
returned by fun.
x = lsqcurvefit(fun,x0,xdata,ydata,lb,ub) defi nes a set of l ower and
upper bounds on the desi gn vari abl es, x, so that the sol uti on i s al ways i n the
range lb <= x <= ub.
1
2
--- F x xdata , ( ) ydata
2
2
x
min
1
2
--- F x xdata
i
, ( ) ydata
i
( )
2
i

=
lsqcurvefit
4-99
x = lsqcurvefit(fun,x0,xdata,ydata,lb,ub,options) mi ni mi zes wi th the
opti mi zati on parameters speci fi ed i n the structure options.
x = lsqcurvefit(fun,x0,xdata,ydata,lb,ub,options,P1,P2,...) passes
the probl em-dependent parameters P1, P2, etc., di rectl y to the functi on fun.
Pass an empty matri x for options to use the defaul t val ues for options.
[x,resnorm] = lsqcurvefit(...) returns the val ue of the squared 2-norm of
the resi dual at x: sum{(fun(x,xdata)ydata).^2}.
[x,resnorm,residual] = lsqcurvefit(...) returns the val ue of the
resi dual , fun(x,xdata)ydata, at the sol uti on x.
[x,resnorm,residual,exitflag] = lsqcurvefit(...) returns a val ue
exitflag that descri bes the exi t condi ti on.
[x,resnorm,residual,exitflag,output] = lsqcurvefit(...) returns a
structure output that contai ns i nformati on about the opti mi zati on.
[x,resnorm,residual,exitflag,output,lambda] = lsqcurvefit(...)
returns a structure lambda whose fi el ds contai n the Lagrange mul ti pl i ers at
the sol uti on x.
[x,resnorm,residual,exitflag,output,lambda,jacobian] = 99
99lsqcurvefit(...) returns the Jacobi an of fun at the sol uti on x.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
lsqcurvefit
4-100
rel evant to lsqcurvefit are i ncl uded bel ow for fun, options, exitflag,
lambda, and output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
vector F of the objecti ve functi ons eval uated at x. You can speci fy
fun to be an i nl i ne object wi th two i nput parameters x and xdata.
For exampl e,
f = ...
inline('x(1)*xdata.^2+x(2)*sin(xdata)','x','xdata');
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function F = myfun(x,xdata)
F = ... % Compute function values at x
I f the Jacobi an can al so be computed and options.Jacobian i s
'on', set by
options = optimset('Jacobian','on')
then the functi on fun must return, i n a second output argument,
the Jacobi an val ue J, a matri x, at x. Note that by checki ng the
val ue of nargout the functi on can avoi d computi ng J when fun i s
cal l ed wi th onl y one output argument (i n the case where the
opti mi zati on al gori thm onl y needs the val ue of F but not J):
function [F,J] = myfun(x,xdata)
F = ... % objective function values at x
if nargout > 1 % two output arguments
J = ... % Jacobian of the function evaluated at x
end
I f fun returns a vector (matri x) of m components and x has l ength
n, then the Jacobi an J i s an m-by-n matri x where J(i,j) i s the
parti al deri vati ve of F(i) wi th respect to x(j). (Note that the
Jacobi an J i s the transpose of the gradi ent of F.)
lsqcurvefit
4-101
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on. Some
parameters appl y to al l al gori thms, some are onl y rel evant when
usi ng the l arge-scal e al gori thm, and others are onl y rel evant when
usi ng the medi um-scal e al gori thm.
We start by descri bi ng the LargeScale opti on si nce i t states a
preferencefor whi ch al gori thm to use. I t i s onl y a preference si nce
certai n condi ti ons must be met to use the l arge-scal e or
medi um-scal e al gori thm. For the l arge-scal e al gori thm, the
nonl i near system of equati ons cannot be under-determi ned; that i s,
the number of equati ons (the number of el ements of F returned by
fun) must be at l east as many as the l ength of x. Furthermore, onl y
the l arge-scal e al gori thm handl es bound constrai nts.
LargeScale Use l arge-scal e al gori thm i f possi bl e when set to
'on'. Use medi um-scal e al gori thm when set to 'off'.
Parameters used by both the l arge-scal e and medi um-scal e
al gori thms:
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
Jacobian Jacobi an for the objecti ve functi on defi ned by user.
See the descri pti on of fun above to see how to defi ne the Jacobi an
i n fun.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
lsqcurvefit
4-102
Parameters used by the l arge-scal e al gori thm onl y:
JacobPattern Sparsi ty pattern of the Jacobi an for
fi ni te-di fferenci ng. I f i t i s not conveni ent to compute the
Jacobi an matri x J i n fun, lsqcurvefit can approxi mate J vi a
sparse fi ni te-di fferences provi ded the structure of J, i .e.,
l ocati ons of the nonzeros, i s suppl i ed as the val ue for
JacobPattern. I n the worst case, i f the structure i s unknown,
you can set JacobPattern to be a dense matri x and a ful l
fi ni te-di fference approxi mati on wi l l be computed i n each
i terati on (thi s i s the defaul t i f JacobPattern i s not set). Thi s can
be very expensi ve for l arge probl ems so i t i s usual l y worth the
effort to determi ne the sparsi ty structure.
MaxPCGIter Maxi mum number of PCG (precondi ti oned
conjugate gradi ent) i terati ons (see the Algorithmsecti on bel ow).
PrecondBandWidth Upper bandwi dth of precondi ti oner for
PCG. By defaul t, di agonal precondi ti oni ng i s used (upper
bandwi dth of 0). For some probl ems, i ncreasi ng the bandwi dth
reduces the number of PCG i terati ons.
TolPCG Termi nati on tol erance on the PCG i terati on.
TypicalX Typi cal x val ues.
Parameters used by the medi um-scal e al gori thm onl y:
DerivativeCheck Compare user-suppl i ed deri vati ves
(Jacobi an) to fi ni te-di fferenci ng deri vati ves.
DiffMaxChange Maxi mum change i n vari abl es for
fi ni te-di fferenci ng.
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te-di fferenci ng.
LevenbergMarquardt Choose Levenberg-Marquardt over
Gauss-Newton al gori thm.
LineSearchType Li ne search al gori thm choi ce.
lsqcurvefit
4-103
Note: The sum of squares shoul d not be formed expl i ci tl y. I nstead, your
functi on shoul d return a vector of functi on val ues. See the exampl es bel ow.
Examples Vectors of data xdata and ydata are of l ength n. You want to fi nd coeffi ci ents x
to fi nd the best fi t to the equati on
that i s, you want to mi ni mi ze
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb.
lambda.upper for the upper bounds ub.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.funcCount The number of functi on eval uati ons.
output.algorithm The al gori thm used.
output.cgiterations The number of PCG i terati ons
(l arge-scal e al gori thm onl y).
output.stepsize The fi nal step si ze taken (medi um-scal e
al gori thm onl y).
output.firstorderopt A measure of fi rst-order opti mal i ty
(l arge-scal e al gori thm onl y).
ydata i ( ) x 1 ( ) xdata i ( )
2
x 2 ( ) xdata i ( ) ( ) si n x 3 ( ) xdata i ( )
3
+ + =
lsqcurvefit
4-104
where F(x,xdata) = x(1)*xdata.^2 + x(2)*sin(xdata) + x(3)*xdata.^3,
starti ng at the poi nt x0 = [0.3, 0.4, 0.1].
Fi rst, wri te an M-fi l e to return the val ue of F (F has n components):
function F = myfun(x,xdata)
F = x(1)*xdata.^2 + x(2)*sin(xdata) + x(3)*xdata.^3;
Next, i nvoke an opti mi zati on routi ne:
% Assume you determined xdata and ydata experimentally
xdata = [3.6 7.7 9.3 4.1 8.6 2.8 1.3 7.9 10.0 5.4];
ydata = [16.5 150.6 263.1 24.7 208.5 9.9 2.7 163.9 325.0 54.3];
x0 = [10, 10, 10] % Starting guess
[x,resnorm] = lsqcurvefit('myfun',x0,xdata,ydata)
Note that at the ti me that lsqcurvefit i s cal l ed, xdata and ydata are assumed
to exi st and are vectors of the same si ze. They must be the same si ze because
the val ue F returned by fun must be the same si ze as ydata.
After 33 functi on eval uati ons, thi s exampl e gi ves the sol uti on:
x =
0.2269 0.3385 0.3021
% residual or sum of squares
resnorm =
6.2950
The resi dual i s not zero because i n thi s case there was some noi se
(experi mental error) i n the data.
Algorithm Large-scale optimization. By defaul t lsqcurvefit wi l l choose the l arge-scal e
al gori thm. Thi s al gori thm i s a subspace trust regi on method and i s based on
the i nteri or-refl ecti ve Newton method descri bed i n [5], [6]. Each i terati on
i nvol ves the approxi mate sol uti on of a l arge l i near system usi ng the method of
precondi ti oned conjugate gradi ents (PCG). See the trust-regi on and
precondi ti oned conjugate gradi ent method descri pti ons i n the Large-Scale
Algorithms chapter.
x
min
1
2
--- F x xdata
i
, ( ) ydata
i
( )
2
i 1 =
n

lsqcurvefit
4-105
Medium-scale optimization. lsqcurvefit wi th options.LargeScale set to 'off'
uses the Levenberg-Marquardt method wi th l i ne-search [1], [2], [3].
Al ternati vel y, a Gauss-Newton method [4] wi th l i ne-search may be sel ected.
The choi ce of al gori thm i s made by setti ng options.LevenbergMarquardt.
Setti ng options.LevenbergMarquardt to 'off' (and options.LargeScale to
'off') sel ects the Gauss-Newton method, whi ch i s general l y faster when the
resi dual i s smal l .
The defaul t l i ne search al gori thm, i .e., options.LineSearchType set to
'quadcubic', i s a safeguarded mi xed quadrati c and cubi c pol ynomi al
i nterpol ati on and extrapol ati on method. A safeguarded cubi c pol ynomi al
method can be sel ected by setti ng options.LineSearchType to 'cubicpoly'.
Thi s method general l y requi res fewer functi on eval uati ons but more gradi ent
eval uati ons. Thus, i f gradi ents are bei ng suppl i ed and can be cal cul ated
i nexpensi vel y, the cubi c pol ynomi al l i ne search method i s preferabl e. The
al gori thms used are descri bed ful l y i n the I ntroduction toAlgorithms chapter.
Diagnostics Large-scale optimization. The l arge-scal e code wi l l not al l ow equal upper and
l ower bounds. For exampl e i f lb(2)==ub(2) then lsqlin gi ves the error
Equal upper and lower bounds not permitted.
(lsqcurvefit does not handl e equal i ty constrai nts, whi ch i s another way to
formul ate equal bounds. I f equal i ty constrai nts are present, use fmincon,
fminimax, or fgoalattain for al ternati ve formul ati ons where equal i ty
constrai nts can be i ncl uded.)
Limitations The functi on to be mi ni mi zed must be conti nuous. lsqcurvefit may onl y gi ve
l ocal sol uti ons.
lsqcurvefit onl y handl es real vari abl es (the user-defi ned functi on must onl y
return real val ues). When x has compl ex vari abl es, the vari abl es must be spl i t
i nto real and i magi nary parts.
Large-scale optimization. The l arge-scal e method for lsqcurvefit does not sol ve
underdetermi ned systems: i t requi res that the number of equati ons, i .e., row
di mensi on of F, be at l east as great as the number of vari abl es. I n the
underdetermi ned case, the medi um-scal e al gori thm wi l l be used i nstead. See
Tabl e 1-4 for more i nformati on on what probl em formul ati ons are covered and
what i nformati on must be provi ded.
F x ( )
2
2
lsqcurvefit
4-106
The precondi ti oner computati on used i n the precondi ti oned conjugate gradi ent
part of the l arge-scal e method forms J
T
J (where J i s the Jacobi an matri x)
before computi ng the precondi ti oner; therefore, a row of J wi th many nonzeros,
whi ch resul ts i n a nearl y dense product J
T
J , may l ead to a costl y sol uti on
process for l arge probl ems.
I f components of xhave no upper (or l ower) bounds, then lsqcurvefit prefers
that the correspondi ng components of ub (or lb) be set to inf (or inf for l ower
bounds) as opposed to an arbi trary but very l arge posi ti ve (or negati ve for
l ower bounds) number.
Currentl y, i f the anal yti cal Jacobi an i s provi ded i n fun, the options parameter
DerivativeCheck cannot be used wi th the l arge-scal e method to compare the
anal yti c Jacobi an to the fi ni te-di fference Jacobi an. I nstead, use the
medi um-scal e method to check the deri vati ves wi th options parameter
MaxIter set to zero i terati ons. Then run the probl em wi th the l arge-scal e
method.
See Also optimset, lsqlin, lsqnonlin, lsqnonneg, \
References [1] Levenberg, K., A Method for the Sol uti on of Certai n Probl ems i n Least
Squares, QuarterlyApplied Math. 2, pp. 164-168, 1944.
[2] Marquardt, D., An Al gori thm for Leastsquares Esti mati on of Nonl i near
Parameters, SI AM J ournal Applied Math. Vol . 11, pp. 431-441, 1963.
[3] More, J. J., The LevenbergMarquardt Al gori thm: I mpl ementati on and
Theory, Numerical Analysis, ed. G. A. Watson, Lecture Notes i n Mathemati cs
630, Spri nger Verl ag, pp. 105-116, 1977.
[4] Denni s, J. E. Jr., Nonl i near Least Squares, Stateof theArt in Numerical
Analysis, ed. D. Jacobs, Academi c Press, pp. 269-312, 1977.
[5] Col eman, T.F. and Y. Li , On the Convergence of Refl ecti ve Newton
Methods for Large-Scal e Nonl i near Mi ni mi zati on Subject to Bounds,
Mathematical Programming, Vol . 67, Number 2, pp. 189-224, 1994.
[6] Col eman, T.F. and Y. Li , An I nteri or, Trust Regi on Approach for Nonl i near
Mi ni mi zati on Subject to Bounds, SI AM J ournal on Optimization, Vol . 6, pp.
418-445, 1996.
lsqlin
4-107
4l sql i n
Purpose Sol ve the constrai ned l i near l east-squares probl em
where C, A, and Aeq are matri ces and d, b, beq, lb, ub, and x are vectors.
Syntax x = lsqlin(C,d,A,b)
x = lsqlin(C,d,A,b,Aeq,beq)
x = lsqlin(C,d,A,b,Aeq,beq,lb,ub)
x = lsqlin(C,d,A,b,Aeq,beq,lb,ub,x0)
x = lsqlin(C,d,A,b,Aeq,beq,lb,ub,x0,options)
[x,resnorm] = lsqlin(...)
[x,resnorm,residual] = lsqlin(...)
[x,resnorm,residual,exitflag] = lsqlin(...)
[x,resnorm,residual,exitflag,output] = lsqlin(...)
[x,resnorm,residual,exitflag,output,lambda] = lsqlin(...)
Description x = lsqlin(C,d,A,b) sol ves the l i near system C*x=d i n the l east-squares
sense subject to A*x<=b, where C i s m-by-n.
x = lsqlin(C,d,A,b,Aeq,beq) sol ves the probl em above whi l e addi ti onal l y
sati sfyi ng the equal i ty constrai nts Aeq*x = beq. Set A=[] and b=[] i f no
i nequal i ti es exi st.
x = lsqlin(C,d,A,b,Aeq,beq,lb,ub) defi nes a set of l ower and upper bounds
on the desi gn vari abl es, x, so that the sol uti on i s al ways i n the range
lb <= x <= ub. Set Aeq=[] and beq=[] i f no equal i ti es exi st.
x = lsqlin(C,d,A,b,Aeq,beq,lb,ub,x0) sets the starti ng poi nt to x0.
x = lsqlin(C,d,A,b,Aeq,beq,lb,ub,x0,options) mi ni mi zes wi th the
opti mi zati on parameters speci fi ed i n the structure options.
[x,resnorm] = lsqlin(...) returns the val ue of the squared 2-norm of the
resi dual : norm(C*xd)^2.
1
2
--- Cx d
2
x
min
2
such that A x b
Aeq x beq =
l b x ub
lsqlin
4-108
[x,resnorm,residual] = lsqlin(...) returns the resi dual , C*xd.
[x,resnorm,residual,exitflag] = lsqlin(...) returns a val ue exitflag
that descri bes the exi t condi ti on.
[x,resnorm,residual,exitflag,output] = lsqlin(...) returns a
structure output that contai ns i nformati on about the opti mi zati on.
[x,resnorm,residual,exitflag,output,lambda] = lsqlin(...) returns a
structure lambda whose fi el ds contai n the Lagrange mul ti pl i ers at the sol uti on
x.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to lsqlin are i ncl uded bel ow for options, exitflag, lambda, and
output.
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameter s usi ng the optimset functi on. Some parameters
appl y to al l al gori thms, some are onl y rel evant when usi ng the
l arge-scal e al gori thm, and others are onl y rel evant when usi ng the
medi um-scal e al gori thm.
We start by descri bi ng the LargeScale opti on si nce i t states a
preferencefor whi ch al gori thm to use. I t i s onl y a preference si nce
certai n condi ti ons must be met to use the l arge-scal e al gori thm.
For lsqlin, when the probl em has only upper and l ower bounds,
i .e., no l i near i nequal i ti es or equal i ti es are speci fi ed, the defaul t
al gori thm i s the l arge-scal e method. Otherwi se the medi um-scal e
al gori thm wi l l be used.
The parameter to set an al gori thm preference:
LargeScale Use l arge-scal e al gori thm i f possi bl e when set to
'on'. Use medi um-scal e al gori thm when set to 'off'.
lsqlin
4-109
Parameters used by both the l arge-scal e and medi um-scal e
al gori thms:
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
Parameters used by the l arge-scal e al gori thm onl y:
MaxPCGIter Maxi mum number of PCG (precondi ti oned
conjugate gradi ent) i terati ons (see the Algorithmsecti on bel ow).
PrecondBandWidth Upper bandwi dth of precondi ti oner for
PCG. By defaul t, di agonal precondi ti oni ng i s used (upper
bandwi dth of 0). For some probl ems, i ncreasi ng the bandwi dth
reduces the number of PCG i terati ons.
TolPCG Termi nati on tol erance on the PCG i terati on.
TypicalX Typi cal x val ues.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb.
lambda.upper for the upper bounds ub.
lambda.ineqlin for the l i near i nequal i ti es.
lambda.eqlin for the l i near equal i ti es.
lsqlin
4-110
Examples Fi nd the l east-squares sol uti on to the over-determi ned system
subject to and .
Fi rst, enter the coeffi ci ent matri ces and the l ower and upper bounds:
C = [
0.9501 0.7620 0.6153 0.4057
0.2311 0.4564 0.7919 0.9354
0.6068 0.0185 0.9218 0.9169
0.4859 0.8214 0.7382 0.4102
0.8912 0.4447 0.1762 0.8936];
d = [
0.0578
0.3528
0.8131
0.0098
0.1388];
A =[
0.2027 0.2721 0.7467 0.4659
0.1987 0.1988 0.4450 0.4186
0.6037 0.0152 0.9318 0.8462];
b =[
0.5251
0.2026
0.6721];
lb = 0.1*ones(4,1);
ub = 2*ones(4,1);
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.algorithm The al gori thm used.
output.cgiterations The number of PCG i terati ons
(l arge-scal e al gori thm onl y).
output.firstorderopt A measure of fi rst-order opti mal i ty
(l arge-scal e al gori thm onl y).
C x d =
A x b l b x ub
lsqlin
4-111
Next, cal l the constrai ned l i near l east-squares routi ne:
[x,resnorm,residual,exitflag,output,lambda] = ...
lsqlin(C,d,A,b,[ ],[ ],lb,ub);
Enteri ng x, lambda.ineqlin, lambda.lower, lambda.upper gets
x =
0.1000
0.1000
0.2152
0.3502
lambda.ineqlin =
0
0.2392
0
lambda.lower =
0.0409
0.2784
0
0
lambda.upper =
0
0
0
0
Nonzero el ements of the vectors i n the fi el ds of lambda i ndi cate acti ve
constrai nts at the sol uti on. I n thi s case, the second i nequal i ty constrai nt (i n
lambda.ineqlin) and the fi rst l ower and second l ower bound constrai nts (i n
lambda.lower) are acti ve constrai nts (i .e., the sol uti on i s on thei r constrai nt
boundari es).
Notes For probl ems wi th no constrai nts, \ shoul d be used: x= A\b.
I n general lsqlin l ocates a l ocal sol uti on.
Better numeri cal resul ts are l i kel y i f you speci fy equal i ti es expl i ci tl y usi ng Aeq
and beq, i nstead of i mpl i ci tl y usi ng lb and ub.
Large-scale optimization. I f x0 i s not stri ctl y feasi bl e, lsqlin chooses a new
stri ctl y feasi bl e (centered) starti ng poi nt.
lsqlin
4-112
I f components of x have no upper (or l ower) bounds, then lsqlin prefers that
the correspondi ng components of ub (or lb) be set to Inf (or Inf for lb) as
opposed to an arbi trary but very l arge posi ti ve (or negati ve i n the case of l ower
bounds) number.
Algorithm Large-scale optimization. When the probl em gi ven to l sql i n has only upper and
l ower bounds, i .e., no l i near i nequal i ti es or equal i ti es are speci fi ed, and the
matri x C has at l east as many rows as col umns, the defaul t al gori thm i s the
l arge-scal e method. Thi s method i s a subspace trust-regi on method based on
the i nteri or-refl ecti ve Newton method descri bed i n [2]. Each i terati on i nvol ves
the approxi mate sol uti on of a l arge l i near system usi ng the method of
precondi ti oned conjugate gradi ents (PCG). See the trust-regi on and
precondi ti oned conjugate gradi ent method descri pti ons i n the Large-Scale
Algorithms chapter.
Medium-scale optimization. lsqlin wi th options.LargeScale set to 'off', or
when l i near i nequal i ti es or equal i ti es are gi ven, i s based on quadprog, whi ch
uses an acti ve set method si mi l ar to that descri bed i n [1]. I t fi nds an i ni ti al
feasi bl e sol uti on by fi rst sol vi ng a l i near programmi ng probl em. See the
quadrati c programmi ng method di scussed i n the I ntroduction toAlgorithms
chapter.
Diagnostics Large-scale optimization. The l arge-scal e code does not al l ow equal upper and
l ower bounds. For exampl e i f lb(2)==ub(2), then lsqlin gi ves the error:
Equal upper and lower bounds not permitted in this large-scale
method.
Use equality constraints and the medium-scale method instead.
At thi s ti me, the medi um-scal e al gori thm must be used to sol ve equal i ty
constr ai ned probl ems.
Medium-scale optimization. I f the matri ces C, A or Aeq are sparse, and the probl em
formul ati on i s not sol vabl e usi ng the l arge-scal e code, lsqlin warns that the
matri ces wi l l be converted to ful l :
Warning: This problem formulation not yet available for sparse
matrices.
Converting to full to solve.
lsqlin
4-113
lsqlin gi ves a warni ng when the sol uti on i s i nfeasi bl e:
Warning: The constraints are overly stringent;
there is no feasible solution.
I n thi s case, lsqlin produces a resul t that mi ni mi zes the worst case constrai nt
vi ol ati on.
When the equal i ty constrai nts are i nconsi stent, lsqlin gi ves
Warning: The equality constraints are overly stringent;
there is no feasible solution.
Limitations At thi s ti me, the onl y l evel s of di spl ay, usi ng the Display parameter i n
options, are 'off' and 'final'; i terati ve output usi ng 'iter' i s not
avai l abl e.
See Also lsqnonneg, quadprog, \
References [1] Gi l l , P.E., W. Murray, and M.H. Wri ght, Practical Optimization, Academi c
Press, London, UK, 1981.
[2] Col eman, T.F. and Y. Li , A Refl ecti ve Newton Method for Mi ni mi zi ng a
Quadrati c Functi on Subject to Bounds on Some of the Vari abl es, SI AM
J ournal on Optimization, Vol . 6, Number 4, pp. 1040-1058, 1996.
lsqnonlin
4-114
4l sqnonl i n
Purpose Sol ve nonl i near l east-squares (nonl i near data-fi tti ng) probl ems
where L i s a constant.
Syntax x = lsqnonlin(fun,x0)
x = lsqnonlin(fun,x0,lb,ub)
x = lsqnonlin(fun,x0,lb,ub,options)
x = lsqnonlin(fun,x0,options,P1,P2, ... )
[x,resnorm] = lsqnonlin(...)
[x,resnorm,residual] = lsqnonlin(...)
[x,resnorm,residual,exitflag] = lsqnonlin(...)
[x,resnorm,residual,exitflag,output] = lsqnonlin(...)
[x,resnorm,residual,exitflag,output,lambda] = lsqnonlin(...)
[x,resnorm,residual,exitflag,output,lambda,jacobian] =
lsqnonlin(...)
Description lsqnonlin sol ves nonl i near l east-squares probl ems, i ncl udi ng nonl i near
data-fi tti ng probl ems.
Rather than compute the val ue f(x) (the sum of squares), lsqnonlin requi res
the user-defi ned functi on to compute the vector-val ued functi on
Then, i n vector terms, thi s opti mi zati on probl em may be restated as
where x i s a vector and F(x) i s a functi on that returns a vector val ue.
x = lsqnonlin(fun,x0) starts at the poi nt x0 and fi nds a mi ni mum to the
sum of squares of the functi ons descri bed i n fun. fun shoul d return a vector of
f x ( )
x
min f
1
x ( )
2
f
2
x ( )
2
f
3
x ( )
2
f
m
x ( )
2
L + + + + + =
F x ( )
f
1
x ( )
f
2
x ( )
f
3
x ( )
=
1
2
--- F x ( )
2
2
x
min
1
2
--- f
i
x ( )
2
i

=
lsqnonlin
4-115
val ues and not the sum-of-squares of the val ues. (fun(x) i s summed and
squared i mpl i ci tl y i n the al gori thm.)
x = lsqnonlin(fun,x0,lb,ub) defi nes a set of l ower and upper bounds on the
desi gn vari abl es, x, so that the sol uti on i s al ways i n the range lb <= x <= ub.
x = lsqnonlin(fun,x0,lb,ub,options) mi ni mi zes wi th the opti mi zati on
parameters speci fi ed i n the structure options.
x = lsqnonlin(fun,x0,lb,ub,options,P1,P2,...) passes the
probl em-dependent parameters P1, P2, etc., di rectl y to the functi on fun. Pass
an empty matri x for options to use the defaul t val ues for options.
[x,resnorm] = lsqnonlin(...) returns the val ue of the squared 2-norm of
the resi dual at x: sum(fun(x).^2).
[x,resnorm,residual] = lsqnonlin(...) returns the val ue of the resi dual ,
fun(x), at the sol uti on x.
[x,resnorm,residual,exitflag] = lsqnonlin(...) returns a val ue
exitflag that descri bes the exi t condi ti on.
[x,resnorm,residual,exitflag,output] = lsqnonlin(...) returns a
structure output that contai ns i nformati on about the opti mi zati on.
[x,resnorm,residual,exitflag,output,lambda] = lsqnonlin(...)
returns a structure lambda whose fi el ds contai n the Lagrange mul ti pl i ers at
the sol uti on x.
[x,resnorm,residual,exitflag,output,lambda,jacobian] =
lsqnonlin(...) returns the Jacobi an of fun at the sol uti on x.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
lsqnonlin
4-116
rel evant to lsqnonlin are i ncl uded bel ow for fun, options, exitflag, lambda,
and output.
fun The functi on to be mi ni mi zed. fun takes a vector x and returns a
vector F of the objecti ve functi ons eval uated at x. You can speci fy
fun to be an i nl i ne object. For exampl e,
x = lsqnonlin(inline('sin(x.*x)'),x0)
Al ternati vel y, fun can be a stri ng contai ni ng the name of a functi on
(an M-fi l e, a bui l t-i n functi on, or a MEX-fi l e). I f fun='myfun' then
the M-fi l e functi on myfun.m woul d have the form
function F = myfun(x)
F = ... % Compute function values at x
I f the Jacobi an can al so be computed and options.Jacobian i s
'on', set by
options = optimset('Jacobian','on')
then the functi on fun must return, i n a second output argument,
the Jacobi an val ue J, a matri x, at x. Note that by checki ng the
val ue of nargout the functi on can avoi d computi ng J when fun i s
cal l ed wi th onl y one output argument (i n the case where the
opti mi zati on al gori thm onl y needs the val ue of F but not J):
function [F,J] = myfun(x)
F = ... % objective function values at x
if nargout > 1 % two output arguments
J = ... % Jacobian of the function evaluated at x
end
I f fun returns a vector (matri x) of m components and x has l ength
n, then the Jacobi an J i s an m-by-n matri x where J(i,j) i s the
parti al deri vati ve of F(i) wi th respect to x(j). (Note that the
Jacobi an J i s the transpose of the gradi ent of F.)
lsqnonlin
4-117
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameter s usi ng the optimset functi on. Some
parameters appl y to al l al gori thms, some are onl y rel evant when
usi ng the l arge-scal e al gori thm, and others are onl y rel evant when
usi ng the medi um-scal e al gori thm.
We start by descri bi ng the LargeScale opti on si nce i t states a
preferencefor whi ch al gori thm to use. I t i s onl y a preference
because certai n condi ti ons must be met to use the l arge-scal e or
medi um-scal e al gori thm. For the l arge-scal e al gori thm, the
nonl i near system of equati ons cannot be under-determi ned; that i s,
the number of equati ons (the number of el ements of F returned by
fun) must be at l east as many as the l ength of x. Furthermore, onl y
the l arge-scal e al gori thm handl es bound constrai nts.
LargeScale Use l arge-scal e al gori thm i f possi bl e when set to
'on'. Use medi um-scal e al gori thm when set to 'off'.
Parameters used by both the l arge-scal e and medi um-scal e
al gori thms:
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
Jacobian Jacobi an for the objecti ve functi on defi ned by user .
See the descri pti on of fun above to see how to defi ne the Jacobi an
i n fun.
MaxFunEvals Maxi mum number of functi on eval uati ons
al l owed.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
lsqnonlin
4-118
Parameters used by the l arge-scal e al gori thm onl y:
JacobPattern Sparsi ty pattern of the Jacobi an for
fi ni te-di fferenci ng. I f i t i s not conveni ent to compute the
Jacobi an matri x J i n fun, lsqnonlin can approxi mate J vi a
sparse fi ni te-di fferences provi ded the structure of J, i .e.,
l ocati ons of the nonzeros, i s suppl i ed as the val ue for
JacobPattern. I n the worst case, i f the structure i s unknown,
you can set JacobPattern to be a dense matri x and a ful l
fi ni te-di fference approxi mati on wi l l be computed i n each
i terati on (thi s i s the defaul t i f JacobPattern i s not set). Thi s can
be very expensi ve for l arge probl ems so i t i s usual l y worth the
effort to determi ne the sparsi ty structure.
MaxPCGIter Maxi mum number of PCG (precondi ti oned
conjugate gradi ent) i terati ons (see the Algorithmsecti on bel ow).
PrecondBandWidth Upper bandwi dth of precondi ti oner for
PCG. By defaul t, di agonal precondi ti oni ng i s used (upper
bandwi dth of 0). For some probl ems, i ncreasi ng the bandwi dth
reduces the number of PCG i terati ons.
TolPCG Termi nati on tol erance on the PCG i terati on.
TypicalX Typi cal x val ues.
Parameters used by the medi um-scal e al gori thm onl y:
DerivativeCheck Compare user-suppl i ed deri vati ves
(Jacobi an) to fi ni te-di fferenci ng deri vati ves.
DiffMaxChange Maxi mum change i n vari abl es for
fi ni te-di fferenci ng.
DiffMinChange Mi ni mum change i n vari abl es for
fi ni te-di fferenci ng.
LevenbergMarquardt Choose Levenberg-Marquardt over
Gauss-Newton al gori thm.
LineSearchType Li ne search al gori thm choi ce.
lsqnonlin
4-119
Note: The sum of squares shoul d not be formed expl i ci tl y. I nstead, your
functi on shoul d return a vector of functi on val ues. See the exampl e bel ow.
Examples Fi nd x that mi ni mi zes
starti ng at the poi nt x = [0.3, 0.4].
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb.
lambda.upper for the upper bounds ub.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.funcCount The number of functi on eval uati ons.
output.algorithm The al gori thm used.
output.cgiterations The number of PCG i terati ons
(l arge-scal e al gori thm onl y).
output.stepsize The fi nal step si ze taken (medi um-scal e
al gor i thm onl y).
output.firstorderopt A measure of fi rst-order opti mal i ty
(l arge-scal e al gori thm onl y).
2 2k e
kx
1
e
kx
2
+ ( )
2
k 1 =
10

lsqnonlin
4-120
Because lsqnonlin assumes that the sum-of-squares i s not expl i ci tl y formed i n
the user functi on, the functi on passed to lsqnonlin shoul d i nstead compute the
vector val ued functi on
for (that i s, F shoul d have k components).
Fi rst, wri te an M-fi l e to compute the k-component vector F:
function F = myfun(x)
k = 1:10;
F = 2 + 2*kexp(k*x(1))exp(k*x(2));
Next, i nvoke an opti mi zati on routi ne:
x0 = [0.3 0.4] % Starting guess
[x,resnorm] = lsqnonlin('myfun',x0) % Invoke optimizer
After about 24 functi on eval uati ons, thi s exampl e gi ves the sol uti on:
x =
0.2578 0.2578
resnorm %residual or sum of squares
resnorm =
124.3622
Algorithm Large-scale optimization. By defaul t lsqnonlin wi l l choose the l arge-scal e
al gori thm. Thi s al gori thm i s a subspace trust regi on method and i s based on
the i nteri or-refl ecti ve Newton method descri bed i n [5], [6]. Each i terati on
i nvol ves the approxi mate sol uti on of a l arge l i near system usi ng the method of
precondi ti oned conjugate gradi ents (PCG). See the trust-regi on and
precondi ti oned conjugate gradi ent method descri pti ons i n the Large-Scale
Algorithms chapter.
Medium-scale optimization. lsqnonlin wi th options.LargeScale set to 'off'
uses the Levenberg-Marquardt method wi th l i ne-search [1], [2], [3].
Al ternati vel y, a Gauss-Newton method [4] wi th l i ne-search may be sel ected.
The choi ce of al gori thm i s made by setti ng options.LevenbergMarquardt.
Setti ng options.LevenbergMarquardt to 'off' (and options.LargeScale to
'off') sel ects the Gauss-Newton method, whi ch i s general l y faster when the
resi dual i s smal l .
F
k
x ( ) 2 2k e
kx
1
e
kx
2
+ =
k 1 to 10 =
F x ( )
2
2
lsqnonlin
4-121
The defaul t l i ne search al gori thm, i .e., options.LineSearchType set to
'quadcubic', i s a safeguarded mi xed quadrati c and cubi c pol ynomi al
i nterpol ati on and extrapol ati on method. A safeguarded cubi c pol ynomi al
method can be sel ected by setti ng options.LineSearchType to 'cubicpoly'.
Thi s method general l y requi res fewer functi on eval uati ons but more gradi ent
eval uati ons. Thus, i f gradi ents are bei ng suppl i ed and can be cal cul ated
i nexpensi vel y, the cubi c pol ynomi al l i ne search method i s preferabl e. The
al gori thms used are descri bed ful l y i n the I ntroduction toAlgorithms chapter.
Diagnostics Large-scale optimization. The l arge-scal e code wi l l not al l ow equal upper and
l ower bounds. For exampl e i f lb(2)==ub(2) then lsqlin gi ves the error:
Equal upper and lower bounds not permitted.
(lsqnonlin does not handl e equal i ty constrai nts, whi ch i s another way to
formul ate equal bounds. I f equal i ty constrai nts are present, use fmincon,
fminimax or fgoalattain for al ternati ve formul ati ons where equal i ty
constrai nts can be i ncl uded.)
Limitations The functi on to be mi ni mi zed must be conti nuous. lsqnonlin may onl y gi ve
l ocal sol uti ons.
lsqnonlin onl y handl es real vari abl es. When x has compl ex vari abl es, the
vari abl es must be spl i t i nto real and i magi nary parts.
Large-scale optimization. The l arge-scal e method for lsqnonlin does not sol ve
under-determi ned systems: i t requi res that the number of equati ons (i .e., the
number of el ements of F) be at l east as great as the number of vari abl es. I n the
under-determi ned case, the medi um-scal e al gori thm wi l l be used i nstead. (I f
bound constrai nts exi st, a warni ng wi l l be i ssued and the probl em wi l l be
sol ved wi th the bounds i gnored.) See Tabl e 1-4 for more i nformati on on what
probl em formul ati ons are covered and what i nformati on must be provi ded.
The precondi ti oner computati on used i n the precondi ti oned conjugate gradi ent
part of the l arge-scal e method forms J
T
J (where J i s the Jacobi an matri x)
before computi ng the precondi ti oner; therefore, a row of J wi th many nonzeros,
whi ch resul ts i n a nearl y dense product J
T
J , may l ead to a costl y sol uti on
process for l arge probl ems.
I f components of x have no upper (or l ower) bounds, then lsqnonlin prefers
that the correspondi ng components of ub (or lb) be set to inf (or inf for l ower
lsqnonlin
4-122
bounds) as opposed to an arbi trary but very l arge posi ti ve (or negati ve for
l ower bounds) number.
Currentl y, i f the anal yti cal Jacobi an i s provi ded i n fun, the options parameter
DerivativeCheck cannot be used wi th the l arge-scal e method to compare the
anal yti c Jacobi an to the fi ni te-di fference Jacobi an. I nstead, use the
medi um-scal e method to check the deri vati ves wi th options parameter
MaxIter set to 0 i terati ons. Then run the probl em wi th the l arge-scal e method.
Medium-scale optimization. The medi um-scal e al gori thm does not handl e bound
constrai nts.
Si nce the l arge-scal e al gori thm does not handl e under-determi ned systems and
the medi um-scal e does not handl e bound constrai nts, probl ems wi th both these
characteri sti cs cannot be sol ved by lsqnonlin.
See Also optimset, lsqcurvefit, lsqlin
References [1] Levenberg, K.,A Method for the Sol uti on of Certai n Probl ems i n Least
Squares, QuarterlyApplied Math. 2, pp. 164-168, 1944.
[2] Marquardt, D.,An Al gori thm for Leastsquares Esti mati on of Nonl i near
Parameters, SI AM J . Applied Math. Vol . 11, pp. 431-441, 1963.
[3] Mor, J.J., The LevenbergMarquardt Al gori thm: I mpl ementati on and
Theory, Numerical Analysis, ed. G. A. Watson, LectureNotes in Mathematics
630, Spri nger Verl ag, pp. 105-116, 1977.
[4] Denni s, J.E., Jr., Nonl i near Least Squares, Stateof theArt in Numerical
Analysis, ed. D. Jacobs, Academi c Press, pp. 269-312, 1977.
[5] Col eman, T.F. and Y. Li , On the Convergence of Refl ecti ve Newton
Methods for Large-Scal e Nonl i near Mi ni mi zati on Subject to Bounds,
Mathemati cal Programmi ng, Vol . 67, Number 2, pp. 189-224, 1994.
[6] Col eman, T.F. and Y. Li , An I nteri or, Trust Regi on Approach for Nonl i near
Mi ni mi zati on Subject to Bounds, SI AM J ournal on Optimization, Vol . 6, pp.
418-445, 1996.
lsqnonneg
4-123
4l sqnonneg
Purpose Sol ves the nonnegati ve l east squares probl em
where the matri x C and the vector d ar e the coeffi ci ents of the objecti ve
functi on. The vector, x, of i ndependent vari abl es i s restri cted to be
nonnegati ve.
Syntax x = lsqnonneg(C,d)
x = lsqnonneg(C,d,x0)
x = lsqnonneg(C,d,x0,options)
[x,resnorm] = lsqnonneg(...)
[x,resnorm,residual] = lsqnonneg(...)
[x,resnorm,residual,exitflag] = lsqnonneg(...)
[x,resnorm,residual,exitflag,output] = lsqnonneg(...)
[x,resnorm,residual,exitflag,output,lambda] = lsqnonneg(...)
Description x = lsqnonneg(C,d) returns the vector x that mi ni mi zes norm(C*xd) subject
to x >= 0. C and d must be real .
x = lsqnonneg(C,d,x0) uses x0 as the starti ng poi nt i f al l x0 >= 0; otherwi se,
the defaul t i s used. The defaul t start poi nt i s the ori gi n (the defaul t i s al so used
when x0==[] or when onl y two i nput arguments are provi ded).
x = lsqnonneg(C,d,x0,options) mi ni mi zes wi th the opti mi zati on
parameters speci fi ed i n the structure options.
[x,resnorm] = lsqnonneg(...) returns the val ue of the squared 2-norm of
the resi dual : norm(C*xd)^2.
[x,resnorm,residual] = lsqnonneg(...) returns the resi dual , C*xd.
[x,resnorm,residual,exitflag] = lsqnonneg(...) returns a val ue
exitflag that descri bes the exi t condi ti on of lsqnonneg.
[x,resnorm,residual,exitflag,output] = lsqnonneg(...) returns a
structure output that contai ns i nformati on about the opti mi zati on.
1
2
--- Cx d
2
x
min
2
such that x 0
lsqnonneg
4-124
[x,resnorm,residual,exitflag,output,lambda] = lsqnonneg(...)
returns the Lagrange mul ti pl i ers i n the vector lambda.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to lsqnonneg are i ncl uded bel ow for options, exitflag, lambda, and
output.
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameters usi ng the optimset functi on.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
TolX Termi nati on tol erance on x.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates the i terati on count was exceeded. I ncreasi ng the
tol erance TolX may l ead to a sol uti on.
lambda Vector contai ni ng the Lagrange mul ti pl i ers: lambda(i)<=0 when
x(i) i s (approxi matel y) 0, and lambda(i) i s (approxi matel y) 0
when x(i)>0.
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.algorithm The al gori thm used.
lsqnonneg
4-125
Examples Compare the unconstrai ned l east squares sol uti on to the lsqnonneg sol uti on
for a 4-by-2 probl em.
C = [
0.0372 0.2869
0.6861 0.7071
0.6233 0.6245
0.6344 0.6170];
d = [
0.8587
0.1781
0.0747
0.8405];
[C\d, lsqnonneg(C,d)] =
2.5627 0
3.1108 0.6929
[norm(C*(C\d)d), norm(C*lsqnonneg(C,d)d)] =
0.6674 0.9118
The sol uti on from lsqnonneg does not fi t as wel l as the l east squares sol uti on.
However, the nonnegati ve l east-squares sol uti on has no negati ve components.
Algorithm lsqnonneg uses the al gori thm descri bed i n [1]. The al gori thm starts wi th a set
of possi bl e basi s vectors and computes the associ ated dual vector lambda. I t
then sel ects the basi s vector correspondi ng to the maxi mum val ue i n lambda i n
order to swap i t out of the basi s i n exchange for another possi bl e candi date.
Thi s conti nues unti l lambda <= 0.
Notes The nonnegati ve l east squares probl em i s a subset of the constrai ned l i near
l east-squares probl em. Thus, when C has more rows than col umns (i .e., the
system i s over -deter mi ned)
[x,resnorm,residual,exitflag,output,lambda] = lsqnonneg(C,d)
lsqnonneg
4-126
i s equi val ent to
[m,n] = size(C);
[x,resnorm,residual,exitflag,output,lambda_lsqlin] =
lsqlin(C,d,eye(n,n),zeros(n,1));
except that lambda = lambda_lsqlin.ineqlin.
For probl ems greater than order twenty, lsqlin may be faster than lsqnonneg,
otherwi se lsqnonneg i s general l y more effi ci ent.
See Also optimset, lsqlin, \
References [1] Lawson, C.L. and R.J. Hanson, SolvingLeast Squares Problems,
Prenti ce-Hal l , Chapter 23, p. 161, 1974.
optimget
4-127
4opti mget
Purpose Get opti mi zati on opti ons parameter val ues.
Syntax val = optimget(options,'param')
val = optimget(options,'param',default)
Description val = optimget(options,'param') returns the val ue of the speci fi ed
parameter i n the opti mi zati on opti ons structure options. You need to type onl y
enough l eadi ng characters to defi ne the parameter name uni quel y. Case i s
i gnored for parameter names.
val = optimget(options,'param',default) returns default i f the speci fi ed
parameter i s not defi ned i n the opti mi zati on opti ons structure options. Note
that thi s form of the functi on i s used pri mari l y by other opti mi zati on functi ons.
Examples Thi s statement returns the val ue of the Display opti mi zati on opti ons
parameter i n the structure cal l ed my_options:
val = optimget(my_options,'Display')
Thi s statement returns the val ue of the Display opti mi zati on opti ons
parameter i n the structure cal l ed my_options (as i n the previ ous exampl e)
except that i f the Display parameter i s not defi ned, i t returns the val ue
'final':
optnew = optimget(my_options,'Display','final');
See Also optimset
optimset
4-128
4opti mset
Purpose Create or edi t opti mi zati on opti ons parameter structure.
Syntax options = optimset('param1',value1,'param2',value2,...)
optimset
options = optimset
options = optimset(optimfun)
options = optimset(oldopts,'param1',value1,...)
options = optimset(oldopts,newopts)
Description options = optimset('param1',value1,'param2',value2,...) creates an
opti mi zati on opti ons parameter structure cal l ed options, i n whi ch the
speci fi ed parameters (param) have speci fi ed val ues. Any unspeci fi ed
parameters are set to [] (parameters wi th val ue [] i ndi cate to use the defaul t
val ue for that parameter when options i s passed to the opti mi zati on functi on).
I t i s suffi ci ent to type onl y enough l eadi ng characters to defi ne the parameter
name uni quel y. Case i s i gnored for parameter names.
optimset wi th no i nput or output arguments di spl ays a compl ete l i st of
parameters wi th thei r val i d val ues.
options = optimset (wi th no i nput arguments) creates an opti ons structure
options where al l fi el ds are set to [].
options = optimset(optimfun) creates an opti ons structure options wi th al l
parameter names and defaul t val ues rel evant to the opti mi zati on functi on
optimfun.
options = optimset(oldopts,'param1',value1,...) creates a copy of
oldopts, modi fyi ng the speci fi ed parameters wi th the speci fi ed val ues.
options = optimset(oldopts,newopts) combi nes an exi sti ng opti ons
structure oldopts wi th a new opti ons structure newopts. Any parameters i n
newopts wi th nonempty val ues overwri te the correspondi ng ol d parameters i n
oldopts.
Parameters For more i nformati on about i ndi vi dual parameters, see the reference pages for
the opti mi zati on functi ons that use these par ameter s, or Tabl e 4-3.
optimset
4-129
I n the l i sts bel ow, val ues i n {}denote the defaul t val ue; some parameters have
di fferent defaul ts for di fferent opti mi zati on functi ons and so no val ues are
shown i n {}.
Opti mi zati on parameters used by both l arge-scal e and medi um-scal e
al gori thms:
Diagnostics [ on | {off} ]
Display [ off | iter | {final} ]
GradObj [ on | {off} ]
Jacobian [ on | {off} ]
LargeScale [ {on} | off ]
MaxFunEvals [ positive integer ]
MaxIter [ positive integer ]
TolCon [ positive scalar ]
TolFun [ positive scalar ]
TolX [ positive scalar ]
Optimization parameters used by large-scale algorithms only:
Hessian [ on | {off} ]
HessPattern [ sparse matrix ]
JacobPattern [ sparse matrix ]
MaxPCGIter [ positive integer ]
PrecondBandWidth [ positive integer | Inf ]
TolPCG [ positive scalar | {0.1} ]
TypicalX [ vector ]
Opti mi zati on parameters used by medi um-scal e al gori thms onl y:
optimset
4-130
DerivativeCheck [ on | {off} ]
DiffMaxChange [ positive scalar | {1e1} ]
DiffMinChange [ positive scalar | {1e8} ]
GoalsExactAchieve [ positive scalar integer | {0} ]
GradConstr [ on | {off} ]
HessUpdate [ {bfgs} | dfp | gillmurray | steepdesc ]
LevenbergMarquardt [ on | off ]
LineSearchType [ cubicpoly | {quadcubic} ]
MeritFunction [ singleobj | {multiobj} ]
MinAbsMax [ positive scalar integer | {0} ]
Examples Thi s statement creates an opti mi zati on opti ons structure cal l ed options i n
whi ch the Display parameter i s set to 'iter' and the TolFun parameter i s set
to 1e8:
options = optimset('Display','iter','TolFun',1e8)
Thi s statement makes a copy of the opti ons structure cal l ed options, changi ng
the val ue of the TolX parameter and stori ng new val ues i n optnew:
optnew = optimset(options,'TolX',1e4);
Thi s statement returns an opti mi zati on opti ons structure options that
contai ns al l the parameter names and defaul t val ues rel evant to the functi on
fminbnd:
options = optimset('fminbnd')
I f you onl y want to see the defaul t val ues for fminbnd, you can si mpl y type
optimset fminbnd
or equi val entl y
optimset('fminbnd')
See Also optimget
quadprog
4-131
4quadpr og
Purpose Sol ve the quadrati c programmi ng probl em
where H, A, and Aeq are matri ces, and f, b, beq, lb, ub, and x are vectors.
Syntax x = quadprog(H,f,A,b)
x = quadprog(H,f,A,b,Aeq,beq)
x = quadprog(H,f,A,b,Aeq,beq,lb,ub)
x = quadprog(H,f,A,b,Aeq,beq,lb,ub,x0)
x = quadprog(H,f,A,b,Aeq,beq,lb,ub,x0,options)
[x,fval] = quadprog(...)
[x,fval,exitflag] = quadprog(...)
[x,fval,exitflag,output] = quadprog(...)
[x,fval,exitflag,output,lambda] = quadprog(...)
Description x = quadprog(H,f,A,b) returns a vector x that mi ni mi zes
1/2*x'*H*x + f'*x subject to A*x <= b.
x = quadprog(H,f,A,b,Aeq,beq) sol ves the probl em above whi l e addi ti onal l y
sati sfyi ng the equal i ty constrai nts Aeq*x = beq.
x = quadprog(H,f,A,b,lb,ub) defi nes a set of l ower and upper bounds on the
desi gn vari abl es, x, so that the sol uti on i s i n the range lb <= x <= ub.
x = quadprog(H,f,A,b,lb,ub,x0) sets the starti ng poi nt to x0.
x = quadprog(H,f,A,b,lb,ub,x0,options) mi ni mi zes wi th the opti mi zati on
parameters speci fi ed i n the structure options.
[x,fval] = quadprog(...) returns the val ue of the objecti ve functi on at x:
fval = 0.5*x'*H*x + f'*x.
[x,fval,exitflag] = quadprog(...) returns a val ue exitflag that
descri bes the exi t condi ti on of quadprog.
1
2
---x
T
Hx f
T
x +
x
min such that A x b
Aeq x beq =
l b x ub
quadprog
4-132
[x,fval,exitflag,output] = quadprog(...) returns a structure output
that contai ns i nformati on about the opti mi zati on.
[x,fval,exitflag,output,lambda] = quadprog(...) returns a structure
lambda whose fi el ds contai n the Lagrange mul ti pl i ers at the sol uti on x.
Arguments The arguments passed i nto the functi on are descri bed i n Tabl e 4-1. The
arguments returned by the functi on are descri bed i n Tabl e 4-2. Detai l s
rel evant to quadprog are i ncl uded bel ow for options, exitflag, lambda, and
output.
options Opti mi zati on parameter opti ons. You can set or change the val ues
of these parameter s usi ng the optimset functi on. Some parameters
appl y to al l al gori thms, some are onl y rel evant when usi ng the
l arge-scal e al gori thm, and others are onl y rel evant when usi ng the
medi um-scal e al gori thm.
We start by descri bi ng the LargeScale opti on si nce i t states a
preferencefor whi ch al gori thm to use. I t i s onl y a preference si nce
certai n condi ti ons must be met to use the l arge-scal e al gori thm.
For quadprog, when the probl em has onlyupper and l ower bounds,
i .e., no l i near i nequal i ti es or equal i ti es are speci fi ed, the defaul t
al gori thm i s the l arge-scal e method. Or, i f the probl em gi ven to
quadprog has only l i near equal i ti es, i .e., no upper and l ower
bounds or l i near i nequal i ti es are speci fi ed, and the number of
equal i ti es i s no greater than the l ength of x, the defaul t al gori thm
i s the l arge-scal e method. Otherwi se the medi um-scal e al gori thm
wi l l be used.
The par ameter to set an al gor i thm prefer ence:
LargeScale Use l arge-scal e al gori thm i f possi bl e when set to
'on'. Use medi um-scal e al gori thm when set to 'off'.
quadprog
4-133
Parameters used by both the l arge-scal e and medi um-scal e
al gori thms:
Diagnostics Pri nt di agnosti c i nformati on about the functi on
to be mi ni mi zed.
Display Level of di spl ay. 'off' di spl ays no output; 'iter'
di spl ays output at each i terati on; 'final' di spl ays just the fi nal
output.
MaxIter Maxi mum number of i terati ons al l owed.
TolFun Termi nati on tol erance on the functi on val ue.
TolX Termi nati on tol erance on x.
Parameters used by the l arge-scal e al gori thm onl y:
MaxPCGIter Maxi mum number of PCG (precondi ti oned
conjugate gradi ent) i terati ons (see the Algorithmsecti on bel ow).
PrecondBandWidth Upper bandwi dth of precondi ti oner for
PCG. By defaul t, di agonal precondi ti oni ng i s used (upper
bandwi dth of 0). For some probl ems, i ncreasi ng the bandwi dth
reduces the number of PCG i terati ons.
TolPCG Termi nati on tol erance on the PCG i terati on.
TypicalX Typi cal x val ues.
exitflag Descri bes the exi t condi ti on:
> 0 i ndi cates that the functi on converged to a sol uti on x.
0 i ndi cates that the maxi mum number of functi on eval uati ons or
i terati ons was reached.
< 0 i ndi cates that the functi on di d not converge to a sol uti on.
lambda A structure contai ni ng the Lagrange mul ti pl i ers at the sol uti on x
(separated by constrai nt type):
lambda.lower for the l ower bounds lb.
lambda.upper for the upper bounds ub.
lambda.ineqlin for the l i near i nequal i ti es.
lambda.eqlin for the l i near equal i ti es.
quadprog
4-134
Examples Fi nd val ues of x that mi ni mi ze
subject to
Fi rst note that thi s functi on may be wri tten i n matri x notati on as
where
Enter these coeffi ci ent matri ces:
H = [1 1; 1 2]
f = [2; 6]
A = [1 1; 1 2; 2 1]
b = [2; 2; 3]
lb = zeros(2,1)
output A structure whose fi el ds contai n i nformati on about the
opti mi zati on:
output.iterations The number of i terati ons taken.
output.algorithm The al gori thm used.
output.cgiterations The number of PCG i terati ons
(l arge-scal e al gori thm onl y).
output.firstorderopt A measure of fi rst-order opti mal i ty
(l arge-scal e al gori thm onl y).
f x ( )
1
2
---x
1
2
x
2
2
x
1
x
2
2x
1
6x
2
+ =
x
1
x
2
+ 2
x
1
2x
2
+ 2
2x
1
x
2
+ 3
0 x
1
0 x
2
,
f x ( )
1
2
---x
T
Hx f
T
x + =
H
1 1
1 2
, = f
2
6
, = x
x
1
x
2
=
quadprog
4-135
Next, i nvoke a quadrati c programmi ng routi ne:
[x,fval,exitflag,output,lambda] = quadprog(H,f,A,b,[],[],lb)
Thi s generates the sol uti on
x =
0.6667
1.3333
fval =
8.2222
exitflag =
1
output =
iterations: 3
algorithm: 'medium-scale: active-set'
firstorderopt: []
cgiterations: []
lambda.ineqlin
ans =
3.1111
0.4444
0
lambda.lower
ans =
0
0
Nonzero el ements of the vectors i n the fi el ds of lambda i ndi cate acti ve
constrai nts at the sol uti on. I n thi s case, the fi rst and second i nequal i ty
constrai nts (i n lambda.ineqlin) are acti ve constrai nts (i .e., the sol uti on i s on
thei r constrai nt boundari es). For thi s probl em, al l the l ower bounds are
i nacti ve.
Notes I n general quadprog l ocates a l ocal sol uti on unl ess the probl em i s stri ctl y
convex.
Better numeri cal resul ts are l i kel y i f you speci fy equal i ti es expl i ci tl y usi ng Aeq
and beq, i nstead of i mpl i ci tl y usi ng lb and ub.
I f components of xhave no upper (or l ower) bounds, then quadprog prefers that
the correspondi ng components of ub (or lb) be set to Inf (or Inf for lb) as
quadprog
4-136
opposed to an arbi trary but very l arge posi ti ve (or negati ve i n the case of l ower
bounds) number.
Large-scale optimization. I f you do not suppl y x0, or x0 i s not stri ctl y feasi bl e,
quadprog chooses a new stri ctl y feasi bl e (center ed) star ti ng poi nt.
I f an equal i ty constrai ned probl em i s posed and quadprog detects negati ve
curvature, then the opti mi zati on termi nates because the constrai nts are not
restri cti ve enough. I n thi s case, exitflag i s returned wi th the val ue 1, a
message i s di spl ayed (unl ess the options Display parameter i s 'off'), and the
x returned i s not a sol uti on but a di recti on of negati ve curvature wi th respect
to H.
Algorithm Large-scale optimization. When the probl em gi ven to quadprog has only upper
and l ower bounds, i .e., no l i near i nequal i ti es or equal i ti es are speci fi ed, the
defaul t al gori thm i s the l arge-scal e method. Or, i f the probl em gi ven to
quadprog has only l i near equal i ti es, i .e., no upper and l ower bounds or l i near
i nequal i ti es are speci fi ed the defaul t al gori thm i s the l arge-scal e method.
Thi s method i s a subspace trust-regi on method based on the i nteri or-refl ecti ve
Newton method descri bed i n [2]. Each i terati on i nvol ves the approxi mate
sol uti on of a l arge l i near system usi ng the method of precondi ti oned conjugate
gradi ents (PCG). See the trust-regi on and precondi ti oned conjugate gradi ent
method descri pti ons i n the Large-ScaleAlgorithms chapter.
Medium-scale optimization. quadprog uses an acti ve set method, whi ch i s al so a
projecti on method, si mi l ar to that descri bed i n [1]. I t fi nds an i ni ti al feasi bl e
sol uti on by fi rst sol vi ng a l i near programmi ng probl em. Thi s method i s
di scussed i n the I ntroduction toAlgorithms chapter.
Diagnostics Large-scale optimization. The l arge-scal e code wi l l not al l ow equal upper and
l ower bounds. For exampl e i f lb(2)==ub(2) then quadprog gi ves the error:
Equal upper and lower bounds not permitted in this large-scale
method.
Use equality constraints and the medium-scale method instead.
I f you onl y have equal i ty constrai nts you can sti l l use the l arge-scal e method.
But i f you have both equal i ti es and bounds, you must use the medi um-scal e
method.
quadprog
4-137
Medium-scale optimization. When the sol uti on i s i nfeasi bl e, quadprog gi ves thi s
warni ng:
Warning: The constraints are overly stringent;
there is no feasible solution.
I n thi s case, quadprog produces a resul t that mi ni mi zes the worst case
constrai nt vi ol ati on.
When the equal i ty constrai nts are i nconsi stent, quadprog gi ves thi s warni ng:
Warning: The equality constraints are overly stringent;
there is no feasible solution.
Unbounded sol uti ons,whi ch can occur when the Hessi an H i s negati ve
semi defi ni te, may resul t i n
Warning: The solution is unbounded and at infinity;
the constraints are not restrictive enough.
I n thi s case, quadprog returns a val ue of x that sati sfi es the constrai nts.
Limitations At thi s ti me the onl y l evel s of di spl ay, usi ng the Display parameter i n options,
are 'off' and 'final'; i terati ve output usi ng 'iter' i s not avai l abl e.
The sol uti on to i ndefi ni te or negati ve defi ni te pr obl ems i s often unbounded (i n
thi s case, exitflag i s returned wi th a negati ve val ue to show a mi ni mum was
not found); when a fi ni te sol uti on does exi st, quadprog may onl y gi ve l ocal
mi ni ma si nce the probl em may be nonconvex.
Large-scale optimization. The l i near equal i ti es cannot be dependent (i .e., Aeq
must have ful l row rank). Note that thi s means that Aeq cannot have more rows
than col umns. I f ei ther of these cases occur, the medi um-scal e al gori thm wi l l
be cal l ed i nstead. See Tabl e 1-4 for more i nformati on on what probl em
formul ati ons are covered and what i nformati on must be provi ded.
References [1] P.E. Gi l l , W. Murray, and M.H. Wri ght, Practical Optimization, Academi c
Press, London, UK, 1981.
[2] Col eman, T.F. and Y. Li , A Refl ecti ve Newton Method for Mi ni mi zi ng a
Quadrati c Functi on Subject to Bounds on some of the Vari abl es, SI AM
J ournal on Optimization, Vol . 6, Number 4, pp. 1040-1058, 1996.
quadprog
4-138
I-1
Index
A
acti ve constrai nts 4-94, 4-111, 4-135
acti ve set method 2-28, 4-40, 4-94, 4-112, 4-136
arguments, addi ti onal 1-15
attai nment factor 4-22
axi s crossi ng. Seezero of a functi on
B
banana functi on 2-4
BFGS formul a 2-6, 4-40, 4-63
bi secti on search 4-90
bound constrai nts, l arge-scal e 3-8
box constrai nts. Seebound constrai nts
C
center i ng parameter 3-15
CG. Seeconjugate gradi ents
compl ementari ty condi ti ons 3-15
compl ex val ues 1-68
compl ex vari abl es 4-105, 4-121
conjugate gradi ents 3-4
constrai ned mi ni mi zati on 4-30
l arge-scal e exampl e 1-43, 1-47
medi um-scal e exampl e 1-8
constrai nts, posi ti ve 1-15
conti nuous functi on and gradi ent methods 2-4
convex probl em 2-23
cubi c i nterpol ati on 2-9
curve-fi tti ng 4-98
D
data-fi tti ng 4-98
data-fi tti ng categori es 1-3
dense col umns, constrai nt matri x 3-16
dependent 4-39, 4-50
dependent constrai nts 3-8
DFP formul a 4-63
di recti on of negati ve curvature 3-4
di sconti nui ti es 1-66, 2-4
di sconti nuous probl ems 4-54, 4-63
di screte var i abl es 1-67
dual probl em 3-14
dual i ty gap 3-15
E
-constr ai nt method 2-37
equal i ty constrai nts
dense col umns 1-53
medi um-scal e exampl e 1-14
equal i ty constrai nts i nconsi stent warni ng, quad-
prog 4-137
equal i ty constrai nts, l i near
l arge-scal e 3-8
equati on sol vi ng categori es 1-3
error, out of memory 1-37
F
feasi bi l i ty condi ti ons 3-15
feasi bl e poi nt, fi ndi ng 2-30
fgoalattain
exampl e 1-26
fi xed vari abl es 3-17
fi xed-step ODE sol ver 1-22
fmincon
l arge-scal e exampl e 1-43, 1-47
medi um-scal e exampl e 1-8
fminimax
exampl e 1-23
Index
I-2
fminunc
l arge-scal e exampl e 1-39
medi um-scal e exampl e 1-6
warni ng messages 1-65
fsolve
l arge-scal e exampl e 1-33
medi um-scal e exampl e 1-15
functi on arguments 4-6
functi on di sconti nui ti es 1-66
G
Gauss-Newton 3-11
Gauss-Newton method 2-18, 2-19, 2-22, 4-84,
4-105, 4-120
gl obal mi ni mum 1-65
gl obal vari abl es 1-15
goal attai nment 2-39, 4-14
exampl e 1-26
goaldemo 4-22
gol den secti on search 4-28
gradi ent checki ng, anal yti c 1-13
gradi ent exampl es 1-11
gradi ent methods 2-4
H
Hessian modified headi ng 2-28
Hessian modified twice headi ng 2-28
Hessi an sparsi ty structur e 1-41
Hessi an update 2-11, 2-27
I
i nconsi stent constrai nts 4-97
i ndefi ni te probl ems 4-137
i nfeasi bl e probl ems 4-41
i nfeasi bl e sol uti on warni ng
linprog 4-97
quadprog 4-137
i nfi ni te l oop 1-67
i nl i ne objects 1-63
i nput arguments 4-6
i nstal l ati on vi i i
i nteger vari abl es 1-67
i nteri or-poi nt l i near programmi ng 3-14
i terati ve di spl ay 1-60
J
Jacobi an exampl es 1-33
Jacobi an sparsi ty pattern 1-35
K
Kuhn-Tucker equati ons 2-23
L
Lagrange mul ti pl i ers
l arge-scal e l i near programmi ng 3-17
l arge-scal e exampl es 1-29
l arge-scal e functi onal i ty coverage 1-29
l east squares 2-19
l east squares categori es 1-3
Levenberg-Marquardt method 2-19, 2-20, 2-22,
4-84, 4-105, 4-120
l i ne search 4-63, 4-84, 4-85, 4-105, 4-121
l i ne search strategy 1-5
l i near equati ons sol ve 4-83
l i near l east squares
constr ai ned 4-107
l arge-scal e al gori thm 3-13
l arge-scal e exampl e 1-50
Index
I-3
l i near l east squares (conti nued)
nonnegati ve 4-123
unconstrai ned 4-111
l i near progr ammi ng 4-91
i mpl ementati on 2-31
l arge-scal e al gori thm 3-14
l arge-scal e exampl e 1-52, 1-53
probl em 2-2
linprog
l arge-scal e exampl e 1-52, 1-53
LI PSOL 3-14
l ower bounds 1-9
lsqlin
l arge-scal e exampl e 1-50
lsqnonlin
convergence 1-68
l arge-scal e exampl e 1-35
medi um-scal e exampl e 1-19
M
maxi mi zati on 1-15
Mehrotras pr edi ctor-corrector al gori thm 3-14,
3-15
meri t functi on 2-31
mi ni max exampl es 1-23
mi ni max probl em 4-43
mi ni mi zati on categori es 1-2
mul ti objecti ve opti mi zati on 2-33, 4-14
exampl es 1-16
N
NCD Bl ockset 1-22
negati ve curvature di recti on 3-4, 3-6
negati ve defi ni te probl ems 4-137
Nel der and Mead 2-4
Newton di r ecti on
approxi mate 3-4
Newtons method 2-4
no deri vati ve method 4-54
no update message 2-28
nonconvex probl ems 4-137
noni nferi or sol uti on 2-34
nonl i near data-fi tti ng 4-114
nonl i near equati ons, sol vi ng 4-77
nonl i near l east squares 2-21, 4-98, 4-114
l arge-scal e al gor i thm 3-11
l arge-scal e exampl e 1-35
nonl i near progr ammi ng 2-2
nonl i near system of equati ons, l arge-scal e exam-
pl e 1-33
normal equati ons 3-11, 3-13
notati on vi i i
O
objecti ve functi on, undefi ned val ues 1-66
opti mal i ty condi ti ons l i near programmi ng 3-14
opti mi zati on parameters structure 1-56, 4-127,
4-128
opti ons par ameters 4-6
out of memory err or 1-37
output arguments 4-6
output di spl ay 1-58
output headi ngs 1-60
l arge-scal e al gor i thms 1-60
medi um-scal e al gori thms 1-58
P
PCG. Seeprecondi ti oned conjugate gradi ents
precondi ti oned conjugate gradi ents 3-4, 3-6, 3-16
al gori thm 3-6
Index
I-4
precondi ti oner 1-34, 3-6
banded 1-43
predi ctor -corrector al gori thm 3-15
preprocessi ng
l i near programmi ng 3-14, 3-17
pri mal probl em 3-14
pri mal -dual al gori thm 3-15
pri mal -dual i nteri or -poi nt 3-14
projecti on method 2-28, 4-136
Q
quadprog
l arge-scal e exampl e 1-49
quadrati c i nterpol ati on 2-9
quadrati c programmi ng 2-2, 4-40, 4-131
l arge-scal e al gori thm 3-12
l arge-scal e exampl e 1-49
quasi -Newton method 2-5, 2-11, 4-63
R
refl ecti ve l i ne search 3-12
refl ecti ve steps 3-9, 3-10
resi dual 2-18
Rosenbrocks functi on 2-4
S
sampl i ng i nterval 4-70
secul ar equati on 3-3
semi -i nfi ni te constrai nts 4-65
Sherman-Morri son formul a 3-16
si gnal processi ng exampl e 1-25
si mpl e bounds 1-9
si mpl ex search 2-4, 4-54
Si mul i nk, mul ti objecti ve exampl e 1-16
si ngl eton rows 3-17
sparsi ty patter n Jacobi an 1-35
sparsi ty structure, Hessi an 1-41
SQP method 2-24, 2-28, 4-40
steepest descent 4-63
stoppi ng cri teri a, l arge-scal e l i near programmi ng
3-16
str uctural rank 3-17
subspace
determi nati on of 3-4
subspace, two-di mensi onal 3-4
system of equati ons, sol vi ng 4-77
T
trust regi on 3-3
two-di mensi onal subspace 3-4
typographi cal conventi ons xi i i
U
unbounded sol uti ons warni ng
linprog 4-97
quadprog 4-137
unconstrai ned mi ni mi zati on 4-52, 4-56
l arge-scal e exampl e 1-39
medi um-scal e exampl e 1-6
one di mensi onal 4-26
unconstrai ned opti mi zati on 2-4
upper bounds 1-9
V
vari abl e-step ODE sol ver 1-22
Versi on 2, changes to cal l i ng sequence 1-70
Versi on 2, converti ng Ver si on 1.5 code 1-69
Index
I-5
W
warni ng
equal i ty constr ai nts i nconsi stent, quadprog
4-137
i nfeasi bl e sol uti on, linprog 4-97
i nfeasi bl e sol uti on, quadprog 4-137
stuck at mi ni mum, fsolve 4-85
unbounded sol uti ons, linprog 4-97
unbounded sol uti ons, quadprog 4-137
warni ngs di spl ayed 1-66
wei ghted sum strategy 2-35
Z
zero curvature di r ecti on 3-6
zero fi ndi ng 4-77
zero of a functi on, fi ndi ng 4-87