Algorithms for Solving Nonlinear

Systems of Equations(*)

Jose Mario Martinez

Department of Applied Mathematics
IMECC-UNICAMP, CP 6065
University of Campinas
13081 Campinas SP
Brazil

ABSTRACT: In this paper we survey numerical methods for solving nonlinear

systems of equations F( x) = 0, where F : IRn _ IRn. We are especially interested in
large problems. We describe modern implementations of the main local algorithms,
as well as their globally convergent counterparts.

1. INTRODUCTION

Nonlinear systems of equations appear in many real - life problems. More [1989] has
reported a collection of practical examples which include: Aircraft Stability problems,
Inverse Elastic Rod problems, Equations of Radiative Transfer, Elliptic Boundary
Value problems, etc .. We have also worked with Power Flow problems, Distribution
of Water on a Pipeline, Discretization of Evolution problems using Implicit Schemes,
Chemical Plant Equilibrium problems, and others. The scope of applications becomes
even greater if we include the family of Nonlinear Programming problems, since the
first-order optimality conditions of these problems are nonlinear systems.
Given F : IRn _ IRn, F = (iI, ... , fnf, our aim is to find solutions of

F(x) = O. (1.1)
We assume that F is well defined and has continuous partial derivatives on an open
set of IRn. We denote J (x) the matrix of partial derivatives of F (Jacobian matrix).
So,
0(0) Work supported by FAPESP (PT NQ 90-3724-6) FAEP-UNICAMP, FINEP and CNPq.
81
E. Spedicato (ed.), Algorithmsjor Continuous Optimization, 81-108.
© 1994 Kluwer Academic Publishers.
82

described by

J(Xk)Sk = _F(xk), (2.2)

Xk+l = Xk + Sk. (2.3)

At each iteration of Newton's method, we must compute the Jacobian J(xk) and
solve the linear system (2.2). Using modern techniques of automatic differentiation
(see RaIl [1984, 1987], Griewank [1992]' and references therein) we can compute F(x)
and J(x) in a reliable and economical way. If, instead of the true Jacobian in (2.2),
we use an approximation by differences of J(xk), which is generally expensive, we
obtain the Finite-Difference Newton's Method, whose convergence properties are very
similar to those of Newton's method.
Now, (2.2) is a linear system of equations. If n is small, this system can be
solved using the LU factorization with partial pivoting or the QR factorization. See
Golub and Van Loan [1989]. Using these linear solvers, the cost of solving (2.2) is
O(n 3 ) floating point operations. If n is large this cost becomes prohibitive. However,
in many situations, where the matrix J(xk) is sparse, we can solve (2.2) using LU
factorizations. In fact, many times the structure of the matrix is such that the
factors Land U of its factorization are also sparse, and can be computed using a
moderate amount of operations. Computer algorithms for sparse LU factorizations
are surveyed in Duff, Erisman and Reid [1989]. In Gomes-Ruggiero, Martinez and
Moretti [1992] we describe the first version of the NIGHTINGALE package for solving
sparse nonlinear systems. In NIGHTINGALE, we use the sparse linear solver of
George and Ng [1987]. The George-Ng method performs the LU factorization with
partial pivoting of a sparse matrix A using a static data structure defined before
beginning numerical computations. In Newton's method we solve a sequence of linear
systems with the same structure, so, the symbolic phase that defines the data structure
is executed only once.
The system (2.2) has a unique solution if and only if J(xk) is nonsingular. If the
Jacobian is singular, the iteration must be modified. Moreover, if J(Xk) is nearly
singular, it is also convenient to modify the iteration in order to prevent numerical
instability. Many modifications are possible to keep this phenomenon controlled. In
the NIGHTINGALE package, when a very small pivot, relative to the size of the
matrix, occurs, it is replaced by a nonzero scalar whose modulus is sufficiently large.
Moreover, nearly singular or ill-conditioned matrices usually cause very large incre-
ments sk. So, Ilskll must also be controlled. Computer algorithms usually normalize
the stepsize by

~}
S
k .
+-mm
{
l'M k
s.

where ~ is a parameter given by the user.

The main convergence result relative to Newton's method is given by the following
 83

J(x) == F'(x) == [ f~~X) ] == [ V'f\(x f ] == [~:(X) ... ~(X)].

f'(x)
n
V'fn(x)T !!h.(x)
ax!
E..i.n.(x)
8xn

We are mostly interested in problems where n is large and J(x) is structurally

sparse. This means that most entries of J(x) are zero for all x in the domain of
F. Sparsity is a particular case of the more general notion of structure. Jacobian
matrices can be symmetric, antisymmetric, positive definite, combination of other
matrices with some particular structure, etc .. Many times we can take advantage of
particular structures of J(x) in order to obtain efficient algorithms for solving (1.1).
Most popular methods for solving nonlinear systems are local. A local method is
an iterative scheme that converges if the initial approximation is close enough to a
particular solution. Frequently, we are also able to prove rate of convergence results
for these methods, which tell something about the asymptotic velocity of convergence
of the process. Fortunately, in many practical cases the domain of convergence of
local methods is large, so that these methods are useful. However, when the initial
estimate of the solution is very poor, local methods must be modified in order to
improve their global convergence properties.
We say that a method for solving (1.1) is globally convergent if at least one limit
point of the sequence generated by the method is a solution or, at least, a stationary
point, where V' IIF( x) W= o. Many times all the limit points are solutions or station-
ary points and, frequently, the whole sequence converges to a solution. In general,
global methods are modifications of local methods that preserve the local convergence
properties of the original algorithm.
This paper is organized as follows. In Section 2, 3, 4 and 5 we survey local meth-
ods, namely: Newton's method, Quasi-Newton methods, Inexact-Newton methods
and Decomposition methods. In Sections 6 and 7 we survey the main procedures
currently used for globalization: Optimization and Homotopies. Conclusions are
given in Section 8.

2. NEWTON'S METHOD

Newton's method is the most widely used algorithm for solving nonlinear systems
of equations. Given an initial estimation XO of the solution of (1.1), this method
considers, at each iteration, the approximation

(2.1)
and computes Xk+l as a solution of the linear system Lk(X) = o. This solution exists
and is unique if J(xk) is nonsingular. Therefore, an iteration of Newton's method is
 84

theorem.

Theorem 2.1. Let us assume that F : n c lR" -+ JR:', n an open and convex set,
F E Cl(n), F(x*) = 0, J(x*) nonsingular, and that there exist L,p > 0 such that
for all x E n

Lllx -
IIJ(x) - J(x*)11 :::; x*W'· (2.4)
Then there exists c > 0 such that if Ilxo - x*11 :::; c,the sequence {xk} generated by
(2.2) - (2.3) is well defined, converges to x*, and satisfies

Proof. See Ortega and Rheinboldt [1970], Dennis and Schnabel [1983], etc. 0

Property (2.5) (quadratic convergence if p = 1) depends on the Holder condition

(2.4). If (2.4) is not assumed, only superlinear convergence of {xk} can be proved
. Il xk+l - x*11
( hm )
= 0 . Clearly, (2.5) implies superlinear convergence but the con-
k-co Ilxk - x*11
verse is not true.

3. QUASI-NEWTON METHODS

In this survey, we call Quasi-Newton methods those methods for solving (1.1) whose
general form is

(3.1)
Newton's method, studied in Section 2, belongs to this family. Most Quasi-
Newton methods use less expensive iterations than Newton, but their convergence
properties are not very different. In general, Quasi-Newton methods avoid either the
necessity of computing derivatives, or the necessity of solving a full linear system per
iteration or both tasks.
The most simple Quasi-Newton method is the Stationary Newton Method, where
Bk = J(XO) for all k E IN. In this method, derivatives are computed at the initial
point and we only need the LU factorization of J(XO). A variation of this method
is the Stationary Newton Method with restarts, where Bk = J(xk) if k is a multiple
of a fixed integer m and Bk = B k- 1 otherwise. The number of iterations used by
this method tends to increase with m, but the average computer time per iteration
decreases. In some situations we can determine an optimal choice for m (Shamanskii
[1967]).
An obvious drawback of the stationary Newton methods is that, except when
k == 0 (mod m), Bk does not incorporate information about xk and F(x k). Therefore,
 85

the adequacy of the model Lk(X) == F(x k) + Bk(X - Xk) to the real function F(x) can
decrease rapidly as k grows. Observe that, due to (3.1), in Quasi-Newton methods
Xk+l is defined as the solution of L k( x) = 0, which exists and is unique if Bk is
nonsingular. One way to incorporate new information about F on the linear model
is to impose the interpolatory conditions

Lk+l(X k ) = F(xk), (3.2)

Lk+l (xk+l) = F( xk+l). (3.3)

Defining

yk = F(xk+l) _ F(xk) (3.4)

and substracting (3.2) from (3.3) we obtain the Secant Equation

Bk+lSk = yk. (3.5)

Reciprocally, if Bk+l satisfies (3.5), Lk+l interpolates F at xk and xk+l. We give the
name Secant Methods to the family of Quasi-Newton methods based on (3.1) and
(3.5).
If n 2: 2, there exist infinite many possible choices of Bk+l satisfying (3.5). If, in
addition to (3.5), we impose that

(3.6)
we obtain the Sequential Secant Method (Wolfe [1959], Barnes [1965], Gragg and
Stewart [1976], Martinez [1979a], etc.). If the set of increments {sk, sk-1, ... , sk-n+1 }
is linearly independent there exists only one matrix Bk+l that satisfies (3.6). In this
case,

B k+1 -- (yk , yk-1 , ... , yk-n+l )(sk "sk-1 ... , Sk-n+1 )-1 (3.7)
and

-1
B k+1 = (k
s,s k-1 , ... ,s k-n+1)( y,y
k k-l , ... ,y k-n+1 )-1 . (3.8)
Bi:~l can be obtained from Bi: 1 using O(n 2 ) floating point operations. However, in
order to ensure numerical stability, the definition of the increments si that appear
in (3.7) and (3.8) must sometimes be modified. When these modifications are not
necessary, the Sequential Secant Method has the following interpolatory property:

Lk+1 (Xk+i) -- F( XHi) , J. -- 0, - 1, ... , -no (3.9)

The Sequential Secant Method is useful in many situations where n is small. When
n is not very small, it is not worthwhile to waste time trying to preserve the inter-
polatory condition (3.9) for j ~ -no It is more profitable to maintain the Secant
 86

Equation (3.5), using the degrees of freedom inherent to this equation to guarantee
numerical stability. Broyden's "good" method (Broyden [1965]) and the Column Up-
dating Method (COLUM) (Martinez [1984]) are two examples of this idea. In both
methods

(3.10)

where

(3.11)
for Broyden's method and

Zk = (3.12)
I(ejkfskl = (3.13)

for COLUM. {e\ ... , en} is the canonical basis of JRn.

Applying the Sherman-Morrison formula to (3.10) (Golub and Van Loan [1989,
p. 51]) we obtain

(Sk B-1yk)(zk)T
B- 1 - B- 1 + - k B- 1 (3 14)
k+l - k (zk)T B;;1yk k· .
Formula (3.14) shows that B;;:1 can be obtained from B;;1 using O(n 2 ) floating
point operations in the dense case. Moreover,

B;;;1 = (I + u k(zkf)B;;1, (3.15)

where uk = (sk - B;; 1yk)/(zkf B;;1yk, so

(3.16)
for k = 1,2, 3 ...
Formula (3.16) is used when n is large. In this case, the vectors uO, zO, ... ,U k- 1, zk-1
are stored and the product B;;1F(xk) is computed using (3.16). In this way, the com-
puter time of iteration k is O(kn) plus the computer time of computing B01F(x k).
If k is large the process must be periodically restarted taking Bk ::::: J(xk).
An intermediate method between Broyden's method and the Sequential Secant
Method is Broyden's Method with Projected Updates, which was introduced by Gay
and Schnabel [1978]. See also Martinez [1979b] and Lopes and Martinez [1980]. This
method is probably more efficient than Broyden for small problems, but we have not
heard about large-scale implementations.
Broyden's method is a particular case of the family of Least Change Secant Up-
date (LCSU) methods (Dennis and Schnabel [1979, 1983], Dennis and Walker [1981]'
 87

Martinez [1990b, 1992a]), which include many algorithms that are useful for partic-
ular structures (Hart-Soul algorithms for boundary value problems (Hart and Soul
[1973], Kelley and Sachs [1987]), Partitioned Quasi-Newton methods for separable
problems (Griewank and Toint [1982a, 1982b, 1982c, 1984], Toint [1986]), Meth-
ods of Direct Updating of Factorizations (Dennis and Marwil [1982]' Johnson and
Austria [1983], Chadee [1985], Martinez [1990a]), BFGS and DFP algorithms for
unconstrained minimization (see Dennis and Schnabel [1983]), etc.
Let us survey the main convergence results related to Quasi-Newton algorithms.
We are going to assume, as in Theorem 2.1, that F : n c lR:' --+ IR n , n open and
convex, F E C1(n), F(x*) = 0, J(x*) nonsingular and that the Holder condition
(2.4) is satisfied. The first result is the "Theorem of two neighborhoods".

Theorem 3.1. Given r E (0,1), there exists €, 8 > 0 such that if I/xo - x* II ::; € and
IIBk - J(x*)11 ::; 8 for all k E lN then the sequence {xk} generated by (3.1) is well
defined, converges to x*, and satisfies

(3.17)
for all k E IN.

Proof. See, for instance, Dennis and Walker [1981]. D.

Using Theorem 3.1 we can prove that the Stationary Newton Method and its
variations with restarts have local convergence at a linear rate. The main tool for
proving superlinear convergence of Quasi-Newton methods, is the following theorem,
due to Dennis and More.

Theorem 3.2. Assume that the sequence {Xk} generated by (3.1) is well defined
and convergent to x*. Then, the two following properties are equivalent.

(a) lim II[Bk - J(x*)](xk+ 1 - xk)11 = o. (3.18)

k. . . oo Ilxk+l - xkll
(b)
. Ilxk+l - x*11 =0.
hm (3.19)
k. . . oo Ilxk - x*11

Proof. See Dennis and More [1974]. o

(3.18) is called the Dennis-More condition. Using (3.18), we can prove that the
Stationary Newton Method with periodic restarts (for which limBk = J(x*)) is
k ..... oo
superlinearly convergent. The Dennis-More condition says that the effect of Bk -
J(x*) on the normalized increment tends to be null when k --+ 00. This condition
 88

is weaker than saying that B" tends to J(x*). The Dennis-More condition is closely
related to the Secant Equation. In fact, from (2.4) we can deduce that, for all x E n,

1!F(z) - F(x) - J(x*)(z - x)1I ~ Lllz - xii max{llx - x*IIP, liz - x*W} (3.20)

(Broyden, Dennis and More [1973]). So, writing x = x", z = x"+l, and assuming
that the sequence converges,

. Ily" - J(x*)s"ll (3.21)

1:.~ Ils"ll = o.
Therefore, if the Secant Equation (3.5) holds,

lim II[ B k+l J(x*)]s"ll =

- o. (3.22)
"-00 lis" II
This allows us to prove the following result.

Lemma 3.3. If the sequence generated by a Secant Method converges to x* and, in

addition,

(3.23)

then the convergence is superlinear.

Proof. Use (3.18) and (3.22). o

Theorem 3.1 does not guarantee local convergence of all Secant Methods. In fact,
the hypothesis of this theorem is that all the B,. '8 belong to a neighborhood of J(x*)
of radius 8. Observe that, even if the first Bo belongs to this neighborhood it could
be possible that IIB,,-J(x*)11 ~ IIBo-J(x*)1I , destroying convergence. Fortunately,
for LCSU methods (including Broyden) we are able to prove that exists 8' > 0 such
that liB,. - J(x*)11 ~ 8 for all k E IN, if IIBa - J(x*)11 ~ 8'. This is a Bounded De-
terioration Property. Moreover, the Successive Projection scheme that characterizes
LCSU methods guarantees also (3.23). Summing up, the following result holds for
Broyden, and other LCSU methods.

Theorem 3.4. There exists €, b > 0 such that, if II xo-x*II ~ € and IIBo-J(x*)II ~ b,
the sequence generated by Broyden's method is well defined, converges to x* and sat-
isfies (3.19).

Proof. See Broyden, Dennis and More [1973]. For an extension to "all" LCSU meth-
ods see Martinez [1990b, 1992a]. 0
 89

For Broyden's method, we also have the following result, which states that the
convergence is 2n-quadratic.

Theor~m 3.5. Under the hypotheses of Theorem 3.4, if p ~ 1, there exists c > 0
such that the sequence generated by Broyden's method satisfies

for all k E IN.

Proof. See Gay [1979]. o

Local superlinear convergence results also hold for suitable implementations of

the Sequential Secant Method and its variations (Gay-Schnabel and Lopes-Martinez).
Under slightly stronger assumptions, we can prove stronger convergence results for
these methods. For the Sequential Secant Method and the Lopes-Martinez Method
the R-order of convergence (Ortega and Rheinboldt [1970]) is the positive root of
t n +! - t n - 1 = o. The Gay-Schnabel method has R-order equal to the positive root
of t 2n - t 2n - 1 - 1 = 0 (Martinez [1979b]). Both numbers tend to 1 when n -+ 00.
After nearly 10 years of practical experience, we verified that COLUM has practi-
cally the same behavior as Broyden, in terms of reliability and number of iterations.
Many times, COLUM uses less computer time than Broyden because the iteration
is slightly less expensive. However, COLUM is not an LCSU method, and local and
superlinear convergence cannot be proved using the classical Bounded Deterioration
techniques. The following convergence results have been proved for COLUM.

Theorem 3.6. Assume that the sequence {xk} is generated by COLUM, except
that when k == 0 (mod m), Bk = J(xk). Then, there exists c > 0 such that, if
Ilxo - x*11 ~ c, the sequence converges superlinearly to x*.
Proof. See Martinez [1984]. For a similar result, concerning an Inverse Column Up-
dating Method, see Martinez and Zambaldi [1992]. 0

Theorem 3.7. Assume that n = 2. Let r E (0,1). Then, there exists c,5 > 0
such that, if Ilxo- x*11 ~ c and IIBo - J(x*)11 ~ 5, the sequence {Xk} generated by
COLUM is well defined, converges to x*, and satisfies (3.17).

Proof. See Martinez [1992c]. o

Theorem 3.8. Assume that the sequence {xk} generated by COL UM is well defined,
converges to x* and satisfies (3.17). Then

. Ilxk+2n - x*11
hm =0 (3.24)
k_oo Ilxk - x*11
 90

and

lim IIxk - x*W/k =

k-+oo
o. (3.25)

Proof. See Martinez [1992c). Property (3.25) is called R-superlinear convergence. 0

A large gap between practice and theory remains to be filled in relation to

COLUM. Some of the relevant questions are:
Does COLUM with restarts (as in the hypothesis of Theorem 3.6), satisfy

(3.26)
for some c > 0, q strictly greater than m? The motivation of this question is that
the Stationary Newton method with restarts every m iterations satisfies (3.26) with
q=m.
Does Theorem 3.7 hold for n > 2?
Is COLUM superlinearly convergent in the sense of (3.19)?

4. INEXACT-NEWTON METHODS

Many times, large nonlinear systems can be solved using Newton's method, employing
a sparse LU factorization for solving (2.2). Most frequently, even if Newton's method
is applicable, more efficient algorithms are obtained using COLUM or Broyden, with
Newton restarts. In the NIGHTINGALE package, an automatic restart procedure has
been incorporated, by means of which a Newton iteration is performed only when
it is expected that its efficiency should be greater than the efficiency of previous
Quasi-Newton iterations.
Sometimes, the structure of the Jacobian matrix is unsuitable for LU factoriza-
tions. That is, a lot of fill-in appears due to that structure and so the iteration
becomes very expensive. In many of these cases a strategy of "false Jacobians" works
well. By this we mean that we use a Quasi-Newton iteration with restarts, where, at
the restarted iterations, Bk is not J(xk) but a "simplified Jacobian" J(xk) such that
its L U factorization can be performed without problems.
Unhappily, in many cases, IIJ(xk) - J(xk)11 is excessively large, and the Quasi-
Newton method looses its local convergence properties. In these cases, it is strongly
recommendable to use Inexact-Newton methods.
The idea is the following. Since we cannot solve (2.2) using a direct (LU) method,
we use an Iterative Linear Method for solving (2.2). Usually, iterative linear methods
based on Krylov subspaces are preferred (Golub and Van Loan [1989], Hestenes and
Stiefel [1952]' Saad and Schulz [1986], etc). Iterative Linear Methods are interesting
for solving large-scale systems of equations because of their low memory requirements.
 91

When we solve (2.2) using an iterative linear method, we need a stopping criterion
for deciding when to finish the calculation. A very reasonable stopping criterion is

(4.1)
where fh E (0,1). The condition fh < 1 is necessary because, otherwise, the null
increment sk == 0 could be accepted as an approximate solution of (2.2). On the other
hand, if fh ~ 0, the number of iterations needed by the Iterative Linear Method to
obtain (4.1) could be excessively large. Therefore, in practice, an intermediate value
fh ~ 0.1 is recommended.
Dembo, Eisenstat and Steihaug introduced the criterion (4.1) and proved the main
local convergence properties of the algorithms based on this criterion.

Theorem 4.1 Assume that F(x*) = 0, J(x·) is nonsingular and continuous at x*,
and Ok ::::; Omax < 0 < 1. Then there exists t: > 0 such that, if Ilxo- x·11 : : ; t:, the
sequence {Xk} obtained using (4.1) and (2.3) converges to x* and satisfies

(4.2)
for all k ~ 0, where Ilyll* = IIJ(x*)yll. If lim Ok = 0 the convergence is superlinear.
k-+oo

Proof See Dembo, Eisenstat and Steihaug [1982]. o

Krylov subspace methods for solving systems like (2.2) are usually implemented
using some preconditioning scheme. See Axelsson [1985]. By this we mean that
the original system is replaced by an equivalent one, which is easier to solve by the
Iterative Linear Solver. In the case of (2.2), we wish to replace the linear system by

(4.3)
where Bkl (or, at least, the product Bkl z) must be easy to compute and Bk ~ J(xk).
For general linear systems, many useful preconditioners Bk have been introduced.
Most of them are based on Incomplete L U Factorizations, or on Stationary Linear
iterations. A very cheap and popular procedure is to use the diagonal of the origi-
nal matrix as preconditioner. Many other preconditioners for specific problems can
be found in the papers published in Spedicato [1991]. A common feature to dif-
ferent preconditioning schemes applied to a linear system Az = b is that the first
iteration of the preconditioned Iterative Linear Solver is zl = >"B-1b, where B is
the preconditioner. So, in the case of the system (2.2), the first increment tried
should be of the form ->..Bk 1 F(xk). This increment will be accepted if it satisfies
(4.1). However, (2.2) is not an isolated linear system of equations. In fact, probably
J(xk) ~ J(xk-l) specially when k --t 00. Therefore, we are motivated to use infor-
mation about Bk,F(xk),F(xk+1),xk+I,xk when we choose the preconditioner B k+1 •
This idea leads to impose a Secant Condition to the preconditioner. So, we would
92

like to introduce an algorithm based on (4.1), where the sequence of preconditioners

Bk are chosen in order to satisfy

(4.4)
for all k E IN.
We saw in Section 2 that there exist infinite many possible choices of Bk+1 satisfy-
ing (4.4). Nazareth and Nocedal [1978] and Nash [1985] suggested to use the classical
BFGS formula in order to precondition (2.2), when we deal with minimization prob-
lems. Our preference is to define

(4.5)
where Ck+1 is a classical preconditioner and Dk is chosen to satisfy (4.4).
The main appeal of Secant Preconditioners is that it has been shown (Martinez
[1992bJ) that using them it is possible to obtain stronger convergence results than

°
the one mentioned in Theorem 4.1. In fact, the main drawback of this result is the
necessity of (h - t for obtaining superlinear convergence. The following Precondi-
tioned Inexact Newton Method was introduced by Martinez [1992b] with the aim of
obtaining superlinear convergence without imposing a precision tending to infinity in
the iterative resolution of (2.2).

Algorithm 4.2. Let (h E (0,0) for all k E lN,O E (0,1) and lim Ok
k--+oo
= O. Assume
that x O E IRn is an initial approximation to the solution of (1.1) and Bo E mnxn is
an initial nonsingular preconditioner. Given xk E mn and Bk nonsingular, the steps
for obtaining xk+ 1 , Bk+1 are the following.

Step 1. Compute

(4.6)

Step 2. If

(4.7)
define

S
k
= sQ'
k
(4.8)
Else, find an increment sk such that (4.1) holds, using some iterative method.

Step 3. Define Xk+l = xk + sk.

 93

The following theorem states the main convergence result relative to Algorithm
4.2.

Theorem 4.3. Assume that F : n c lR" -+ lR", n an open and convex set,
F E C 1 (n), J(x*) nonsingular, F(x*) = a and (2.4) holds for some L 2: a,p 2: 1.
Suppose that IIBkll and IIB;lll are bounded and that the Dennis-More condition
(3.18) is satisfied. Then, there exists c > a such that, if Ilxo - x*11 :::; c, the sequence
{xk} generated by Algorithm 4.2 converges superlinearly to x*. Moreover, there ex-
ists ko E IN such that sk = s~ for all k 2: ko.

Proof. See Martinez [1992b]. 0

Theorem 4.3 states that, if we use preconditioners satisfying the Dennis-More
condition, superlinear convergence is obtained without lim fh = O. In fact, the
k ..... oo
first iteration s~ of the preconditioned iterative linear method will satisfy (4.6), and
so it will be accepted as the new increment sk, preserving superlinearity. Least-
Change Secant-Update formulae can be used for obtaining preconditioners satisfying
the hypotheses of Theorem 4.3.
Recently, Abaffy [1992] considered the possibility of using, for solving (2.2), it-
erative linear algorithms that work componentwise, that is, without evaluating the
whole residual at each step. For this type of algorithms, he introduced a new stopping
criterion, different from (4.1), which also ensures superlinear convergence.

5. DECOMPOSITION METHODS

The methods studied in the previous sections evaluate all the components of the
function F at the same points. This is not always the best possible strategy. In many
practical problems, given a guess of the solution, the evaluation of a few components
of F is enough to suggest a new useful estimate. Methods that evaluate different
components at different points are called Decomposition Methods.

I ),
The (Block) SOR-Newton method proceeds as follows. Assume that the compo-

nent, of F ",e divided into blocl" F" .. . , Fm and that we deoompo," x ~(

h
were Xi E IRn;, D ()
£i X E IRn ;, Z. = 1, ... , m. UT ox. t he matnx. 0 f partla. I
vve d enote oF;
derivatives of Fi with respect to the variables Xj.
J
The SOR-Newton method is defined
by

k+1 _
Xi
k
- Xi - W
(OFi)-l (
OXi
k+1 k+1 k k )F.( k+1
X l ' ..• 'X i_ l 'Xi' •.• 'X m i Xl
k+1 k k)
, ..• 'X i _ l ,Xi' ... 'X m

(5.1)
 94

for i = 1, ... , m, where wE (0,2) is a relaxation parameter. A generalization of (5.1)

IS

k+I
Xi = Xik - W
B-1 D ( k+I k+I k k )
i,k 1.'i Xl , ... ,Xi_I' Xi' ... 'X m . (5.2)
Methods of type (5.2) are called SOR-Quasi-Newton methods (Martinez [1992d,
1992eJ). If Bi,k+I satisfies

B i,k+l ( Xik+I - k)
Xi = ri
D ( k+I
Xl , ••. , Xi
k+I k ) D ( k+I
, ... , Xm - ri Xl
k k )
, •.• 'Xi" •. , Xm (5.3)

we say that (5.2) defines a SOR-Secant method (Martinez [1992eJ). The local conver-
gence analysis of (5.1), (5.2) and (5.3) has been made in Martinez [1992e]. Essentially,
the condition for the local linear convergence of (5.1) - (5.3) is the same convergence
condition of the linear SOR method for a linear system with matrix J(x*). If ni is
small for all i = 1, ... , n, SOR methods have low storage requirements and so, they
are useful for large-scale problems.

The asynchronous generalization of (5.2) can be very useful in parallel computer

environments. Assume that we work with m parallel processors, where processor i is
specialized in the computation of new guesses of the component Xi. That is, processor
i only computes Xi - WB;1 Fi(X) for given Bi and x. Different processors use different
times for their respective computations and, as soon as a processor produces a result,
this result is communicated to the other processors. In this way, all the processors
are working all the time and full advantage is taken from parallelism. See Bertsekas
and Tsitsiklis [1989]. Let us now formalize these ideas. The index k E IN represents
the times at which at least one component of X is changed. We define Ti C IN by
Ti = Set of times at which Xi is updated.
Then, the Asynchronous SOR method is defined by

X k+I = xk _ WB:-I D'(xv(i,l,k) xk xV(i,m,k») (5.4)

t J z,k I't 1 , ... , ,&"", m

if k + 1 E T i , and
(5.5)
if k + 1 ¢ T i , where 0::; v(i,j,k) ::; k for all k E IN,i,j = 1,oo.,m. (5.2) is a
particular case of (5.4) - (5.5) defining Ti = {i + jm,j E IN}, v(i,j, k) = k for all
i,j, ... , n, i #- j, k E IN. The Jacobi-Quasi-Newton method corresponds to Ti =
{1,2,3, ... },v(i,j,k) = k. Secant Asynchronous methods based on (5.4) can also be
considered.
The SOR and Jacobi methods for linear systems are strongly related with the pro-
jection methods of Kaczmarz [1937] and Cimmino [1938] respectively. The advantage
of Kaczmarz and Cimmino is that convergence is guaranteed for every linear system,
 95

while SOR and Jacobi require a condition on the spectral radius of the transforma-
tion. However, in general, when SOR and Jacobi converge, they are far more efficient
than Kaczmarz and Cimmino. Nonlinear generalizations of Kaczmarz and Cimmino
may be found in Tompkins [1955], McCormick [1977], Meyn [1983) , Martinez [1986a,
1986b, 1986c] and Diniz - Ehrhardt and Martinez [1992].
Different decomposition methods are motivated by direct methods for solving lin-
ear systems, for example, the family of Nonlinear ABS algorithms developed in Abaffy
and Spedicato [1989), Abaffy, Broyden and Spedicato [1984]' Abaffy, Galantai and
Spedicato [1987], Spedicato, Chen and Deng [1992), etc .. These methods generalize
classical algorithms due to Brown [1969 ], Brent [1973], Gay [1975) and Martinez
[1979c, 1980]). The idea is the following. Divide the components of F into m groups
FI, . .. ,Fm. Assume that xk has been computed. We generate Xk,l, . .. ,xk,m by:

(5.6)
Ji(Xk,i)(Xk,i+1 _ xk,i) = -Fi(xk,i), (5.7)
J j (Xk,j)(xk,i+1 - xk,i) = O,j = 1, ... , i-I, (5.8)
where JAx) = Fj(x), i = 0,1, ... ,m - 1.

(5.9)
Clearly, the scheme (5.6) - (5.9) solves a linear system of equations in one cycle.
However, ther are infinite many ways to choose the intermediate points xk,l, ... ,xk,m-l.
Different choices of these points originate different methods of the ABS class. The
first motivation given in the papers of Brown and Brent for methods of type (5.6)
- (5.9) was that, using suitable factorizations, the derivatives can be approximated
by differences in a more economic way than in the Finite Difference Newton method.
Methods of this class have, in general, the same local convergence properties of New-
ton, though the proofs are technically complicated.
Many other decomposition algorithms have been introduced with the aim of taking
advantage of particular structures. For systems that are reducible to block lower tri-
angular form see Eriksson [1976] and Dennis, Martinez and Zhang [1992]. For Block
Tridiagonal Systems, see Hoyer, Schmidt and Shabani [1989]. Much theory about
decomposition methods has been produced by the German school (Schmidt [1987),
Burmeister and Schmidt [1988], Hoyer [1987], Hoyer and Schmidt [1984], Schmidt,
Hoyer and Haufe [1985], etc.)

6. GLOBALIZATION BY OPTIMIZATION

In the previous sections we studied local methods, that is, algorithms that converge,
usually with a high rate of convergence, if the initial point is close enough to the
solution. Luckily, in many cases the domain of convergence of local algorithms is
large enough to guarantee practical efficiency. However, locally convergent methods
96

may not converge if the starting point is very poor, or if the system is highly nonlinear.
By this reason, local methods are usually modified in order to improve their global
convergence properties. The most usual way to do this is to transform (1.1) into an
Optimization Problem, with the objective function J(x) = ~IIF(x)W. Then, (1.1)
becomes the problem of finding a global minimizer of J.
However, the decision of merely using a method to minimize J, in order to solve
(1.1), is not satisfactory. In fact, sometimes efficient local methods converge rapidly
to a solution but the generated sequence {xk} does not exhibit monotonic behav-
ior in J(x k ). In these cases, the pure local method is much more efficient than the
J-minimization method. Often, the minimization method converges to a local (non-
global) minimizer of J, while the local method converges to a solution of (1.1). By
these reasons, it is necessary to give a chance to the local method before calling
the minimization algorithm. Different solutions have been proposed to this problem
(Grippo, Lampariello e Lucidi [1986]). Here we describe a strategy that combines lo-
cal algorithms and minimization methods introduced in the NIGHTINGALE package.
We define "ordinary iterations" and "special iterations". By an ordinary iteration
we understand an iteration produced by any of the methods described in sections 2
to 4 of this paper. Decomposition methods can also be considered, with some mod-
ifications. A special iteration is an iteration produced by a minimization algorithm
applied to J. We define, for all k E IN,

ak = Argmin U(X O), ••• , J(x k )}. (6.1)

Ordinary and special iterations are combined by the following strategy.

Algorithm 6.1. Initialize k t- 0, FLAG t- 1. Let q 2:: 0 be an integer, , E (0,1).

Step 1. If FLAG = 1, obtain Xk+l using an ordinary iteration. Else, obtain xk+1
using a special iteration.

Step 2. If

J(ak+ 1) ~ ,J(ak- q ) (6.2)

set FLAG t - 1, k t - k + 1 and go to Step 1. Else, re-define Xk+l t- ak+l. Set
FLAG t - -1, k t - k + 1 and go to Step 1. 8

If the test (6.2) is satisfied infinite many times, then there exists a subsequence
of {xk} such that lim IIF(xk)11 = o. So, if the sequence is bounded, we will be able
k-+oo
to find a solution of (1.1) up to any prescribed accuracy. Conversely, if (6.2) does
not hold for all k 2:: ko, then all the iterations starting from the ko-th will be special,
and the convergence properties of the sequence will be those of the minimization
algorithm.
 97

In principle, we can use as minimization algorithm to define special iterations any

method for minimization described in the literature (Dennis and Schnabel [1983],
Fletcher [1987], etc.). For large-scale problems, the NIGHTINGALE package uses a
strategy based on Trust Regions derived from the Inexact Newton approach (Fried-
lander, Gomes-Ruggiero, Martinez and Santos [1993]). This strategy is described by
the following algorithm.

Algorithm 6.2. Assume that ~min > 0, a: E (0,1) are given independently of the
iteration k. Define "pk(X) = IIF(x k) + J(xk)(x - xk)W , ~ 2 ~min •

Step 1. Compute an approximate minimizer x of "pk(X) on the box Ilx - xkll oo ~ :s

:s
such that "pk(X) "pk(X~), x~ is the projection of xk - 2J(xk)T F(xk)/Mk on the box
and Mk 2 211J( Xk) IIt IIJ( Xk) 1100.

Step 2. If

(6.3)
define xk+1 = x. Else, choose ~new E [O.lllx- xkll,0.9~], replace ~ by ~new and go
to Step 1. 8

Most of the work of Algorithm 6.2 is concentrated on the approximate solution of

(6.4)

(6.4) is the problem of minimizing a convex quadratic with box constraints. For this
problem, algorithms based on combinations of Krylov Subspace methods with Gra-
dient Projection strategies are currently preferred. In NIGHTINGALE, the approxi-
:s
mate solution of (6.4) is defined as a point that satisfies "pk( x) "pk( x~) and where, in
addition, the norm of projected gradient of "pk(X) is less than O.lIIJ(xk)T F(xk)ll. We
also choose: ~min = O.OOlx (typicalllxll), initial choice of ~ == ~o = Ilxoll,~new =
0.511x - xkll, further choice is ~ = 4 x ~.
The convergence properties of Algorithm 6.2 were given in Friedlander, Gomes
- Ruggiero, Martinez and Santos [1993]. Every limit point x* of a sequence {xk}
generated by this algorithm satisfies J(x*f F(x*) = o. Therefore, x* is a solution
of (1.1) if J(x*) is nonsingular. Unhappily, if J(x*) is singular, it is possible that
F(x*) =f. O. This is the main weakness of algorithms based on optimization for the
globalization of (1.1).
An advantage of special iterations based on box trust regions is that they can be
easily adapted to situations where we have natural bounds for the solution of (1.1).
Other recent methods based on the inexact Newton approach with global convergent
properties were given by Deuflhard [1991] and Eisenstat and Walker [1993].
 98

7. GLOBALIZATION BY HOMOTOPIES

°
In Section 6 we saw that local methods for solving F( x) = can be "globalized"
through their transformation into minimization problems. In this section we study
another popular technique to solve (1.1) when the initial approximation is poor.
This technique is based on homotopies. A homotopy associated to this problem is a
function H(x, t) : lR" x JR -+ JR such that

H(x,l) = F(X)}
H(xO, 0) = ° (7.1)

If H satisfies (7.1) we can expect that r == {(x, t) JRn x JR I H(x, t) =

°: :; t :::; I} will be an arc connecting the initial approximation XO with a solution
E
0,
x*. Homotopic techniques consist in tracing r fromt = °
to t = 1 in a reliable
and efficient way. The first to propose the homotopic idea were Lahaye [1935] and
Davidenko [1955]. The homotopic principle is very popular. In fact, this is one of the
techniques mentioned in the article Numerical Analysis of Encyclopaedia Britannica.
Sometimes, the homotopy has an interest by itself. In other cases we are interested
only in the solution of H(x, 1) = 0. In this section we deal with the latter situation.
This has a practical consequence. If we are interested only in the solution of H(x, 1) =
0, it makes sense to interrupt the tracing of r when t is close to 1, trying to apply a
local method from the current point. In general, this will be more efficient than to
insist in carefully tracing r.
Unhappily, r could not be an arc of finite length. It is possible that, if we
trace r starting from (XO,O), we never arrive to (x*,l). However, some classical
results of Differential Geometry help us to identify situations in which tracing r
leads to the solution of (1.1). If H'(x, t) == (H~(x, t), H:(x, t)) has full rank for all
(x, t) E H- 1({O}), then H-l( {O}) is a discrete union of curves homomorphic to JR or
to the one-dimensional sphere Sl. See Milnor [1969]. In this case, each component
of H- 1 ( {O}) can be traced efficiently using numerical methods. The condition that
H' (x, t) has full rank results, in many practical cases, from the application of a
theorem of Chow, Mallet-Paret and Yorke [1979]. See also Watson [1979]. However,
this does not guarantee that the component of H-l( {O}) that passes through (XO,O)
should arrive to (x*, 1).
°
Now, if the unique solution of H(x,O) = is xO and H~(xO,O) is nonsingular, it
can be proved that the component of H-1({0}) that passes through (XO,O) is homo-
morphic to JR. In fact, even weaker hypotheses are sufficient (Ortega and Rheinboldt
[1970, Ch. 6]). These hypotheses are easy to verify and guarantee that, if we trace
r starting from (xO,O), it is not possible to return to the xo. However this does not
guarantee yet that (x*, 1) will be reached.
To be sure that this homotopy is useful we must verify, in addition to the previous
assumptions, that r is bounded. In this case, r will be necessarily a segment of arc
joining (xO,O) with (x*, 1). When we want to make sure that a given homopopy will
 99

solve a problem, the difficult part is precisely to prove boundedness of r.

Sometimes, "natural" homotopies are used. This mean, homotopies where the
parameter t that is suggested by the problem under consideration. However, there
exist useful "artificial" homotopies. Let us mention here two of them. The homotopy
of Reduction of Residual is

H(x,t) = F(x) + (t -l)F(xO).

The "regularizing homotopy", used in the well known package HOMPACK (Watson,
Billups and Morgan [1987]) is

H(x, t) = tF(x) + (1 - t)(x - XO).

After the choice of H, we must use a numerical method to trace r. First, r

must be parametrized. Frequently, we describe r as a function of the parameter t.
However, if, for some to, we have that H~(x, to) is singular, x cannot be expressed as
a function of t in a neighborhood of to, and, instead of increasing t, we must decrease
this parameter, in order to advance in r. By this reason, it is usual to trace r using
s, the arclength, as parameter.
If this is the case, the procedure generally recommended to trace r is of Predictor-
Corrector type. Given a set of points (X(Sl), t(Sl)),"" (x(sm), t(Sm)) computed
consecutively on r, and an increment 6. > 0, we calculate a polynomial inter-
polating those points and, using this polynomial, we compute a predictor point
(x(sm + 6.),t(sm + 6.)). This new point does not necessarily belong to r. So, using
this point as initial approximation, we compute a point on r using a local method
for the nonlinear system H(x, t) = O. If, in this system, we consider that t is also
a variable, we have n equations with n + 1 unknowns. Special local algorithms for
underdetermined nonlinear systems were developed by Walker and Watson [1989],
Martinez [1991], etc. This is the Corrector Phase. When we arrive to a point (x, t)
with t close to 1, we try the direct application of a local method for F(x) = O.
An interesting discussion on homotopy methods is given in a recent survey by
Forster [1993].

8. CONCLUSIONS

Many practical applications give rise to large scale nonlinear systems of equations.
From the local point of view, the most interesting techniques for this case are varia-
tions of the Inexact Newton method. To develop preconditioners for Krylov Subspace
solvers, taking into account the structure of these methods and problems, is a chal-
lenging problem. Since solving a linear system is a particular case of minimizing a
quadratic on a box, it turns out that to solve efficiently this last problem is crucial.
Moreover the problem appears again in the development of methods for globalization
using optimization, and in the Corrector Phase of Homotopy methods.
100

For very small problems Newton's method continues to be the best choice, and for
medium to large problems a combination Newton-Quasi-Newton seems to be better.
Many decomposition methods are interesting when they are induced by characteristics
of the problem structure , or when decomposition is suggested by the computer
architecture.
In this paper, we surveyed methods based on first-order approximations of the
nonlinear system. Methods based on approximations of higher order have also been
developed (Schnabel and Frank [1984]) and are useful in the presence of singulari-
ties of the Jacobians. Large - scale implementations of these methods seem to be hard.

References

Abaffy, J. [1992]: Superlinear convergence theorems for Newton-type methods for

nonlinear systems of equations, JOTA , 73, pp. 269 - 277.

Abaffy, J.; Broyden, C.G.; Spedicato, E. [1984]: A class of direct methods for linear
equations, Numerische Mathematik 45, pp. 361-376.

Abaffy, J.; Galantai, A. ; Spedicato, E. [1987]: The local convergence of ABS

method for nonlinear algebraic system, Numerische Mathematik 51, pp. 429 -
439.

Abaffy, J.; Spedicato, E. [1989]: ABS Projection Algorithms, Mathematical Tech-

niques for Linear and Nonlinear Equations, Ellis Horwood, Chichester.

Axelsson, O. [1985]: A survey of preconditioned iterative methods for linear systems

of equations, BIT 25, pp. 166 - 187.

Barnes, J.G.P. [1965]: An algorithm for solving nonlinear equations based on the
secant method, Computer Journal 8, pp. 66 - 72.

Bertsekas, D.P.; Tsitsiklis, C. [1989]: Parallel and Distributed Computation. Nu-

merical Methods, The Johns Hopkins University Press, New York.

Brent, R.P. [1973]: Some efficient algorithms for solving systems of nonlinear equa-
tions, SIAM Journal on Numerical Analysis 10, pp. 327 - 344.

Brown, K.M. [1969]: A quadratically convergent Newton - like method based upon
Gaussian elimination, SIAM Journal on Numerical Analysis 6, pp. 560 - 569.
 101

Broyden, C.G. [1965): A class of methods for solving nonlinear simultaneous equa-
tions, Mathematics of Computation 19, pp. 577-593.

Broyden, C.G.; Dennis Jr., J.E.; More, J.J. [1973): On the local and superlinear
convergence of quasi-Newton methods, Journal of the Institute of Mathematics
and its Applications 12, pp. 223-245.

Burmeister, W.; Schmidt, J.W. [1978): On the k-order of coupled sequences arising
in single - step type methods, Numerische Mathematik 33, pp. 653 - 66l.

Chadee, F.F. [1985): Sparse quasi-Newton methods and the continuation problem,
T.R. S.O.L. 85-8, Department of Operations Research, Stanford University.

Chow, S.N.; Mallet-Paret, J.; Yorke, J.A. [1978): Finding zeros of maps: Homotopy
methods that are constructive with probability one, Mathematics of Computa-
tion 32, pp. 887-899.

Cimmino, G. [1938): Calcolo approsimato per la soluzione dei sistemi di equazioni

lineari, La Ricerca Scientifica Ser II, Ano IV 1 pp. 326-333.

Coleman,T. F.; Garbow, B. S.; More,J. J. [1984): Software for estimating sparse
Jacobian matrices, ACM Trans. Math. Software 11, pp. 363-378.

Coleman,T. F.; More,J. J. [1983): Estimation of sparse Jacobian matrices and graph
coloring problems, SIAM Journal on Numerical Analysis 20, pp. 187-209.

Curtis,A. R.; Powell, M. J. D.; Reid, J. K. [1974}, On the estimation of sparse

Jacobian matrices, Journal of the Institute of Mathematics and its Applications
13, pp. 117-120.

Davidenko, D.F. [1953): On the approximate solution of nonlinear equations, Ukrain.

Mat. Z. 5, pp. 196 -206.

Dembo, R.S.; Eisenstat, S.C.; Steihaug, T. [1982): Inexact Newton methods, SIAM
Journal on Numerical Analysis 19, pp. 400-408.

Dennis Jr., J.E.; Martinez, J.M.; Zhang, X. [1992): Triangular Decomposition

Methods for Reducible Nonlinear Systems of Equations, Technical Report,
Department of Mathematical Sciences, Rice University.

Dennis Jr., J.E.; Marwil, E.S. [1982): Direct secant updates of matrix factorizations,
Mathematics of Computation 38, pp. 459-476.

Dennis Jr., J.E.; More, J.J. [1974): A characterization of superlinear convergence

and its application to quasi - Newton methods, Mathematics of Computation
28, pp. 549 -560.
102

Dennis Jr., J.E.; More, J.J. [1977]: Quasi-Newton methods, motivation and theory,
SIAM Review 19, pp. 46-89.

Dennis Jr.,J.E.; Schnabel,R.B. [1979]: Least change secant updates for quasi-
Newton methods, SIAM Review 21, pp. 443-459.

Dennis Jr.,J.E.; Schnabel,R.B. [1983]:Numerical Methods for Unconstrained Opti-

mization and Nonlinear Equations, Prentice-Hall, Englewood Cliffs.

Dennis Jr., J.E. ; Walker, H.F. [1981]: Convergence theorems for least-change
secant update methods, SIAM Journal on Numerical Analysis 18, pp. 949-987.

Deufihard, P. [1991]: Global inexact Newton methods for very large scale nonlinear
problems Impact of Computing in Science and Engineering 3, pp. 366-393.

Diniz-Ehrhardt, M.A.; Martinez, J.M. [1992]: A parallel projection method for

overdetermined nonlinear systems of equations, to appear in Numerical Algo-
rithms.

Duff, I.S. [1977]: MA28 - a set of Fortran subroutrines for sparse unsymmetric
linear equations. AERE R8730, HMSO, London.

Duff, I.S.; Erisman, A.M.; Reid, J.K. [1989]:Direct Methods for Sparse Matrices,
Oxford Scientific Publications.

Eisenstat, S.C.; Walker, H.F. [1993]: Globally convergent inexact Newton methods,
to appear in SIAM Journal on Optimization.

Eriksson, J. [1976]: A note on the decomposition of systems of sparse nonlinear

equations, BIT 16, pp. 462 - 465.

Fletcher, R. [1987]: Practical Methods of Optimization (2nd edition), John Wiley

and Sons, New York.

Forster, W. [1993]: Homotopy methods, to appear in Handbook of Global Optimiza-

tion, Kluwer.

Friedlander, A.; Gomes-Ruggiero, M.A.; Martinez, J.M.; Santos, S.A. [1993]: A

globally convergent method for solving nonlinear systems using a trust - region
strategy, Technical Report, Department of Applied Mathematics, University of
Campinas.

Gay, D.M. [1975]: Brown's method and some generalizations with applications to
minimization problems, Ph D Thesis, Computer Science Department, Cornell
University, Ithaca, New York.
 103

Gay, D.M. [1979]: Some convergence properties of Broyden's method, SIAM Jour-
nal on Numerical Analysis 16, pp. 623 - 630.

Gay, D.M.; Schnabel, R.B. [1978]: Solving systems of nonlinear equations by Broy-
den's method with projected updates, in Nonlinear Programming 3, edited by
O. Mangasarian, R. Meyer and S. Robinson, Academic Press, New York, pp.
245-281.

George, A.; Ng, E. [1987]: Symbolic factorization for sparse Gaussian elimination
with partial pivoting, SIAM Journal on Scientific and Statistical Computing 8,
pp. 877-898.

Golub, G.H.; Van Loan, Ch.F. [1989]: Matrix Computations, The Johns Hopkins
University Press, Baltimore and London.

Gomes-Ruggiero, M.A.; Martinez, J.M. [1992]: The Column-Updating Method

for solving nonlinear equations in Hilbert space, Mathematical Modelling and
Numerical Analysis 26, pp 309-330.

Gomes-Ruggiero, M.A.; Martinez, J.M.; Moretti, A.C. [1992]: Comparing algo-

rithms for solving sparse nonlinear systems of equations, SIAM Journal on
Scientific and Statistical Computing 13, pp. 459 - 483.

Gragg, W.B.; Stewart, G.W. [1976]: A stable variant of the secant method for
solving nonlinear equations, SIAM Journal on Numerical Analysis 13, pp. 127
- 140.

Griewank, A. [1986]: The solution of boundary value problems by Broyden based

secant methods, CTAC - 85. Proceedings of the Computational Techniques and
Applications Conference, J. Noye and R. May (eds.), North Holland, Amster-
dam.

Griewank, A. [1992]: Achieving Logarithmic Growth of Temporal and Spacial Com-

plexity in Reverse Automatic Differentiation, Optimization Methods and Soft-
ware 1, pp. 35 - 54.

Griewank, A.; Toint, Ph.L. [1982a]: On the unconstrained optimization of partially

separable functions, in Nonlinear Optimization 1981, edited by M.J.D. Powell,
Academic Press, New York.

Griewank, A.; Toint, Ph.L. [1982b]: Partitioned variable metric for large structured
optimization problems, Numerische Mathematik 39, pp. 119 - 137.

Griewank, A.; Toint, Ph.L. [1982c]: Local convergence analysis for partitioned
quasi-Newton updates, Numerische Mathematik 39, pp. 429-448.
104

Griewank, A.; Toint, Ph.L. [1984]: Numerical experiments with partially separable
optimization problems, in Numerical Analysis Proceedings Dundee 1983, edited
by D.F. Griffiths, Lecture Notes in Mathematics vol. 1066, Springer - Verlag,
Berlin, pp. 203-220.
Grippo, L.; Lampariello, F.; Lucidi, S. [1986]: A nonmonotone line search technique
for Newton's method, SIAM Journal on Numerical Analysis 23, pp. 707 - 716.

Hart, W.E.; Soul, S.O.W. [1973]: Quasi-Newton methods for discretized nonlinear
boundary value problems, J. Inst. Math. Applies. 11, pp. 351 - 359.

Hestenes, M.R.; Stiefel, E. [1952]: Methods of conjugate gradients for solving linear
systems, Journal of Research of the National Bureau of Standards B49, pp. 409
- 436.

Hoyer, W. [1987]: Quadratically convergent decomposition algorithms for nonlinear

systems with special structure, Inform. Tech. Univ. Dresden 07 , pp. 23-87.

Hoyer, W.; Schmidt, J.W. [1984]: Newton-type decomposition methods for equa-
tions arising in network analysis, Z. Angew-Math. Mech. 64, pp. 397 - 405.

Hoyer, W.; Schmidt, J.W.; Shabani, N. [1989]: Superlinearlyconvergent decomposi-

ton methods for block - tridiagonal nonlinear systems of equations, Numerical
Functional Analysis and Optimization 10 (9 & 10), pp. 961 - 975.

Iri, M. [1984]: Simultaneous computations of functions, partial derivatives and

estimates of rounding errors. Complexity and Practicality, Japan Journal of
Applied Mathematics 1, pp. 223 - 252.

Johnson, G.W.; Austria, N.H. [1983]: A quasi-Newton method employing direct

secant updates of matrix factorizations, SIAM Journal on Numerical Analysis
20, pp. 315-325.
Kaczmarz, S. [1937]: Angenaherte Auflosung von Systemen linearer Gleichungen,
Bull. Acad. Polon. Sci. Lett. ASS, pp. 355-357.

Kelley, C.T.; Sachs, E.W. [1987]: A quasi-Newton method for elliptic boundary
value problems, SIAM Journal on Numerical Analysis 24, pp. 516 - 53l.

Lahaye, E. [1934]: Une methode de resolution d'une categorie d'equations tran-

scendantes, Comptes Rendus Acad. Sci. Paris 198, pp. 1840-1842.

Lopes, T.L.; Martinez, J.M. [1980]: Combination of the Sequential Secant Method
and Broyden's method with projected updates, Computing 25, pp. 379-386.

Martinez, J.M. [1979a]: Three new algorithms based on the sequential secant
method, BIT 19, pp. 236-243.
 105

Martinez, J.M. [1979b]: On the order of convergence of Broyden - Gay - Schnabel's

method, Commentationes Mathematicae Universitatis Carolinae 19, pp. 107-
118.

Martinez, J.M. [1979c]: Generalization of the methods of Brent and Brown for
solving nonlinear simultaneous equations, SIAM Journal on Numerical Analysis
16, pp. 434 - 448.

Martinez, J.M. [1980]: Solving nonlinear simultaneous equations with a generaliza-

tion of Brent's method, BIT 20, pp. 501 - 510.

Martinez, J.M. [1983]: A quasi-Newton method with a new updating for the LDU
factorization of the approximate Jacobian, Matematica Aplicada e Computa-
cional2, pp. 131-142.

Martinez, J.M. [1984]: A quasi-Newton method with modification of one column

per iteration, Computing 33, pp. 353-362.

Martinez, J.M. [1986a]: The method of Successive Orthogonal Projections for solv-
ing nonlinear simultaneous equations, Calcolo 23, pp. 93 - 105.

Martinez, J.M. [1986b]: Solving systems of nonlinear simultaneous equations by

means of an accelerated Successive Orthogonal Projections Method, Computa-
tional and Applied Mathematics 165, pp. 169 - 179.

Martinez, J.M. [1986c]: Solution of nonlinear systems of equations by an optimal

projection method, Computing 37, pp. 59 - 70.

Martinez, J.M. [1987]: Quasi-Newton Methods with Factorization Scaling for Solv-
ing Sparse Nonlinear Systems of Equations, Computing 38, pp. 133-141.
Martinez, J.M. [1990a]: A family of quasi-Newton methods for nonlinear equations
with direct secant updates of matrix factorizations, SIAM Journal on Numerical
Analysis 27, pp. 1034-1049.

Martinez, J.M. [1990b]: Local convergence theory of inexact Newton methods based
on structured least change updates, Mathematics of Computation 55, pp. 143-
168.

Martinez, J.M. [1991]: Quasi-Newton Methods for Solving Under determined Non-
linear Simultaneous Equations, Journal of Computational and Applied Mathe-
matics 34, pp. 171-190.

Martinez, J.M. [1992a]: On the relation between two local convergence theories
of least change secant update methods, Mathematics of Computation 59, pp.
457-481.
106

Martinez, J.M. [1992bJ: A Theory of Secant Preconditioners, to appear in Mathe-

matics of Computation.

Martinez, J.M. [1992cJ: On the Convergence of the Column-Updating Methods,

Technical Report, Department of Applied Mathematics, University of Camp-
mas.

Martinez, J.M. [1992dJ: Fixed-Point Quasi-Newton Methods, SIAM Journal on

Numerical Analysis 29, pp. 1413-1434.

Martinez, J.M. [1992eJ: SOR - Secant Methods, to appear in SIAM Journal on

Numerical Analysis.

Martinez, J.M.; Zambaldi, M.C. [1992J: An inverse Column-Updating Method for

solving Large-Scale Nonlinear Systems of Equations, to appear in Optimization
Methods and Software.

Matthies, H.; Strang, G. [1979J: The solution of nonlinear finite element equations,
International Journal of Numerical Methods in Engineering 14, pp. 1613 - 1626.

Me Cormick, S.F. [1977J: The method of Kaczmarz and row-orthogonalization for

solving linear equations and least-squares problems in Hilbert space, Indiana
University Mathematical Journal 26, pp. 1137 - 1150.

Meyn, K-H [1983]: Solution of underdetermined nonlinear equations by stationary

iteration, Numerische Mathematik 42, pp. 161-172.

Milnor, J.W. [1969]: Topology from the differential viewpoint, The University Press
of Virginia, Charlottesville, Virginia.

More, J.J. [1989]: A collection of nonlinear model problems, Preprint MCS - P60
- 0289, Mathematics and Computer Science Division, Argonne National Labo-
ratory, Argonne, Illinois.

Nazareth, 1.; Nocedal, J. [1978]: A study of conjugate gradient methods, Report

SOL 78-29, Department of Operations Research, Stanford University.

Nash, S.G. [1985J: Preconditioning of Truncated Newton methods, SIAM Journal

on Scientific and Statistical Computing 6, pp. 599 -616.

Ortega, J.M.; Rheinboldt, W.G. [1970]:Iterative Solution of Nonlinear Equations

in Several Variables, Academic Press, NY.

Ostrowski, A.M. [1973]: Solution of Equations in Euclidean and Banach Spaces,

Academic Press, New York.
 107

RaIl, L.B. [1984]: Differentiation in PASCAL - SC: Type Gradient, ACM Transac-
tions on Mathematical Software 10, pp. 161-184.

RaIl, L.B. [1987]: Optimal Implementation of Differentiation Arithmetic, in Com-

puter Arithmetic, Scientific Computation and Programming Languages, U. Kiilisch
(ed.), Teubner, Stuttgart.

Rheinboldt, W.C. [1986]: Numerical Analysis of Parametrized Nonlinear Equations,

J. Wiley, Interscience, New York.
Saad, Y.; Schultz, M.H. [1986]: GMRES: A generalized minimal residual algorithm
for solving nonsymmetric linear systems, SIAM Journal on Numerical Analysis
7, pp. 856-869.
Schmidt, J.W. [1987]: A class of superlinear decomposition methods in nonlinear
equations, Numerical Functional Analysis and Optimization 9, pp. 629 - 645.

Schmidt J.W.; Hoyer, W.; Haufe, Ch. [1985]: Consistent approximations in Newton
- type decomposition methods, Numerische Mathematik 47, pp. 413 - 425.
Schnabel, R.B.; Frank, P.D. [1984]: Tensor methods for nonlinear equations, SIAM
Journal on Numerical Analysis 21, pp. 815 - 843.

Shamanskii, V.E. [1967]: A modification of Newton's method, Ukrain Mat. Z. 19,

pp. 133-138.
Schwandt, H. [1984]: An interval arithmetic approach for the construction of an
almost globally convergent method for the solution of the nonlinear Poisson
equation on the unit square, SIAM Journal on Scientific and Statistical Com-
puting 5, pp. 427 - 452.

Schwetlick, H. [1978]: Numerische Losung Nichtlinearer Gleichungen, Deutscher

Verlag der Wissenschaften, Berlin.
Spedicato, E. [1991] (editor): Computer Algorithms for Solving Linear Algebraic
Equations. The State of Art, NATO ASI Series, Series F: Computer and Sys-
tems Sciences, Vol. 77, Springer Verlag, Berlin.

Spedicato, E.; Chen, Z.; Deng, N. [1992]: A class of difference ABS - type algo-
rithms for a nonlinear system of equations, Technical Report, Department of
Mathematics, University of Bergamo.

Tewarson, R.P. [1988]: A new Quasi-Newton algorithm, Applied Mathematics Let-

ters 1, pp. 101 - 104.

Tewarson, R.P.; Zhang, Y. [1987]: Sparse Quasi-Newton LDU update, International

Journal on Numerical Methods in Engineering 24, pp. 1093-1100.
108

Tompkins, C. [1955]: Projection methods in calculation, Proceedings Second Sym-

posium on Linear Programming, Washington D.C., pp. 425 - 448.

Walker, H.F.; Watson, L.T. [1989]: Least - Change Update Methods for under de-
termined systems, Research Report, Department of Mathematics, Utah State
University.

Watson, L.T. [1979]: An algorithm that is globally convergent with probability one
for a class of nonlinear two-point boundary value problems, SIAM Journal on
Numerical Analysis 16, pp. 394-401.

Watson, 1. T. [1980]: Solving finite difference approximations to nonlinear two-point

boundary value problems by a homotopy method,SIAM Journal on Scientific
and Statistical Computing 1, pp. 467-480.

Toint, Ph.L. [1986]: Numerical solution of large sets of algebraic nonlinear equa-
tions, Mathematics of Computation 16, pp. 175 - 189.

Watson, L.T. [1983]: Engineering applications of the Chow-Yorke algorithm, in

Homotopy Methods and Global Convergence (B.C. Eaves and M.J. Todd eds.),
Plenum, New York.

Watson, 1.T.; Billups, S.C.; Morgan, A.P. [1987]: Algorithm 652: HOMPACK: A
suite of codes for globally convergent homotopy algorithms, ACM Trans. Math.
Software 13, pp. 281-310.

Watson, L.T.; Scott, M.R. [1987]: Solving spline-collocation approximations to

nonlinear two-point boundary value problems by a homotopy method, Applied
Mathematics and Computation 24, pp. 333-357.

Watson, L.T.; Wang, C.Y. [1981]: A homotopy method applied to elastica prob-
lems, International Journal on Solid Structures 17, pp. 29-37.

Wolfe, P. [1959]: The secant method for solving nonlinear equations ,Communica-
tions ACM 12, pp. 12 - 13.

Zambaldi, M.C. [1990]: Estruturas estdticas e dinamicas para resolver sistemas niio
lineares esparsos, Tese de Mestrado, Departamento de Matematica Aplicada,
Universidade Estadual de Campinas, Campinas, Brazil.

Zlatev, Z.; Wasniewski, J.; Schaumburg, K. [1981]: Y12M. Solution of large and
sparse systems of linear algebraic equations, Lecture Notes in Computer Science
121, Springer-Verlag, New York, Berlin, Heidelberg and Tokyo.