Newton’s Method

Relaxation
g(x) = x (x)f (x)
I (x) is differentiable in neighborhoods of fixed points of g.
I Simple Iteration: (xk ), xk +1 = xk (xk )f (xk )
Newton’s Method

Suppose limk !1 xk = x⇤ , then g(x⇤ ) = x⇤ .

x⇤ = g(x⇤ ) = x⇤ (x⇤ )f (x⇤ )

=) f (x⇤ ) = 0 provided (x⇤ ) 6= 0

Newton’s Method
Choose (x) so that |g 0 (x⇤ )| is as small as possible.
|g 0 (x⇤ )| = |1 (x⇤ )f 0 (x⇤ ) 0
(x⇤ )f (x⇤ )|
0
= |1 (x⇤ )f (x⇤ )|

For instance, choose (x) = 1/f 0 (x) leading to Newton’s Method (Continued)
f (x)
g(x) = x .
f 0 (x)

Definition
Newton’s method to solve the equation f (x) = 0 is defined by

f (xk )
xk +1 = xk , k 0
f 0 (xk )

for a given x0 2 R.

Newton’s Method Newton’s Method

f (x)
Simple iteration on g(x) = x
f 0 (x) Geometric interpretation
xk +1 by Newton’s method is such that
Newton Sequence
f (xk ) `(xk +1 ) = 0, where `(x) = f (xk ) + f 0 (xk )(x xk ).
(xk ) s.t. xk +1 = xk , k 0 for a given x0 2 R
f 0 (xk )

20

Motivation.
g(x⇤ ) = x⇤ =) f (x⇤ ) = 0, g 0 (x⇤ ) = 0 10

(assuming f 0 (x⇤ ) 6= 0)
0
Superlinear Convergence.
|xk +1 x⇤ | -10
lim = |g 0 (x⇤ )| = 0
k !1 |xk x⇤ |
(assuming limk !1 xk = x⇤ s.t. f 0 (x⇤ ) 6= 0) -3 -2 -1 0 1 2 3
Newton’s Method Newton’s Method
4

2
Example: f (x) = ex x 2

j
o
0

da Y Newton sequence xx
-2
exk xk 2
xk +1 = xk
exk 1
-4 Xz
-6
for a given x0 2 R.
IC xx

-8
0 0.5 1 1.5 2

Higher Orders of Convergence Higher Orders of Convergence

Definition
Suppose limk !1 xk = x⇤ , and there exists "k such that Examples:
k
|xk x⇤ |  "k , as well as 1. (10 2 ) = (10 1 , 10 2 , 10 4 , . . . ) converges to 0
"k +1 quadratically, i.e.,
lim = µ
k !1 ("k )q 2k +1 2k +1
10 10
lim 2k
= lim 2k +1
= 1.
for some q > 1 and µ > 0. k !1 (10 )2 k !1 10

1. (xk ) converges to x⇤ with at least order q.

2. If "k = |xk x⇤ |, then xk converges to x⇤ with order q.
3. If "k = |xk x⇤ | and q = 2, then xk converges to x⇤
2. (10 pk ), p
wittiest
> 1 converges to 0 with order p.

10 31
1
quadratically.

cubic
l im 112 51
Eat Xk X l and 9 3 then
convergence
a
posy

Newton’s Method

Theorem
Let x⇤ 2 R be such that f (x⇤ ) = 0 and f 00 (x) be continuous on
[x⇤ , x⇤ + ] for some > 0.
Furthermore, suppose f 0 (x⇤ ) 6= 0, f 00 (x⇤ ) 6= 0, and
f 00 (x)
 A 8x, y 2 [x ,x + ]
Secant Method
f 0 (y )
for some A > 0.

The sequence (xk ) by Newton’s method converges to x⇤

quadratically for all x0 2 [x⇤ h, x⇤ + h], h := min{ , 1/A}.

I When f 00 (x⇤ ) = 0, a convergence of order 3 (i.e., cubic

convergence) an exercise for you
Secant Method Secant Method
Approximates f 0 (xk ) in Newton’s method with Geometric interpretation
f (xk ) f (xk 1 ) xk +1 by secant method is such that
f 0 (xk ) ⇡ f (xk ) f (xk 1 )
xk xk 1 s(xk +1 ) = 0, where s(x) = f (xk ) + (x xk ).
xk xk 1

Definition 20
Secant method to solve the equation f (x) = 0 is defined by
15
f (xk )(xk xk 1 ) 10
xk +1 = xk , k 0
f (xk ) f (xk 1 )
5
for given x0 , x1 2 R.
0
I Does not fit into xk +1 = g(xk ).
-5
I Fits into xk +1 = g(xk , . . . , xk ` ) with ` = 1.
-3 -2 -1 0 1 2 3

Secant Method Secant Method

4

Example: f (x) = ex x 2
2
Xo x

0
Y The sequence by the secant method

(exk xk 2)(xk xk 1 )
X2
I xk +1 = xk
(exk xk ) (exk 1 xk 1 )

-2 44 for a given x0 , x1 2 R.

-4
0 0.5 1 1.5 2