Introduction to Numerical

Analysis

 What are NUMERICAL METHODS?

 Why do we need them?
 Topics covered in NA.

1
Motivation: Numerical
Methods
 Numerical methods are techniques where
mathematical problems are formulated so that they
can be solved with arithmetic operations.

 Although there are many kinds of numerical

methods, they have one common characteristic:
they invariably involve large numbers of tedious
arithmetic calculations.
 It is little wonder that with the development of fast,
efficient digital computers, the role of numerical
methods in engineering problem solving has
increased dramatically in recent years.

2
Numerical Methods
Why do we need them?
1. No analytical solution exists,
2. An analytical solution is difficult to obtain
or not practical.

3
What do we need?
Basic Needs in the Numerical Methods:
 Practical:
Can be computed in a reasonable amount of time.
 Accurate:
 Good approximate to the true value,
 Information about the approximation error

(Bounds, error order,… ).

4
Outlines of the Course
 Taylor Theorem  Least Squares curve
 Number fitting
Representation  Solution of ordinary
 Solution of nonlinear differential equations
Equations
 Interpolation
 Numerical
Differentiation
 Numerical Integration

5
Solution of Nonlinear
Equations
 Some simple equations can be solved analytically:
x 2  4 x  3 0
 4  4 2  4(1)(3)
Analytic solution roots 
2(1)
x  1 and x  3

 Many other equations have no analytical solution:

x 9  2 x 2  5 0

 x  No analytic solution
x e 

6
Methods for Solving Nonlinear
Equations

o Bisection Method

o Newton-Raphson Method

o Secant Method

7
Solution of Systems of Linear
Equations
x1  x2 3
x1  2 x2 5
We can solve it as :
x1 3  x2 , 3  x2  2 x2 5
 x2 2, x1 3  2 1
What to do if we have
1000 equations in 1000 unknowns.
8
Cramer’s Rule is Not
Practical
Cramer' s Rule can be used to solve the system :
3 1 1 3
5 2 1 5
x1  1, x2  2
1 1 1 1
1 2 1 2

But Cramer' s Rule is not practical for large problems.

To solve N equations with N unknowns, we need (N  1)(N  1)N!
multiplications.
To solve a 30 by 30 system, 2.3 1035 multiplications are needed.
A super computer needs more than 1020 years to compute this.
9
Methods for Solving Systems
of Linear Equations
Gaussian Elimination

10
Curve Fitting
 Given a set of data:

x 0 1 2
y 0.5 10.3 21.3

 Select a curve that best fits the data. One

choice is to find the curve so that the sum
of the square of the error is minimized.

11
Interpolation
 Given a set of data:

xi 0 1 2
yi 0.5 10.3 15.3

 Find a polynomial P(x) whose graph passes

through all tabulated points.

yi P ( xi ) if xi is in the table
12
Methods for Curve Fitting
o Least Squares
o Linear Regression
o Nonlinear Least Squares Problems

o Interpolation
o Newton Polynomial Interpolation
o Lagrange Interpolation

13
Integration
 Some functions can be integrated
analytically:

3 3
1 2 9 1

1
xdx  x   4
2 1 2 2
But many functions have no analytical solutions :
a
 x2
e
0
dx ?

14
Methods for Numerical
Integration
o Upper and Lower Sums

o Trapezoid Method

o Gauss Quadrature

15
Solution of Ordinary Differential
Equations
A solution to the differential equation :
x(t )  3 x (t )  3 x(t ) 0
x (0) 1; x(0) 0
is a function x(t) that satisfies the equations.

* Analytical solutions are available for

special cases only.
16
Summary
 Numerical Methods:
Algorithms that are used to obtain numerical solution of a
mathematical problem.

 We need them when

No analytical solution exists or it is difficult to obtain it.

17
Number Representation and
Accuracy
 Number Representation
 Normalized Floating Point Representation
 Significant Digits
 Accuracy and Precision
 Rounding and Chopping

18
Representing Real Numbers
 You are familiar with the decimal system:

312.45 3 10 2  1101  2 100  4 10  1  5 10  2

 Decimal System: Base = 10 , Digits (0,1,…,9)

 Standard Representations:

 3 1 2 . 4 5
sign integral fraction
part part
19
Normalized Floating Point
Representation
 Normalized Floating Point Representation:

 d . f1 f 2 f 3 f 4  10n
sign mantissa exponent

d 0, n : signed exponent

 Scientific Notation: Exactly one non-zero digit appears
before decimal point.
 Advantage: Efficient in representing very small or very
large numbers.
20
Binary System
 Binary System: Base = 2, Digits {0,1}

 1. f1 f 2 f 3 f 4  2 n
sign mantissa signed exponent

(1.101)2 (1  1 2  1  0 2  2  1 2  3 )10 (1.625)10

21
Fact
 Numbers that have a finite expansion in one numbering
system may have an infinite expansion in another
numbering system:

(1.1)10 (1.0001100110 01100 ...)2

 You can never represent 1.1 exactly in binary system.

22
IEEE 754 Floating-Point
Standard
 Single Precision (32-bit representation)
 1-bit Sign + 8-bit Exponent + 23-bit Fraction

S Exponent8 Fraction23

 Double Precision (64-bit representation)

 1-bit Sign + 11-bit Exponent + 52-bit Fraction

S Exponent11 Fraction52
(continued)

23
Significant Digits

 Significant digits are those digits that can be

used with confidence.
 Single-Precision: 7 Significant Digits

1.175494… × 10-38 to 3.402823… × 1038

 Double-Precision: 15 Significant Digits

2.2250738… × 10-308 to 1.7976931… × 10308

24
Remarks
 Numbers that can be exactly represented are called
machine numbers.

 Difference between machine numbers is not uniform

 Sum of machine numbers is not necessarily a machine

number

25
Calculator Example
 Suppose you want to compute:
3.578 * 2.139
using a calculator with two-digit fractions

3.57 * 2.13 = 7.60

True answer: 7.653342

26
Significant Digits - Example
48.9

27
Accuracy and Precision

 Accuracy is related to the closeness to the true

value.

 Precision is related to the closeness to other

estimated values.

28
29
Rounding and Chopping

 Rounding: Replace the number by the nearest

machine number.

 Chopping: Throw all extra digits.

30
Error Definitions – True Error

Can be computed if the true value is known:

Absolute True Error

Et  true value  approximation
Absolute Percent Relative Error
true value  approximation
t  *100
true value

31
Error Definitions – Estimated Error

When the true value is not known:

Estimated Absolute Error

Ea  current estimate  previous estimate
Estimated Absolute Percent Relative Error
current estimate  previous estimate
a  *100
current estimate

32
Notation
We say that the estimate is correct to n
decimal digits if:
n
Error 10

We say that the estimate is correct to n

decimal digits rounded if:
1 n
Error  10
2

33
Summary
 Number Representation
Numbers that have a finite expansion in one numbering system
may have an infinite expansion in another numbering system.

 Normalized Floating Point Representation

 Efficient in representing very small or very large numbers,
 Difference between machine numbers is not uniform,
 Representation error depends on the number of bits used in
the mantissa.

34
 Taylor Theorem
 Motivation
 Taylor Theorem
 Examples

35
Motivation
 We can easily compute expressions like:
3 10 2
2( x  4)
But, How do you compute 4.1, sin(0.6) ?

Can we use the definition

b
a
to compute sin(0.6)?
0.6
Is this a practical way?

36
Remark

 In this course, all angles are assumed to

be in radian unless you are told otherwise.

37
Taylor Series
The Taylor series expansion of f ( x ) about a :
' f ( 2) (a ) 2 f ( 3) ( a )
f (a )  f (a ) ( x  a )  ( x  a)  ( x  a ) 3  ...
2! 3!
or

1 (k )
Taylor Series   f (a ) ( x  a )k
k 0 k!
If the series converge, we can write :
∞
1 (k )
f ( x) ∑ f (a ) ( x  a )k
k 0
k!

38
Maclaurin Series
 Maclaurin series is a special case of Taylor
series with the center of expansion a = 0.
The Maclaurin series expansion of f ( x ) :
( 2) ( 3)
' f ( 0 ) 2 f ( 0) 3
f ( 0)  f ( 0) x  x  x  ...
2! 3!
If the series converge, we can write :
∞
1 (k )
f ( x) ∑ f ( 0) x k
k 0
k!
39
Maclaurin Series – Example 1
Obtain Maclaurin series expansion of f ( x ) e x
f ( x ) e x f (0) 1
f ' ( x ) e x f ' (0) 1
f ( 2 ) ( x ) e x f ( 2 ) (0) 1
f ( k ) ( x ) e x f ( k ) (0) 1 for k 1
∞ ∞
1 (k ) xk x2 x3
e x ∑ f ( 0) x  ∑
k
1  x    ...
k 0
k! k 0
k! 2! 3!
The series converges for x  ∞.
40
Taylor Series
3

Example 1
2.5
exp(x)
1+x+0.5x 2
2

1+x

1.5

1
1

0.5

0
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

41
Maclaurin Series – Example 2
Obtain Maclaurin series expansion of f ( x ) sin( x ) :
f ( x ) sin( x ) f (0) 0
f ' ( x ) cos( x ) f ' (0) 1
f ( 2 ) ( x )  sin( x ) f ( 2 ) (0) 0
f ( 3) ( x )  cos( x ) f ( 3) (0)  1
∞
f ( k ) ( 0) k x3 x5 x7
sin( x )  ∑ x x     ....
k 0
k! 3! 5! 7!
The series converges for x  ∞.
42
4

3
x

1 x-x 3/3!+x 5/5!

0 sin(x)

-1
x-x 3/3!

-2

-3

-4
-4 -3 -2 -1 0 1 2 3 4

43
Maclaurin Series – Example 3
Obtain Maclaurin series expansion of : f ( x) cos( x)
f ( x ) cos( x ) f (0) 1
f ' ( x )  sin( x ) f ' (0) 0
f ( 2 ) ( x )  cos( x ) f ( 2 ) (0)  1
f ( 3) ( x ) sin( x ) f ( 3) (0) 0
∞
f ( k ) ( 0) x 2
x 4
x 6
cos( x )  ∑ ( x ) k 1     ....
k 0
k! 2! 4! 6!
The series converges for x  ∞.
44
Maclaurin Series – Example 4
1
Obtain Maclaurin series expansion of f(x) 
1 x
1
f ( x)  f (0) 1
1 x
1
f ' ( x)  2
f ' (0) 1
1  x 
( 2) 2 ( 2)
f ( x)  3
f (0) 2
1  x 
6
f ( 3) ( x )  4
f ( 3)
(0) 6
1  x 
1
Maclaurin Series Expansion of : 1  x  x 2  x 3  ...
1 x
Series converges for | x |  1
45
Example 4 - Remarks
 Can we apply the series for x≥1??

 How many terms are needed to get a good

approximation???

These questions will be answered using

Taylor’s Theorem.

46
Taylor Series – Example 5
1
Obtain Taylor series expansion of f(x)  at a 1
x
1
f ( x)  f (1) 1
x
1
f ' ( x)  2 f ' (1)  1
x
2
f ( 2) ( x )  3 f ( 2 ) (1) 2
x
6
f ( 3) ( x )  4 f ( 3) (1)  6
x
Taylor Series Expansion ( a 1) : 1  ( x  1)  ( x  1) 2  ( x  1) 3  ...
47
Taylor Series – Example 6
Obtain Taylor series expansion of f(x) ln( x ) at ( a 1)

1 ( 2) 1 ( 3) 2
f ( x ) ln( x ) , f ' ( x )  , f ( x )  2 , f ( x )  3
x x x
f (1) 0, f ' (1) 1, f ( 2 ) (1)  1 f ( 3) (1) 2

1 2 1
Taylor Series Expansion : ( x  1)  ( x  1)  ( x  1) 3  ...
2 3

48
Convergence of Taylor
Series

 The Taylor series converges fast (few terms

are needed) when x is near the point of
expansion. If |x-a| is large then more terms
are needed to get a good approximation.

49
Taylor’s Theorem
If a function f ( x ) possesses derivatives of orders 1, 2, ..., ( n  1)
on an interval containing a and x then the value of f ( x ) is given by :

(n+1) terms Truncated

Taylor Series
n
f ( k ) (a )
f ( x)  ∑ k!
( x  a)k  Rn
k 0
Remainder
where :
f ( n 1) ( )
Rn  ( x  a ) n 1 and  is between a and x.
( n  1)!
50
Taylor’s Theorem
We can apply Taylor' s theorem for :
1
f(x)  with the point of expansion a 0 if | x |  1.
1 x

If x 1, then the function and its

derivatives are not defined.
 Taylor Theorem is not applicable.

51
Error Term
To get an idea about the approximation error,
we can derive an upper bound on :
( n 1)
f ( )
Rn  ( x  a ) n 1
( n  1)!
for all values of  between a and x.

52
Error Term - Example
x
How large is the error if we replaced f ( x ) e by
the first 4 terms ( n 3) of its Taylor series expansion
at a 0 when x 0.2 ?
f ( n ) ( x ) e x f ( n ) ( ) ≤ e 0.2 for n ≥ 1
f ( n 1) ( )
Rn  ( x  a ) n 1
( n  1)!
e 0.2 n 1
Rn  0.2   R3 8.14268E  05
( n  1)!
53
0.00008142685
Alternative form of Taylor’s
Theorem
Let f ( x ) have derivatives of orders 1, 2, ..., ( n  1)
on an interval containing x and x  h then :
n (k )
f ( x) k
f ( x  h)   h  Rn ( h step size)
k 0 k!

f ( n 1) ( ) n 1
Rn  h where  is between x and x  h
( n  1)!

54
Taylor’s Theorem –
Alternative forms
n ( n 1)
f ( k ) (a ) f ( )
f ( x)   k
( x  a)  ( x  a ) n 1
k 0 k! ( n  1)!
where  is between a and x.

a  x, x  x  h

n
f ( k ) ( x ) k f ( n 1) ( ) n 1
f ( x  h)   h  h
k 0 k! ( n  1)!
where  is between x and x  h.
55
Mean Value Theorem
If f ( x ) is a continuous function on a closed interval [a , b]
and its derivative is defined on the open interval ( a , b)
then there exists ξ  ( a , b)
f(b)  f(a)
f ' (ξ ) 
b a
Proof : Use Taylor' s Theorem for n 0, x a , x  h b
f(b)  f(a)  f ' (ξ ) (b  a )

56
Alternating Series Theorem
Consider the alternating series :
S a1  a2  a3  a4  
 a a a a  The series converges
 1 2 3 4

If  and then  and
 lim a 0  S  S n an 1
 n  n 

S n : Partial sum (sum of the first n terms)

an 1 : First omitted term
57
Alternating Series –
Example
1 1 1
sin(1) can be computed using : sin(1) 1     
3! 5! 7!
This is a convergent alternating series since :
a1 a2 a3 a4  and lim an 0
n 

Then :
 1 1
sin(1)   1   
 3!  5!
 1 1 1
sin(1)   1    
 3! 5!  7!
58
Example 7
Obtain the Taylor series expansion
of f ( x ) e 2 x 1 at a 0.5 (the center of expansion)
How large can the error be when ( n  1) terms are used
to approximate e 2 x 1 with x 1 ?

59
Example 7 – Taylor Series
Obtain Taylor series expansion of f ( x ) e 2 x 1 , a 0.5
f ( x) e 2 x 1 f (0.5) e 2
f ' ( x) 2e 2 x 1 f ' (0.5) 2e 2
f ( 2) ( x) 4e 2 x 1 f ( 2) (0.5) 4e 2
f ( k ) ( x) 2 k e 2 x 1 f ( k ) (0.5) 2 k e 2
∞
f ( k ) (0.5)
e 2 x 1
∑ ( x  0.5) k
k 0
k!

2 2 ( x  0.5) 2
2 k 2 ( x  0.5)
k
e  2e ( x  0.5)  4e  ...  2 e  ...
2! k!
60
Example 7 – Error Term
f ( k ) ( x) 2 k e 2 x 1
f ( n 1) ( )
Error  ( x  0.5) n 1
(n  1)!
n 1 2 1 (1  0.5) n 1
Error  2 e
(n  1)!
n 1
( 0.5)
Error 2 n 1 max e 2 1
(n  1)!  [ 0.5,1]
e3
Error 
(n  1)!
61