Numerical Methods

Lecture Notes #01

Pavel Ludvk,
<[email protected]>

Department of Mathematics and Descriptive Geometry

VB-TUO

http://homen.vsb.cz/~lud0016/

February 10, 2016

Numerical Methods
Lecture Notes #01

M.Sc. Carlo Viggiano

2.09.2016
 Contact information

Carlo Viggiano, M.Sc. Power Engineering (TUM)

B.Sc. Electrical Engineering (U. Valle)
Director of Smart Grid and Renewable Energy Dpt. @ GERS

E-mail: [email protected]
Office Hours: by appointment
 Course information

Evaluation:

Exam 1: 35% (7 sessions, 21.10.2016)

Exam 2: 35% (6 sessions, 9.12.2016)
Project: 30% (SCILAB, 16.12.2016)

Attendance will not be evaluated at your own risk!

Optional exams: 4.01.2016, 6.01.2016 (not fixed yet)

Language lectures and tutorials: (Espaol! )

slides and books in English!

Programming and number language are universal

 Course information

Some statistics:
Content languages for websites:
English 53.6%
Spanish 4.9%

Internet users by language:

English 25.9%
Spanish 7.6%
 Course information

Slides and tutorials will be provided at least 3 days before the

lecture day.

It is expected that all students review material and try to solve

the excerice by their own before the lecture

Why?
1. Learning process is improved.
2. This semester is extremely short!

So always come with questions!

 Course information

About SCILAB:

Open source, cross-platform numerical

computational package and a high-level, numerically
oriented programming language.

Many applications: signal processing, statistical

analysis, image enhancement, fluid
dynamics simulations, numerical optimization, and
modeling, simulation of explicit and implicit dynamical
systems.
 Course information

About SCILAB:

Scilab is one of the two major open-source alternatives

to MATLAB, the other one being GNU Octave. Scilab is
similar enough to MATLAB that some book authors (who
use it) argue that it is easy to transfer skills between the two
systems.

Available here: http://www.scilab.org/

Tutorial SCILAB:
https://www.youtube.com/watch?v=0F8k-rXlVZ4
(en Espaol! )
Lecture Notes #01
Course Information
Book and Other Study Materials
The recommended text for the course is the book:

Title: Numerical Analysis

Authors: Richard L. Burden, John D. Faires
Edition: 9
Publisher: Cengage Learning, 2011
Lecture Notes #01
Course Information
Book and Other Study Materials
Other materials:

Title: Numerical Methods for Engineers

Authors: Steven Chapra, Raymond Canale
Edition: 6
Publisher: McGraw-Hill Education, 2009

Solved examples:
http://mdg.vsb.cz/wiki/public/ZM_NM_examples.pdf
My web: http://homen.vsb.cz/~lud0016/

Qaurteroni, A., Sacco, R., Saleri, F.: Numerical Mathematics. Springer,

2007.

Sli, E., Mayers, D.: An introduction to Numerical Analysis. Cambridge

University Press, 2003.
Lecture Notes #01
Course Information
Syllabus
What are numerical methods and what is it for?

Q: What are numerical methods?

A: Numerical methods are algorithms based on simple arithmetic

operations on numbers.

Q: What are numerical methods for?

A: To accurately approximate solutions of problems that cannot be solved

exactly. They reduce the dicult analytic problems to purely
arithmetical ones.

Q: What kind of applications can benet from numerical studies?

A: Image processing / computer vision, computer graphics (rendering,

animation), climate modeling, weather predictions, virtual
crash-testing of cars, medical imaging (CT = Computer Tomography),
AIDS research (virus decay vs. medication), nancial mathematics
Lecture Notes #01
Calculus Review

Calculus Review
Lecture Notes #01
Calculus Review

Q: Why to review calculus? In numerical mathematics??

A: When developing numerical schemes we will use theorems from calculus

to guarantee that our algorithms make sense.

Key concepts from calculus:

Limits

Continuity

Convergence

Dierentiability

Extreme Value Theorem

Intermediate Value Theorem

Mean Value Theorem

Taylor's Theorem
Lecture Notes #01
Calculus Review

Limit/Convergence

Denition (Limit)

A function f dened on a set X R has the limit L at x0 , written

lim f (x) = L
xx0

if for any sequence (xn ) approaching x0 a sequence (f (xn )) approaches L.

Denition (Continuity (at a point))

Let f be a function dened on a set X R, and x0 X . Then f is

continuous at x0 if
lim f (x) = f (x0 ).
xx0
Lecture Notes #01
Calculus Review

Continuity/Convergence

Denition (Continuity (in an interval))

A function f is continuous on a set X R (i.e., f C (X )) if it is

continuous at each point x X.
Lecture Notes #01
Calculus Review

Dierentiability
Theorem

If f is a function dened on a set X R and x0 X , then the following

statements are equivalent:
(a) f is continuous at x0
(b) If {xn }
n=1 is any sequence in X converging to x0 , then limn f (xn ) = f (x0 ).

Denition

Let f be a function dened on an open interval containing x0 (i.e.,

x0 (a, b)). Then f is dierentiable at x0 if

f (x) f (x0 )
f 0 (x0 ) = lim exists.
xx0 x x0

If the limit exists, we call f 0 (x0 ) a derivative of f at x0 .

Lecture Notes #01
Calculus Review

Continuity

Theorem (Dierentiability Continuity)

If f is dierentiable at x0 , then f is continuous at x0 .

Lecture Notes #01
Calculus Review

Extreme Value Theorem

Theorem (Extreme Value Theorem)

If f C [a, b] then
m, M [a, b]x [a, b] : f (m) f (x) f (M).

I.e., f attains its minimum at m and maximum at M .

Moreover, if f is dierentiable on (a, b) then the numbers m, M occur either
at endpoints a, b or where f 0 (x) = 0.
Lecture Notes #01
Calculus Review

Mean and Intermediate Value Theorem

Theorem (Mean Value Theorem)

If f C [a, b] and f is dierentiable on (a, b), then c (a, b) such that

f (b) f (a)
f 0 (c) = .
ba

Theorem (Intermediate Value Theorem)

If f C [a, b] and K (f (a), f (b)), then there ex ts a number c (a, b) for

which f (c) = K .
Lecture Notes #01
Calculus Review

Taylor's Theorem

Theorem (Taylor's Theorem)

Suppose f C [a, b], f (n+1) exists on (a, b) and x0 [a, b]. Then
x (a, b), (x0 , x) with

f (x) = Pn (x) + Rn (x)

where
Pn is called the Taylor polynomial of degree n, and Rn (x) is the
remainder term (truncation error).

This theorem is extremely important for numerical analysis:

Taylor expansion is a fundamental step in the derivation of many of the
algorithms we see in this class.
Lecture Notes #01
Computer Arithmetic and Finite Precision

Computer Arithmetic and Finite Precision

Lecture Notes #01
Computer Arithmetic and Finite Precision

Finite Precision: A 64-bit real number, double

The Binary Floating Point Arithmetic Standard 754-1985 (IEEE - The

Institute for Electrical and Electronics Engineers) standard specied the
following layout for a 64-bit real number:

sc10 c9 . . . c1 c0 m51 m50 . . . m1 m0

where

Symbol Bits Description

s 1 The sign bit: 0 = positive, 1 = negative
c 11 The characteristic (exponent)
m 52 The mantisa

10 51
X X mk
r = (1)s 2c1023 (1 + m), c= ck 2k , m=
252k
k=0 k=0
Lecture Notes #01
Computer Arithmetic and Finite Precision

Finite Precision: Examples

10 51
s c1023
X
k
X mk
r = (1) 2 (1 + m), c= ck 2 , m=
252k
k=0 k=0

Remarks:

10
2 = 1024 and (11111111111)2 = 2047.
We cannot represent an exact zero!

Example 1: 3.0

0 10000000000 1000000000000000000000000000000000000000000000000000

(1)0 22 1023 1 + = 1 2 1 = 3 .0
1 3
 
10

2 2
Lecture Notes #01
Computer Arithmetic and Finite Precision

Finite Precision: Examples

10 51
X X mk
r = (1)s 2c1023 (1 + m), c= ck 2k , m=
252k
k=0 k=0

Example 2: The Smallest Positive Real Number

0 00000000000 0000000000000000000000000000000000000000000000000001

r = (1)0 201023 1 + 52 = (1 + 252 ) 21023 1 10308

1
 

2
Example 3: The Largest Positive Real Number

0 11111111110 1111111111111111111111111111111111111111111111111111

r = (1)0 21023 1 + + 2 + + 51 + 52 = 21023 2 52 10308

1 1 1 1 1
   

2 2 2 2 2
Lecture Notes #01
Computer Arithmetic and Finite Precision

Finite Precision: Consequences

There are gaps in the oating-point representation. I.e., any number in

the interval

 
1
3.0, 3.0 +
252

is represented by value 3.0.

Floating point numbers represents intervals!

Lecture Notes #01
Computer Arithmetic and Finite Precision

Quantifying the Error

Let p be an approximation to p, then

Denition (The Absolute Error)

|p p |

Denition (The Relative Error)

|pp |
|p| , p 6= 0
Lecture Notes #01
Computer Arithmetic and Finite Precision

Sources of Numerical Errors - Roundo Errors (Rounding and

Truncating) I

Examples in 5-digit arithmetic

Rounding 5-digit arithmetic:

(96384 + 26.678) 96410 =
(96384 + 00027) 96410 =
96411 96410 = 1.0000

Truncating 5-digit arithmetic:

(96384 + 26.678) 96140 =
(96384 + 00026) 96410 =
96410 96410 = 0.0000
Lecture Notes #01
Computer Arithmetic and Finite Precision

Sources of Numerical Errors - Roundo Errors (Rounding and

Truncating) II

Rearrangement changes the result:

(96384 96410) + 26.678 = 26.000 + 26.678 = 0.67800

Numerically, order of computation matters!

Lecture Notes #01
Algorithms

Algorithms
Lecture Notes #01
Algorithms

Denition (Algorithm)

An algorithm is a procedure that describes, in an unambiguous manner, a

nite sequence of steps to be performed in a specic order.

In this class, the objective of an algorithm is to solve a problem or

approximate a solution to a problem.
Algorithms work very similarly to the meal recipes.
Lecture Notes #01
Algorithms

Key Concepts for Numerical Algorithms  Stability

Denition (Stability)

An algorithms is said to be stable if small changes in the input, generate

small changes in the output.

At some point we need to quantify what small means!

If an algorithm is stable for a certain range of initial data, then it is said to
be conditionally stable.
Lecture Notes #01
Algorithms

Key Concepts for Numerical Algorithms  Error Growth

Suppose E0 > 0 denotes the initial error, and En represents the error after n
operations.

If En C E0 n (for a constant C which is independat of n), then the

growth is linear.
If En C n E0 , C > 1, the the growth is exponential  in this case the
error will dominate very fast (undesirable scenario).

Linear error growth is usually unavoidable, and in the case where C and
E0 are small the results are generally acceptable.  Stable algorithm.

Exponential error growth is unacceptable. Regardless of the size of E0 the

error grows rapidly.  Unstable algorithm.
Lecture Notes #01
Algorithms

Reducing the Eects of Roundo Error

The eects of roundo errors can be reduced by using higher-order-digit

arithmetic such as the double or multiple-precision arithmetic available on
most computers.

Disadvantages in using double precision arithmetic are that it takes more

computation time andthe growth of the roundo error is not
eliminated but only postponed.
Sometimes, but not always, it is possible to reduce the growth of the
roundo error by restructuring the calculations.
Lecture Notes #01
Algorithms

Key Concepts - Rate of Convergence

Denition (Rate of Convergence)

Suppose the sequence = {n }

n=1 converges to zero, and = {n }n=1
converges to a number .

If K > 0 : |n | < K n , for n large enough, then we say that {n }n=1
converges to with
a Rate of Convergence O(n ) (Big Oh of n ).
We write
n = + O(n )

Note: The sequence = {n }

n=1 is usually chosen to be

1
n =
np
for some positive value of p.
Lecture Notes #01
Algorithms

Rate of Convergence: Example

Consider the sequence (as n )

 
1 1
n = sin .
n n
1


n3
Then n = O .
Lecture Notes #01
Algorithms

Generalizing to Limits of Functions

Denition (Rate of Convergence)

Suppose
lim G (h) = 0, and lim F (h) = L.
h0 h0

If K > 0h < H (for some H > 0):

|F (h) L| K |G (h)|

then
F (h) = L + O(G (h)).
We say that F (h) converges to L with a Rate of convergence O(G (h)).
Note: Usually we consider G (h) = hp for some positive p.
Lecture Notes #01
Algorithms

Rate of Convergence: Example

Consider the function

1
(h) = sin(h) .
h
Then (h) = O(h3 ).