ENDG 407

COMPUTATIONAL NUMERICAL METHODS

Chapter 1: Introduction
and Mathematical Background

Dr. Hongzhou Yang

University of Calgary
Schulich School of Engineering
Department of Geomatics Engineering

ENDG 407 - Computational Numerical Methods 1.1

ENDG 407 - Computational Numerical Methods 1.2
 Introduction (1)

v What is numerical analysis?

q It is the development and analysis/study of algorithms (methods, techniques)
for obtaining numerical solutions to mathematical problems
q Numerical methods are mathematical techniques for solving numerically,
rather than analytically, mathematical problems
q Often called mathematics of scientific computing as it involves a computer
program or code to communicate the algorithm to the computer
v Analytical versus Numerical Methods
q Example 1.1:
u Find the minimum of the function y = x2 – 3x + 2
q Analytical (and exact) solution:
u We know from calculus that the minimum occurs where dy/dx = 0, i.e.,
dy 3
= 0 ⇒ 2x − 3 = 0 ⇒ xmin = = 1.5
dx 2
ENDG 407 - Computational Numerical Methods 1.3
 Introduction (2)

q Numerical (and approximate) solution:

u Compute the values of y over a range of x values at a constant increment Dx, and
select the value of x for which the corresponding value of y is the smallest
o E.g., if we choose Dx = 0.2 and 1 ≤ x ≤ 2, we get

x 1.0 1.2 1.4 1.6 1.8 2.0

y 0 -0.16 -0.24 -0.24 -0.16 0

which indicates that 1.4 < xmin < 1.6

o To pinpoint the solution, we must repeat (iterate) the solution with smaller interval
u Notes:
o How quickly and how precisely we approximate the solution depends on the
increment value, the search interval and the initial value of x
o Numerical analysis helps us to choose efficient algorithms, proper values for the
search interval, the increment and the initial value, and to estimate the error in the
approximation
o Error estimation is necessary to assess the quality of the solution
ENDG 407 - Computational Numerical Methods 1.4
 Introduction (3)

Αnalytical Methods Numerical Methods

Advantages • Direct and exact • Can be used with complex
solutions functions/problems
• Efficient solutions • Easy to include constraints
Disadvantages • Practical only for simple • Require considerable
functions/problems amount of iterations/time
• Become complex when • Solutions are not exact and
constraints are involved require initial estimates

v Process of solving an engineering problem

q Problem statement
u Definition of the problem
u Parameters and variables involved
u Constraints/conditions, if any
u Specifications
o Required “quality” of the solution
ENDG 407 - Computational Numerical Methods 1.5
 Introduction (4)

q Problem modeling (formulation of the solution)

u Physical law that describes the problem
u Equation(s) that need to be solved
u Choice of analytical or numerical solution

q Programming algorithm for obtaining the solution

u Selection of numerical method/technique that will yield the solution, depending on
o accuracy, efficiency (computing time), programming difficulty
u Computer implementation
o Algorithm: plan/steps of carrying out the numerical method
o Code: List of computer commands for the execution of the algorithm
q Interpretation of the solution
u Validation of the solution
u Quality of the estimated solution
o Error analysis and error propagation

ENDG 407 - Computational Numerical Methods 1.6

 Introduction (5)

q Example 1.2: Show the problem modeling, simple programming algorithm

and code for finding the real roots of the quadratic equation ax2 + bx + c = 0
u Modeling:
−b ± D
o For D = b 2 − 4ac ≥ 0, x1,2 = For D < 0, there are no real roots
2a
u Algorithm:
o Calculate the value of D
o If D ≥ 0, calculate x1 and x2 from the equation above, and display their values
o If D < 0, display message "The equation has no real root"
u Code:
o a = input('Enter a '); b = input('Enter b '); c =
input('Enter c ');
o D = b^2 – 4*a*c
o if D >= 0
o x1 = (-b+sqrt(D))/(2*a); x2 = (-b-sqrt(D))/(2*a)
o fprintf('x1 = %f, x2 = %f ', x1, x2)
o else
o disp('Equation has no real roots.')
o end
ENDG 407 - Computational Numerical Methods 1.7
 Introduction (9)

v How numbers are represented on a computer

q Base-b number representation
u Any integer digit b > 1 can be used as the base (or radix) for a number system
o If b > 10, then usually letters like A,B,C,D,E,F, … are introduced to represent
10,11,12,13,14,15, …, b-1, respectively; see for example base-16 on next page
u Numbers N in base b are represented by a sequence of digits 0, 1, 2, …, b-1
corresponding to multiples of powers of b as follows:
n m
N β = [an an−1 ...a2 a1a0 . b1b2 ...bm ]β = [an an−1 ...a2 a1a0 ]β + [0.b1b2 ...bm ]β = ∑ ak β + ∑ bk β − k
k

k=0 k=1
u Common base-b representations
o Decimal (b = 10) or base-10 representation – digits: 0,1,2,3,4,5,6,7,8,9
² 12345.6710 = 1∙104+2∙103+3∙102+4∙101+5∙100 + 6∙10-1+7∙10-2
² The RHS of the formula above converts a number from any base to the decimal one
o Binary (b = 2) or base-2 representation – digits: 0,1
² 11111.1112 = 1∙24+1∙23+1∙22+1∙21+1∙20 + 1∙2-1 + 1∙2-2+ 1∙2-3 =
= 16+8+4+2+1 + (7/8) = 31.87510

ENDG 407 - Computational Numerical Methods 1.8

 Introduction (10)
o Octal (b = 8) or base-8 representation – digits: 0,1,2,3,4,5,6,7
² 12345.678 = 1∙84+2∙83+3∙82+4∙81+5∙80 + 6∙8-1 + 7∙8-2 =
= 4096+1024+192+32+5 + (55/64) = 5349.85937510
o Hexadecimal (b = 16) or base-16 representation: 0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F
² 7AF2.C9E16 = 7∙163+10∙162+15∙161+2∙160 + 12∙16-1+9∙16-2+14∙16-3 =
= 28762+2560+240+2 + (3230 /4096) = 31564.7885742187510

Hex 0 1 2 3 4 5 6 7 8 9 A B C D E F

Dec 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Oct 00 01 02 03 04 05 06 07 10 11 12 13 14 15 16 17

Bin 0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111

u Notes:
o Numbers with finite representation in one base may have infinite representation in an-
other, e.g., [0.1]10 = [0.0 0011 0011 0011 …]2 ⇨ truncation and round-off errors
o There is also a need to store very large and very small numbers, for which we use a
floating point representation with signs, exponents and mantissas ⇨ overflow and
underflow errors
ENDG 407 - Computational Numerical Methods 1.9
 Introduction (11)
o Most computers store and process numbers in binary form
² 0 and 1 can be represented by the off and on position of a switch (transistor)
² 0's and 1's are called binary digits or bits
² Thus there is a need to convert numbers from decimal to binary (and between any two bases)

q Converting a number N from base-a to base-b

u Procedure 1 (applies to both the integer and the fractional part of N):
o Express Na in nested form using powers of a
n m
Nα = ∑ akα + ∑ bkα − k = a0 + α (a1 + α (a2 + ...+ α (an−1 + α (an ))...)) +
k

k=0 k=1

+α −1 (b1 + α −1 (b2 + ...+ α −1 (bm−1 + α −1 (bm ))...))

o Replace each resulting digit by the corresponding base-b number
o Carry out the indicated arithmetic operation in base-b
o Example 1.3a – decimal to binary conversion: Convert (3781.75)10 to binary
3781.7510 = 1+10(8+10(7+10(3))) + 10-1(7+10-1(5)) =
= 12+10102 ∙ (10002+10102 ∙ (1112+10102 ∙ 112)) + 10102-1∙ (1112+10102-1 ∙ 1012) =
= 111011000101.112

ENDG 407 - Computational Numerical Methods 1.10

 Introduction (12)
o Because it involves arithmetic in different bases, Procedure 1 is efficient for
computers but tedious for humans and for hand computations
u Procedure 2 (treats separately the integer and the fractional part of N)
o For the integer part of N, we seek to find the digits ai such that
[an an−1 ...a2 a1a0 ]β = a0 + β (a1 + β (a2 + ...+ β (an−1 + β (an ))...))
o Observing that division by b yields
² a remainder equal to a0, and
² a quotient equal to a1 + β (a2 + ...+ β (an−1 + β (an )...)
we see that the ai's, from a0 to an, can be obtained by successive divisions of the
quotients by b
o Example 1.3b – For the integer part of 3781.7510 we get:
Quotients Remainders
2)3781 2)118 0 = a4 2)3 1 = a9
2)1890 1 = a0 2)59 0 = a5 2)1 1 = a10
2)945 0 = a1 2)29 1 = a6 0 1 = a11
2)472 1 = a2 2)14 1 = a7
2)236 0 = a3 2)7 0 = a8 Thus, 378110 = 1110110001012

ENDG 407 - Computational Numerical Methods 1.11

 Introduction (13)
o For the fractional part of N, we seek to find the digits bi such that
[0.b1b2 ...bm−1bm ]β = β −1 (b1 + β −1 (b2 + ...+ β −1 (bn−1 + β −1 (bn ))...)
o Observing that multiplication by b yields [b1. b2…bm-1bm]b , i.e., a number with
² an integer part equal to b1, and
² a fractional part equal to 0. b2…bm-1bm
we see that the bi's, from b1 to bm, can be obtained by successive multiplications of the
fractional parts by b
o Example 1.3b – For the fractional part of 3781.7510 we get:
2 ∙ Fractional Integer
2 ∙ 0.75
1.50 1 = b1
2 ∙ 0.50
1.00 1 = b2
2 ∙ 0.00 Thus 0.7510 = .112 and,
0.00 0 = b3 combining both parts, 3781.7510 = 111011000101.112

u Notes:
o Procedure 1 is preferred when a < b and Procedure 2 is preferred when a > b
ENDG 407 - Computational Numerical Methods 1.12
 Introduction (14)
o When converting between decimal and binary by hand, it is convenient to use the
octal representation as an intermediate step because the 8⟷2 conversion is very simple,
i.e., instead of 10⟷2 directly, we can carry out a 10⟷8⟷2 conversion in two steps:
² 1st step: 10⟷8 with one of the two procedures discussed previously
² 2nd step: 8⟷2 by translating groups of three binary digits directly into single octal digits using
this table
Oct 0 1 2 3 4 5 6 7
² e.g., [551.624]8 =
Bin 000 001 010 011 100 101 110 111
= [101 101 001.110 010 100]2
² Proof: If we start from a fractional binary number x2 = [0.b1b2b3b4b5b6…]2 we can express it as
x2 = b12-1 + b22-2 + b32-3 + b42-4 + b52-5 + b62-6 + …
= (4b1+ 2b2 + b3)8-1 + (4b4+ 2b5 + b6)8-2 + … = c18-1 + c28-2 + … = [0.c1c2c3…]8
In the above, we have set the sums in the parentheses equal to the ci coefficients in base-8
because, since each bi is either 0 or 1, the sums in the parentheses are digits between 0 and 7,
i.e., the digits of base-8
o With exactly the same logic, when converting between decimal and hexadecimal by
hand, it is convenient to use 10⟷8⟷2⟷16 because the 16⟷2 conversion is also very
simple: each group of four binary digits translates directly into a hexadecimal digit
² e.g., [2BAD]16 = [0010 1011 1010 1101]2

ENDG 407 - Computational Numerical Methods 1.13

 Introduction (15)
q Floating point representation
u Used in order to accommodate real numbers that are very small or very large
u Decimal floating point representation or scientific notation
o Form: ± d0 . d1d2 d3d4 …dn ⋅10 p
o 0. d1d2d3d4…dn is called the mantissa
o p (an integer) is called the exponent
o 10p represents the number's order of magnitude
² If the preceding number is smaller than 5, the number is of order 10p, denoted as O(10p)
² Otherwise the number is of order 10p+1, O(10p+1)
o n+1 is the number of significant digits (where dn ≠ 0?)
o Example 1.4:
standard notat. scientific notat. order of magn. # of sign. digits
4519.23 4.51923 ∙103 O(103) 6
732.5051 7.325051∙102 O(103) 7
-0.00012 -1.2∙10-4 O(10-4) 2
-0.005612 -5.612∙10-3 O(10-2) 4

ENDG 407 - Computational Numerical Methods 1.14

 Introduction (16)

u Binary floating point representation (BFPR)

o Form: ± 1. d1d2 d3d4 …dn ⋅ 2 b1b2 ...bm
o The digits di in the mantissa and bi in the exponent are binary digits
o n+1 is the number of significant digits (where dn ≠ 0?)
o A decimal number is converted to binary floated point representation in two steps:
² 1st step: The number is normalized by dividing it by the largest power of 2 that is smaller than the
number itself. Its new (decimal) form is ± q ∙ 2p , where 1 ≤ q < 2 and p is an integer
² 2nd step: Then p and the mantissa part of q are converted from decimal to binary
o Example 1.5:
Dec. number normalization binary notat. # of sign. digits
50 50/25 ∙ 25 = 1.5625 ∙ 25 1.1001 ∙ 2101 5
(50/32 ∙ 25 )
0.3125 0.3125/2-2 ∙ 2-2 = 1.25 ∙ 2-2 1.01 ∙ 2-10 3
(0.3125/0.25 ∙ 2-2 )
1344 1344/210 ∙ 210 = 1.3125 ∙ 210 1.0101∙ 21010 5
(1344/1024 ∙ 210 )

ENDG 407 - Computational Numerical Methods 1.15

 Introduction (17)

q Storing numbers in BFPR in single or double precision

u IEEE-754 standard of storing numbers in a computer
o Single precision (SP) and double precision (DP) numbers are stored in strings of 32 bits
(or 4 bytes) and 64 bits (or 8 bytes), respectively, as follows:
² the leading 1 in front of the decimal point is not stored
² 1st bit in SP and in DP is for storing the sign: 0 for + and 1 for -
² next 8 bits in SP and 11 bits in DP are for the exponent plus bias of 127 in SP and 1023 in DP
² last 23 bits in SP and 52 bits in DP are for storing the mantissa

DP (64 bits)
1 8 23 SP (32 bits)

ENDG 407 - Computational Numerical Methods 1.16

 Introduction (18)

o The bias is introduced in order to avoid using one of the bits for the sign of the exponent
² The largest exponent that can be stored with 11 bits is 211-1 = 2047. The bias that is used is 1023:
² the largest positive actual exponent is 1024, resulting in exp.+bias = 2047 = [11111111111]2
² the smallest negative actual exponent is -1023, resulting in exp.+bias = 0 = [00000000000]2
o Example 1.6: How is the number 22.5 stored in BFPR in double precision?
normalization: 22.5/24 ∙ 24 = 22.5/16 ∙ 24 = 1.40625 ∙ 24
binary 1-bit sign: 0
binary 11-bit exponent + bias: 4+1023 = 1027 = [10000000011]2
binary 52-bit mantissa: 0.40625 = [0.011010000…0000] 2

ENDG 407 - Computational Numerical Methods 1.17

 Introduction (19)

u Overflow and underflow errors

o Due to the limited number of bits available to store numbers in a computer, the range
of numbers is also limited. In double precision
² the smallest mantissa, and smallest mantissa difference between numbers, is 2-52 ≈ 2.2 ∙ 10-16
â for numbers of O(1), this constant is called the machine epsilon: eps = 2.2204460492500313 ∙ 10-16
² the largest number is 21024 ≈ 1.8 ∙ 10308
² the smallest number is 2-1023 ≈ 1.1 ∙ 10-308
o Numbers outside this range cause overflow or underflow errors as shown in the graph
o Errors are also caused when storing numbers that have non-finite BFPR

ENDG 407 - Computational Numerical Methods 1.18

 Introduction (20)

v Errors in numerical solutions

q Absolute and relative errors
For a number x and its numerical approximation x
u the true error is E x = x − x and the true absolute eror is E xA = x − x
u the true relative error is E xR = (x − x ) / x
u the estimated relative error is E xER = ( x n − x n−1 ) / x n
when the solution x of a numerical problem has been estimated by n iterations
q Round-off errors and loss of significance/precision
u Due to the limited bits available in a computer to store the mantissa, 'longer'
numbers need to be shortened by rounding or chopping them in order to fit
resulting in a round-off error ERO
u Round-off errors are occurring when subtracting two nearly identical numbers or
when two vastly different numbers are involved in a calculation, and are the
reason for the resulting loss of significance/precision

ENDG 407 - Computational Numerical Methods 1.19

 Introduction (21)

u Example 1.7: In a computer having a 5-digit mantissa (a) what are the absolute and
relative errors of representing x = 0.003721478693, y = 0.003730230572, x-y?
(b) How many significant digits will be lost when storing the value r = x/y?
(c) How can the loss of significance be avoided/minimized?
x = 0.00372, E xA = x − x = 0.003721478693-0.00372 = 0.000001478693
y = 0.00373, E yA = y − y = 0.003730230572-0.00373 = 0.000000230752
x − y = −0.000008751879, x − y = −0.00001, A
E x−y = (x − y) − ( x − y ) = 0.000001248121
E xA 0.000001478693 E yA A
E x−y
Relative errors: = ≈ 0.040%, = 0.006%, ≈ 14.261%
x 0.003721478693 y x−y

o Notice that the relative error of the difference is very significant

x x
r= = 0.997653796774469 ⇒ r = 0.99765, = 0.997319034852547 ⇒ r = 0.99732
y y
o Thus 2 significant digits are lost in the machine representation of r = x/y
o The loss of significance can be minimized by, e.g., pre-multiplying x and y by an
appropriate constant and then dividing the resulting (x-y) by the same constant

ENDG 407 - Computational Numerical Methods 1.20

 Introduction (22)

q Truncation error E TR = x − x̂
u The difference between the true solution x and the solution x̂ calculated by the
specific approximate mathematical algorithm employed to solve the problem
u Example 1.8: Computers use a series expansion to compute sine values. The
magnitude of ETR depends on the number of terms that are retained in the
expansion. E.g., for x = p/6, y = sin(x) can be approximated as follows:
x 3 x 5 x 7 x 9 x11
y = sin( x) = x - + - + - + ...
3! 5! 7! 9! 11!
all erms: y = sin(p / 6) = 0.5000000 (exact solution)
1st term: yˆ1 = p / 6 » 0.5235988 y = y - y1 = -0.0235988
E TR ˆ
(p / 6)3
2 terms: yˆ 2 = p / 6 - » 0.4996742 y = y - y2 = -0.0003258
E TR ˆ
3!
q Total Error E TE = E RO + E TR
u The total (round-off plus truncation) error of the numerical solution
o We can again define a true, an absolute and a relative total error
ENDG 407 - Computational Numerical Methods 1.21
 Introduction (23)

q Rules for significant digits

u A significant digit in a number is the least significant figure that is considered
reliable as a result of measurements or calculations
o The number of significant figures in a result indicates the number of digits that can be
used with confidence
o A common mistake is to show too many figures in the answer, which gives readers
the idea that the answer is more accurate than it really is
u Rule 1: Same significant digit must be used in a coherent set of engineering
operations and data processing
o When different significant digits are involved in an operation, a truncation and
rounding operation must be carried out in processing the result
u Rule 2: The rule of rounding in significant digits processing is to "truncate 4 and
round up 5". E.g.,
o 6.2548 → 6.3 (for 1 significant decimal digit)
o 6.2548 → 6.25 (for 2 significant decimal digits)
o 6.2548 → 6.255 (for 3 significant decimal digits)
ENDG 407 - Computational Numerical Methods 1.22
 Introduction (24)

u Rule 3: The result of addition and subtraction should show significant digits only
as far to the right as is seen in the least precise number in the calculation
o The insignificant digits following the least significant figure should be rounded to the
least significant figure. E.g.,
o 3.51 + 2.246 + 0.0192 = 5.7752 → 5.78
o 3.510 + 2.246 + 0.0192 = 5.7752 → 5.775
o 1725.463 - 189.2 = 1536.263 → 1536.3
o 23578.3 + 0.1892 = 23578.4892 → 23578.5
u Rule 4: The result of multiplication or division should show significant digits
only as far to the right as seen in the number with the fewest significant digits
o The insignificant digits following the least significant figure should be rounded to the
least significant figure. E.g.,
o 2.43 × 17.675 = 42.950250 → 42.95
o 75.22 ÷ 25.1 = 2.9968127 → 3.0
o 75.220 ÷ 25.100 = 2.9968127 → 2.997

ENDG 407 - Computational Numerical Methods 1.23

 Mathematical Background (1)

v Review of basic calculus concepts

q Limit of a function: lim f (x) = L
x→a
u It means that for each number e > 0,
∃( there exists) a number d > 0 such that
|f(x) - L| < e if 0 < |x - a| < d
u Example 1.9:
o For f(x) = x2, it is true that lim f (x) = 4 because it can be shown that for an e > 0
x→2

x 2 - 4 | < e if | x - 2 | < e / (5 + e )
|
f ( x) L a
 
d
o For f(x) = |x| / x, lim f (x) = L is not true, i.e., the limit does not exit, for an e > 0.
x→0
Take e = 1. For the limit to exist, it must be | |x|/x - L | < 1 if |x - 0| < d, or there must
be an L that satisfies simultaneously |-1-L| < 1 and |1-L| < 1. No such L exists!
q Continuity of a function
u f(x) is continuous at x = a if lim f (x) = f (a)
x→a

ENDG 407 - Computational Numerical Methods 1.24

 Mathematical Background (2)

u If f is continuous in a region X, we write f ∈ C(X). E.g.,

o f ∈ C(R) or f ∈ C(-∞,∞), if continuous at all x ∈ R (continuous everywhere)
o f ∈ C(a,b), if continuous in a < x < b
o f ∈ C[a,b], if continuous in a ≤ x ≤ b
u Intermediate value theorem
o If f ∈ C[a,b], then for a number M ∈ [f(a), f(b)]
∃ c ∈ [a,b] such that f(c) = M
u Example 1.10:
o f(x) = x2 is continuous at x = 2 because lim f (x) = f (2) = 4
x→2
o f(x) = |x| / x is not continuous at x = 0
because lim | x | does not exist
x→0 x
q Derivative of a function
df ( x) f ( x) - f (a )
f '(a ) = = lim
dx x =a x ®a x-a
tana = f '(a)
ENDG 407 - Computational Numerical Methods 1.25
 Mathematical Background (3)

u Usefulness of derivative of function

o From 1st equation: the derivative indicates the rate of change of f(x) w.r.t. x
o From 2nd equation: the derivative is the slope of the tangent of the function
² very useful in finding minima or maxima of functions, which occur where f '(x) = 0

u If f is differentiable in a region X, we write f ∈ D(X). E.g.,

o f ∈ D(a,b), if differentiable in a < x < b
o f ∈ D[a,b], if differentiable in a ≤ x ≤ b
o f ∈ Dk(R), if differentiable k times at all x ∈ R a
u Mean value theorem for derivatives
o If f ∈ C[a,b] and f ∈ D(a,b), then ∃c ∈ (a,b) such that

o Useful in finding bounds for the order of magnitude of errors in numerical methods
df (x) f (b) − f (a)
f '(c) = =
dx x=c b−a

ENDG 407 - Computational Numerical Methods 1.26

 Mathematical Background (4)

q Integral of a function
u Indefinite integral or antiderivative: The inverse of the derivative
dF(x)
f (x) = F '(x) = ⇔ F(x) = ∫ f (x) dx = ∫ F '(x) dx
dx
u Definite integral (it is a number I) defined on a closed interval [a,b]:
b N
I= ∫ f (x) dx = lim
Δ xi →0
∑ f (c )Δ x
i i
a i=1

o I is the area under the f(x) curve from a to b

o Useful in developing methods for the numerical
integration of functions
u Fundamental theorem of calculus
o If f ∈ C[a,b] and F is the antiderivative of f in [a,b], then
b

I= ∫ f (x) dx = F(b) − F(a)

a

ENDG 407 - Computational Numerical Methods 1.27

 Mathematical Background (5)

u Mean value theorem for integrals

o Looking at the figure we see that
f(a)(b – a) < I < f(b)(b – a)
and that, if f ∈ C[a,b], ∃a c such that
b

I= ∫ f (x) dx = f (c)(b − a)
a
u f(c) above is the average value ⟨ f ⟩ of the
function in the interval [a,b], i.e.,
b
1
f = ∫ f (x) dx
(b − a) a
u Second fundamental theorem of calculus
o If f ∈ C[a,b] and c ∈ (a,b), then ∀ x ∈ [a,b]
x
d
( ∫ f (ξ ) d ξ ) = f (x)
dx c
o Useful in evaluating the derivative of a definite integral
ENDG 407 - Computational Numerical Methods 1.28
 Mathematical Background (6)

v Derivatives of functions of two independent variables

q Partial derivative of z = f(x,y)
∂z ∂f (x, y) f (x + Δ x, y) − f (x, y)
= fx = = lim
∂x ∂x Δ x→0 Δx
u 2nd and higher order partial derivatives are calculated by successive application of
first derivatives. E.g.,
∂2 z ∂ " ∂f (x, y) % ∂
= fyx = $ '= fx
∂y∂x ∂y # ∂x & ∂y
q Chain rule (simplest case)
u Total differential
∂f ∂f
dz = df = dx + dy = fx dx + fy dy
∂x ∂y
u When both x and y are functions of another variable t, then
∂z ∂f ∂f dx ∂f dy
= = +
∂t ∂t ∂x dt ∂y dt
ENDG 407 - Computational Numerical Methods 1.29
 Mathematical Background (7)

q The Jacobian matrix J

u For a set of n functions f1, f2, …, fn, each " ∂f1 ∂f1 ∂f1 %
$  '
depending on the same m parameters $ ∂x1 ∂x2 ∂xm '
x1, x2, …, xm, the Jacobin matrix is the $ ∂f2 ∂f2 ∂f2 '
$  '
n × m matrix of all partial derivatives J = [J ] = $ ∂x1 ∂x2 ∂xm '
$     '
o Very useful in the solution of $ '
$ ∂fn ∂fn ∂fn '
systems of nonlinear equations 
$ ∂x1 ∂x2 ∂xm '
# &
v Taylor series expansion
q Taylor's theorem in one variable
u If f ∈ Cn[a,b] and f ∈ Dn+1(a,b), then ∀ x, xo ∈ [a,b] ∃a x ∈ [x, xo] such that

df (x − xo )2 d 2 f (x − xo )3 d 3 f
f (x) = f (xo ) + (x − xo ) + +
dx xo 2! dx 2 xo
3! dx 3 xo

(x − xo )n d n f (x − xo )n+1 d n+1 f
+…+ +
n! dx n xo
(n + 1)! dx n+1 ξ

ENDG 407 - Computational Numerical Methods 1.30

 Mathematical Background (8)
n
(x − xo )k (k ) (x − xo )n+1 (n+1)
f (x) = ∑ f + Rn (x), Rn (x) = f
k=0 k! xo (n + 1)! ξ

o xo is called the point of expansion

u Rn is the remainder term and cannot be computed exactly since x is unknown
o Useful because it can give an estimate of the upper bound of the error we commit
when we truncate a series at a particular n
o Also useful in estimating the rate of convergence of Taylor series
u Taylor expansions are very useful when we need to approximate, with a particular
accuracy, the value of the function near a point xo where we know its value f(xo)
o The approximation is better the higher the n and the closer x is to xo
u Truncation error by Taylor series
o Since in computers many basic functions are estimated by truncated Taylor series, i.e.,
by the summation term in the equation above, the remainder term is the truncation
error, i.e., the difference between the true value of the function and the estimated one
o The exact value of this error can be obtained only when we know in advance the exact
value of the function we are trying to approximate; see Example 2-3 in textbook
ENDG 407 - Computational Numerical Methods 1.31
 Mathematical Background (9)

u Example 1.11: (a) Expand lnx into a Taylor series around xo = 1 in the interval
[1,2]. (b) What is the upper bound of the truncation error of computing ln2 with
an accuracy of 1 part in 108? How many terms are needed to obtain this accuracy?
How many terms would we need to obtain ln1.5 with the same accuracy?
(a) Recall that d(lnx)/dx = 1/x, therefore
1 1 1 1
ln(x) = (x − 1) − (x − 1)2 + (x − 1)3 − (x − 1)4 +…+ (−1)n−1 (x − 1)n + Rn (x)
2 3 4 n
−(n+1)
n ξ
Rn (x) = (−1) (x − 1)n+1
n +1
(b) Since x > 1, x–(n+1) < 1 and therefore we can obtain the upper bound for Rn
ξ −(n+1) 1
Rn (x) = (x − 1)n+1 < (x − 1)n+1
n +1 n +1
(c) We need to have |Rn(2)| ≤ 1/108, which requires 100 million terms !!
1 1
Rn (2) < (2 − 1)n+1 ≤ 8 ⇒ n + 1 ≥ 10 8
n +1 10
(d) We need to have |Rn(1.5)| ≤ 1/108, which requires only 22 terms !!
1 1 n +1
Rn (2) < (1.5 − 1)n+1 ≤ 8 ⇒ n+1
≥ 10 8 ⇒ n ≥ 22
n +1 10 0.5
ENDG 407 - Computational Numerical Methods 1.32
 Mathematical Background (10)

u Rolle's theorem
o It is easy to see that for n = 0, the Taylor series equation becomes
df f (x) − f (xo )
f (x) = f (xo ) + (x − xo ) ⇒ f '(ξ ) =
dx ξ x − xo
which for x = b, xo = a and x = c is the mean value theorem for derivatives (see p. 1.28)
o Rolle's theorem is a special case of the mean value theorem when f(a) = f(b) = 0:
If f ∈ C[a,b], f ' ∈ D(a,b) and f(a) = f(b) = 0,
then∃a c ∈ (a,b) such that f '(c) = 0
o Interpretation: When f(a) = f(b) = 0, f(x) will have
be at least one maximum or minimum in (a,b)
² Useful to find the min/max values of a function

q Alternative form of Taylor's theorem

u If we use h = x – xo, then an alternative form of the Taylor series can be obtained
which is very useful in practical applications:
n
h k (k ) h n+1 (n+1)
f (x + h) = ∑ f (x) + Rn , Rn = f (ξ )
k=0 k! (n +1)!
ENDG 407 - Computational Numerical Methods 1.33
 Mathematical Background (11)

q Taylor's theorem in two or more variables

u The Taylor expansion of a function of two or more variables can be done in the
same way as for one variable but it involves partial derivatives
u For two variables, the expansion of f(x,y) around a point xo,yo can be found in the
textbook: eq. (2.78), p. 49
u The alternative form in two variables (where h, i are small increments to x, y) is
n
1 ∂ ∂ 1 ∂ ∂
f (x + h, y + i) = ∑ (h + i )k f (x, y) + Rn , Rn = (h + i )n+1 f (x + θ h, y + θ i)
k=0 k! ∂x ∂y (n + 1)! ∂x ∂y

∂ ∂
where (h + i )0 f (x, y) = f
∂x ∂y
∂ ∂ ∂f ∂f
(h + i )1 f (x, y) = h + i
∂x ∂y ∂x ∂y
2
∂ ∂ 2 2 ∂ f ∂2 f 2
2 ∂ f
(h + i ) f (x, y) = h + 2hi +i
∂x ∂y ∂x 2 ∂x∂y ∂y 2

ENDG 407 - Computational Numerical Methods 1.34
 Review Engineering Problems (1)

v Efficient programming of Taylor series and computation of

estimated relative errors
q Example 1.12: Solve textbook problem 1.42

ENDG 407 - Computational Numerical Methods 1.35

 Review Engineering Problems (2)

q Matlab code and results

Results

u Results agree to 8th decimal

ENDG 407 - Computational Numerical Methods 1.36

 Review Engineering Problems (3)

v Coordinate transformations and the Jacobian matrix

q Example 1.13: Solve textbook problem 2.21 and also show how we can get
Catresian coordinates from polar coordinates

u Solution

u y/x = rsinq / rcosq = tanq ⇒ q = tan-1(y/x), x2 + y2 = r2 ⇒ r = (x2 + y2)1/2

ENDG 407 - Computational Numerical Methods 1.37

 Review Engineering Problems (4)

v Propagation of uncertainty using the total differential

q Example 1.14: Solve textbook example 2.32

q Solution
u (a)

u (b)

ENDG 407 - Computational Numerical Methods 1.38