Chapter 1:Part II Contents

Numerical Errors
• Error Definitions
• Truncation Errors and Taylor Series
• Round-off Errors
• Total Numerical Error
– Worksheet 1

1
 Content Objectives
• Learning how to quantity error;
• Learning how error estimates can be used to decide when to
terminate an iterative calculation;
• Recognizing that truncation errors occur when exact
mathematical formulations are represented by
approximations;
• Knowing how to use the Taylor series to estimate
truncation errors;
• Understanding how to write forward, backward and
centered finite-difference approximations of first and
second derivatives;
• Understanding how round-off errors occur because digital
computers have a limited ability to represent numbers; 2
 Introduction
• Engineers constantly find themselves having to
accomplish objectives based on uncertain information.
• E.g., despite the fact that the model developed from
Newton's 2nd law is a good approximation, it would
never in practice exactly predict the jumper's fall.
• Factors like wind & slight variations in air resistance
would result in deviations from the prediction.
• If these deviations are high or low, then we might need
to develop a new model.
• If they are randomly distributed and tightly grouped
around the prediction, then the deviations might be
considered negligible and the model deemed adequate.
3
 Introduction
• Numerical approximations also introduce similar
discrepancies into the analysis.
• The errors associated with both calculations and
measurements can be characterized with regard to their
accuracy and precision.
– Accuracy refers to how closely a computed or measured
value agrees with the true value.
– Precision refers to how closely individually
computed/measured values agree with each other.
• These concepts can be illustrated graphically using an
analogy from target practice.
– Bullet holes on each target represent the predictions of a
numerical technique, whereas
4
– Bull’s-eye represents the truth.
 Introduction
• Fig. An example from marksmanship illustrating the concepts of
accuracy and precision. (a) Inaccurate and Imprecise; (b) Accurate and
Imprecise; (c) Inaccurate and Precise; (d) Accurate and Precise.

5
 Introduction
• Inaccuracy (also called bias) is defined as systematic
deviation from the truth.
• Imprecision (also called uncertainty), on the other hand,
refers to the magnitude of the scatter.
• Numerical methods should be sufficiently accurate or
unbiased to meet the requirements of a particular
engineering problem.
• They also should be precise enough for adequate
engineering design.
• In this course, we will use the collective term error to
represent both the inaccuracy and the imprecision of our
predictions.
• We can now discuss the factors that contribute to the error
6
of numerical computations.
 Error Definitions
• Numerical errors arise from the use of approximations
to represent exact mathematical operations and
quantities.
• The relationship between the exact, or true, result and
the approximation can be formulated as:
…… (1.1)
• By rearranging Eq. (1), the numerical error is equal to
the discrepancy between the truth and the
approximation as in:
……. (1.2)
• In Eqn. (2), Et is used to designate the exact value of
the error. The subscript “t” designates the "true" error.
7
 Error Definitions
• A shortcoming of this definition is that it takes no
account of the order of magnitude of the value.
• For example, an error of a centimeter is much more
significant if we are measuring a rivet than a bridge.
• One way to account for the magnitudes of the
quantities is to normalize the error to the true value,

• The relative error can also be multiplied by 100% as:

……….. (1.3)
• where εt designates the true percent relative error.

8
 Error Definitions
• For example, suppose that you have the task of
measuring the lengths of a bridge and a rivet and come
up with 9999 cm and 9 cm, respectively.
• If the true values are 10,000 and 10 cm, respectively,
the error in both cases is 1cm.
• However, their percent relative errors can be computed
using Eq. (3) as 0.01% and 10%, respectively.
• Thus, although both measurements have an absolute
error of 1cm, the relative error for the rivet is much
greater.
• We would probably conclude that we have done an
adequate job of measuring the bridge, whereas our
9
estimate for the rivet leaves something to be desired.
 Error Definitions
• For Eqs. (2) and (3), E and ε are subscripted with a “t”
to signify that the error is based on the true value.
• For the example of the rivet and the bridge, we were
provided with this value.
• However, in actual situations such information is rarely
available.
• For numerical methods, the true value will only be
known when we deal with functions that can be solved
analytically.
• In real-world applications, we will obviously not know
the true answer a priori.
10
 Error Definitions
• For these situations, an alternative is to normalize the
error using the best available estimate of the true value,
that is, to the approximation itself, as in:
……… (1.4)

• where the subscript “a” signifies that the error is

normalized to an approximate value.
• For real-world applications, Eq. (2) cannot be used to
calculate the error term in the numerator of Eq. (4).
• One of the challenges of numerical methods is to
determine error estimates in the absence of knowledge
regarding the true value. 11
 Error Definitions
• Certain numerical methods use iteration to compute
answers. In such cases, a present approximation is
made on the basis of a previous approximation.
• This process is performed repeatedly, or iteratively, to
successively compute better and better approximations.
• For such cases, the error is often estimated as the
difference between the previous and present
approximations.
• Thus, percent relative error is determined according to:

……. (1.5)
12
 Error Definitions
• The signs of Eqs. (2) to (5) may be either positive or
negative. Often, when performing computations, we
may not be concerned with the sign of the error but are
interested in whether the absolute value of the percent
relative error is lower than pre-specified tolerance εs.
• Therefore, it is often useful to employ the absolute
value of Eq. (2.5).
• For such cases, the computation is repeated until:
…………….. (1.6)
• This relationship is referred to as a stop criterion.
• If it is satisfied, our result is assumed to be within the
pre-specified acceptable level εs. 13
 Error Definitions
• It is also convenient to relate these errors to the number
of significant figures in the approximation.
• It can be shown (Scarborough, 1966) that if the
following criterion is met, we can be assured that the
result is correct to at least n significant figures:
…………. (1.7)
Example 1.3 : Error Estimates for Iterative Methods
• In mathematics, functions can often be represented by
infinite series. For example, the exponential function
can be computed using:
……… (E1)
14
 Example 1.3
• As more terms are added, the approximation becomes a
better and better estimate of the true value of ex.
• Equation (E1) is called a Maclaurin series expansion.
• Starting with the simplest version, ex = 1, add terms
one at a time to estimate e0.5.
• After each new term is added, compute the true and
approximate percent relative errors with Eqs. (3) and
Eqs. (5), respectively.
• Note that the true value is e0.5 = 1.648721 . . . .
• Add terms until the absolute value of the approximate
error estimate εa falls below a pre-specified error
criterion εs conforming to three significant figures. 15
 Example 1.3 Solution
• First determine the error criterion that ensures a result is
correct to at least three significant figures:

• Thus, we will add terms to the series until εa falls below

this level.
• The first estimate is equal to 1.
• The second estimate is then generated by adding the
second term, as in
ex = 1 + x; for x = 0.5, → e0.5 = 1 + 0.5 = 1.5
• This represents a true percent relative error of

16
 Example 1.3 Solution
• The approximate estimate of the error is,

• Because εa is not less than the required value of εs, we would

continue the computation by adding another term, x2∕2!, and
repeating the error calculation.
• The process is continued until εa < εs.
• The entire computation can be summarized as,

• Thus, after six terms are included, the approximate error

17
falls below εs = 0.05% and the computation is terminated.
 Truncation Errors and Taylor Series
• Truncation errors result from using an approximation in
place of an exact mathematical procedure.
• For example, we approximated the derivative of velocity of
a bungee jumper by a finite-difference equation of the
form:
……….. (1.8)
• A truncation error was introduced into the numerical
solution because the finite-difference equation only
approximates the true value of the derivative.
• We use a mathematical formulation that is used widely in
numerical methods to express functions in an approximate
fashion – the Taylor series.
18
 Taylor Series
• The Taylor series provides a means to predict a function
value at one point in terms of the function value and its
derivatives at another point.
• In particular, the theorem states that any smooth function
can be approximated as a polynomial.
• A useful way to understand the Taylor series is to build it
term by term. The first term in the series is:
………… (1.9)
• This zero-order approximation, indicates that the value of
the function f at the new point is the same as the value at
the old point. This result makes intuitive sense because if xi
and xi+1 are close to each other, it is likely that the new
value is probably similar to the old value.
19
 Taylor Series
• Equation (2.9) provides a perfect estimate if the function
being approximated is a constant.
• However, if the function changes at all over the interval,
additional terms of the Taylor series are required to provide
a better estimate.
• For example, the first-order approximation is developed by
adding another term to yield:
….. (1.10)
• The additional first-order term consists of a slope f‘(xi)
multiplied by the distance between xi and xi+1.
• Thus, the expression is now in the form of a straight line
and is capable of predicting an increase or decrease of the
function between xi and xi+1. 20
 Truncation Errors and Taylor Series
• Although Eq. (2.10) can predict a change, it is only
exact for a straight-line, or linear, trend.
• A second-order term is added to the series to capture
some of the curvature that the function might exhibit:
… (1.11)

• In a similar manner, additional terms can be included to

develop the complete Taylor series expansion:

(1.12)

21
 Truncation Errors and Taylor Series
• A remainder term is included to account for all terms
from n + 1 to infinity:
……… (1.13)

• The subscript “n” implies that this is the remainder for

the nth-order approximation and ξ is a value of x that
lies somewhere between xi and xi+1.
• The introduction of the ξ is used to recognize that there
is such a value that provides an exact determination of
the error.

22
 Truncation Errors and Taylor Series
• Taylor series can be simplified by defining a step size
h = xi+1 − xi and Eq. (2.12) can be expressed as:

….. (1.14)
• where the remainder term is now,
………….. (1.15)

• In general, the nth-order Taylor series expansion will be

exact for an nth-order polynomial.
• For other differentiable and continuous functions, such
as exponentials and sinusoids, a finite number of terms
will not yield an exact estimate. 23
 Truncation Errors and Taylor Series
• In most cases, the inclusion of only a few terms will
result in an approximation that is close enough to the
true value for practical purposes.
• The assessment of how many terms are required to get
close enough, is based on the remainder term of the
expansion (Eq. 2.15).
• We do have control over the term h in the equation, i.e,
we can choose how far away from x we want to
evaluate f(x), and we can control the number of terms
we include in the expansion. Consequently, Eq. (2.15)
is often expressed as
• where the nomenclature O(hn+1) means that the
truncation error is of the order of hn+1. 24
 Truncation Errors and Taylor Series
• If the error is O(h), halving the step size will halve the
error; if the error is O(h2), halving the step size will
quarter the error.
• In many cases, if h is sufficiently small, the first- and
other lower-order terms usually account for a
disproportionately high percent of the error.

25
 Example 1.4
Example 1.4: Taylor Series Approximation of a
Polynomial

• Use zero- through fourth-order Taylor series expansions

to approximate the function
f (x) = − 0.1x4 − 0.15x3 − 0.5x2 − 0.25x + 1.2
• from xi = 0 with h = 1. That is, predict the function’s
value at xi+1 = 1.

26
 Example 1.4 Solution
• Because we are dealing with a known function, we can
compute values for f(x) between 0 and 1.
• The results (see figure on next slide) indicate that the
function starts at f(0) = 1.2 and then curves downward to
f(1) = 0.2.
• Thus, the true value that we are trying to predict is 0.2.
• The Taylor series approximation with n = 0 is:
f(xi+1) ≅ 1.2
• Thus, the zero-order approximation is a constant.
• Using this formulation results in a truncation error of
Et = 0.2 - 1.2 = -1.0, at x = 1.
27
 Example 1.4 Solution
• The approximation of f(x) at x = 1 by zero-order,
first-order, and second-order Taylor series expansions.

28
 Example 1.4 Solution
• For n = 1, the first derivative must be determined and
evaluated at x = 0:
f ′(0) = -0.4(0.0)3 - 0.45(0.0)2 - 1.0(0.0) - 0.25 = -0.25
• Therefore, the first-order approximation is
f(xi+1) ≅ 1.2 - 0.25h
• which can be used to compute f(1) = 0.95.
• Consequently, the approximation begins to capture the
downward trajectory of the function in the form of a
sloping straight line.
• This results in a reduction of the truncation error to
Et = 0.2 - 0.95 = -0.75
29
 Example 1.4 Solution
• For n = 2, the second derivative is evaluated at x = 0:
f ″(0) = -1.2(0.0)2 - 0.9(0.0) - 1.0 = -1.0
• Therefore,
f(xi+1) ≅ 1.2 - 0.25h - 0.5h2
• and substituting h = 1, f(1) = 0.45.
• The inclusion of the second derivative now adds
some downward curvature resulting in an improved
estimate, as seen in the figure.
• The truncation error is reduced further to:
0.2 - 0.45 = -0.25.

30
 Example 1.4 Solution
• Additional terms would improve the approximation even
more.
• In fact, the inclusion of the third and the fourth derivatives
results in exactly the same equation we started with:
f(x) = 1.2 - 0.25h - 0.5h2 - 0.15h3 - 0.1h4
• where the remainder term is

• the fifth derivative of a fourth-order polynomial is zero.

• Consequently, the Taylor series expansion to the fourth
derivative yields an exact estimate at xi+1 = 1:
f(1) = 1.2 - 0.25(1) - 0.5(1)2 - 0.15(1)3 - 0.1(1)4 = 0.231
 Example 1.5
Example 1.5: Taylor Series Expansion to Approximate
a Function with Infinite Number of Derivatives

• Use Taylor series expansions with n = 0 to 6 to

approximate f (x) = cos x at xi+1 = π/3 on the basis of the
value of f (x) and its derivatives at xi = π/4.
• Note that this means that h = π/3 − π/4 = π/12.

32
 Example 1.5 Solution
• Our knowledge of the true function means that we can
determine the correct value, f(π∕3) = 0.5.
• The zero-order approximation is:

• which represents a percent relative error of

• For the first-order approximation, we add the first

derivative term where f′(x) = -sin x:

• which has εt = -4.40%.

33
 Example 1.5 Solution
• For the second-order approximation, we add the second
derivative term where f ″(x) = - cos x:

• with εt = 0.449%. Thus, the inclusion of additional terms

results in an improved estimate.
• The process can be continued and the results tabulated as in
Table (see next slide).
• Notice that the derivatives never go to zero, as was the case
with the polynomial in Example 1.4.
• Therefore, each additional term results in some improvement
in the estimate.
34
 Example 1.5 Solution
• Taylor series approximation of f(x) = cos x at xi+1 = π∕3 using a
base point of π∕4. Values are shown for various orders (n) of
approximation.

• by the time we have added the third-order term, the error is

reduced to 2.62 × 10-2 percent, which means that we have
attained 99.9738% of the true value.
• Consequently, although the addition of more terms will reduce the
error further, the improvement becomes negligible. 35
 The Remainder for the Taylor Series Expansion
• Suppose that we truncated the Taylor series expansion, Eq. (2.14)
after the zero-order term to yield:

• The remainder, or error, of this prediction, consists of the infinite

series of terms that were truncated:

• It is obviously inconvenient to deal with the remainder in this

infinite series format. One simplification might be to truncate the
remainder itself, as in:
………………… (1.16)
• Although, lower-order derivatives usually account for a greater
share of the remainder than the higher-order terms, this result is
still inexact because of the neglected second- and higher-order
terms.
36
 The Remainder for the Taylor Series Expansion
• An alternative simplification that transforms the approximation
into an equivalent equation is based on a graphical insight.
• As shown in the figure (next slide), the derivative mean-value
theorem states that “if a function f (x) and its first derivative are
continuous over an interval from xi to xi+1, then there exists at
least one point on the function that has a slope, designated by
f’(ξ), that is parallel to the line joining f (xi) and f (xi+1)”.
• The parameter ξ marks the x value where this slope occurs.
• By invoking this theorem it is simple to realize that, as illustrated
in the figure, the slope f‘ (ξ) is equal to the rise R0 divided by the
run h, or
• which can be rearranged to give
……………. (1.17)

37
 The Remainder for the Taylor Series Expansion
• Graphical depiction of the derivative – mean value theorem

• Thus, we have derived the zero-order version of Eq. (2.15).

• The higher-order versions are merely a logical extension of the
reasoning used to derive Eq. (2.17). The first-order version is:
….………………. (1.18)
• For this case, the value of ξ conforms to the x value
38
corresponding to the 2nd derivative that makes Eq. (2.18) exact.
Using the Taylor Series to Estimate Truncation Errors
• Recall that the objective of both Examples 1.1 and 1.2 was to
predict velocity as a function of time, i.e., we were interested in
determining v(t).
• Using Eqn. (1.12), v(t) can be expanded in a Taylor series:
…. (1.19)
• Now let us truncate the series after the first derivative term:
……. (1.20)
• Eqn. (1.20) can be solved for

…...…… (1.21)

• The first part of Eqn. (1.21), i.e., the first – order approximation,
is exactly the same relationship that was used to approximate the
derivative in Example 1.2. 39
Using the Taylor Series to Estimate Truncation Errors
• However, because of the Taylor series approach, we have now
obtained an estimate of the truncation error associated with this
approximation of the derivative.
• Using Eqs. (2.13) and (2.21) yields:
• or …. (1.22)

• Thus, the estimate of the derivative [Eq. (1.11) or the first part of
Eq. (1.21)] has a truncation error of order ti+1 − ti.
• In other words, the error of our derivative approximation should
be proportional to the step size.
• Consequently, if we halve the step size, we would expect to halve
the error of the derivative.

40
 Round-off Errors
• Round-off errors arise because digital computers cannot represent
some quantities exactly.
• They can lead to a calculation going unstable and yielding
erroneous results.
• Such calculations are said to be ill-conditioned.
• Worse still, they can lead to subtler discrepancies that are difficult
to detect.
• There are two major facts of round-off errors involved in
numerical calculations:
– Digital computers have size and precision limits on their
ability to represent numbers.
– Certain numerical manipulations are highly sensitive to round-
off errors. This can result from both mathematical
considerations as well as from the way in which computers
41
perform arithmetic operations.
 Total Numerical Error
• The total numerical error is the summation of the truncation and
round-off errors.
• In general, the only way to minimize round-off errors is to
increase the number of significant figures of the computer.
• Further, round-off error may increase due to subtractive
cancellation or due to an increase in the number of computations
in an analysis.
• In contrast, truncation error can be reduced by decreasing the step
size.
• Because a decrease in step size can lead to subtractive
cancellation or to an increase in computations, the truncation
errors are decreased as the round-off errors are increased.

42
 Total Numerical Error
• Figure represents a graphical depiction of the trade-off between
round-off and truncation error that sometimes comes into play in
the course of a numerical method.

• The point of diminishing returns is shown, where round-off error

begins to negate the benefits of step-size reduction.
• Therefore, we are faced by the following dilemma: The strategy
for decreasing one component of the total error leads to an
increase of the other component. 43
 Total Numerical Error
• In a computation, we could conceivably decrease the step size to
minimize truncation errors only to discover that in doing so, the
round-off error begins to dominate the solution and the total error
grows! Thus, our remedy becomes our problem.
• One challenge that we face is to determine an appropriate step
size for a particular computation.
• We would like to choose a large step size to decrease the amount
of calculations and round-off errors without incurring the penalty
of a large truncation error.
• If the total error is as shown in the figure, the challenge is to
identify the point of diminishing returns where round-off error
begins to negate the benefits of step-size reduction.
• When using computer programs, such situations are relatively
uncommon because of its 15- to 16- digit precision.
44