CHAPTER 3: ACCURACY AND SPEED

3.1 VARIABLES AND RANGES

Python variables can hold numbers spanning a wide range of values, including huge
numbers, but they cannot hold arbitrarily large numbers. Complex numbers are
similar. Large numbers can be specified in scientific notation, using an “e” to denote
the exponent. For instance, $2e^9$ means $2\times10^9$ and $1.602e^{-19}$
means $1.602×10^{−19}$. Note that numbers specified in scientific notation are
always float. If the number is, mathematically speaking, an integer (like $2e^9$), the
computer will still treat it as a float.

In Python, single-precision floating-point numbers are represented by a 32-bit float.

The built-in float type in Python is double precision (64-bit), so if you want single
precision, you need to use libraries like numpy.

Here’s how you can represent single-precision numbers:

Using numpy for single precision To use single precision, you can use numpy's float32
data type:

In [36]: import numpy as np

# Single precision float
num = np.float32(10.0)
print(num)

10.0

This will explicitly store num as a 32-bit floating-point number. Checking the precision of
a number You can check the size (in bytes) of a floating-point number using itemsize in
numpy:

In [21]: # Double precision (64-bit)

double_num = np.float64(10.0)
print(double_num.itemsize) # Output: 8 bytes (64 bits)
# Single precision (32-bit)
single_num = np.float32(10.0)
print(single_num.itemsize) # Output: 4 bytes (32 bits)

8
4

If the value of a variable exceeds the largest floating-point number that can be
stored on the computer we say the variable has overflowed.

For instance, if a floating-point variable x holds a number close to the maximum allowed
value of $10^{308}$ and then we execute a statement like “$y =10*x$” it is likely that the
result will be larger than the maximum and the variable y will overflow (but not the
variable x, whose value is unchanged).
In [40]: x=1e308
y=10*x
print(x, y)

1e+308 inf

There is a smallest number (meaning smallest magnitude) that can be represented

by a floating-point variable. In Python, the smallest number is less than
$10^{−308}$. If you go

any smaller than this—if the calculation underflows—the computer will just set the
number to zero.

In [42]: x=1e-308
y=x/1e16
print(x, y)

1e-308 0.0

In [32]: #print(2**1000000)

In [36]: #import sys

#sys.set_int_max_str_digits(1000000) # Set the limit high enough to handle larg
## Now you can print the result without error
#result = 2**1000000
#print(result)

In [20]: import numpy as np

ii8 = np.iinfo(np.int8)
print(ii8.min, ii8.max)
ii16 = np.iinfo(np.int16)
print(ii16.min, ii16.max)
ii32 = np.iinfo(np.int32)
print(ii32.min, ii32.max)
ii64 = np.iinfo(np.int64)
print(ii64.min, ii64.max)

-128 127
-32768 32767
-2147483648 2147483647
-9223372036854775808 9223372036854775807

Python's built-in int type does not overflow or underflow due to its arbitrary
precision.
Fixed-size integers (e.g., in numpy) can overflow or underflow, and you can check
this using np.iinfo() to get the range of values allowed.

In [3]: import numpy as np

print(np.pi)

3.141592653589793

In [28]: import math

print(math.pi)

3.141592653589793
 Both of these methods give a reasonable precision of π for most purposes. If you need
even more precision, you can use specialized libraries like mpmath for arbitrary precisio

In [32]: from mpmath import mp

# Set precision (e.g., 50 decimal places)

mp.dps = 50
print(mp.pi)

3.1415926535897932384626433832795028841971693993751

In [1]: # Expected result should be 0, but due to floating-point inaccuracies, it's not.
result = 0.1 + 0.2 - 0.3
print(f"Result of 0.1 + 0.2 - 0.3: {result}") # This won't print exactly 0

Result of 0.1 + 0.2 - 0.3: 5.551115123125783e-17

Improving Accuracy: Using the decimal module. The decimal module provides higher
precision for decimal numbers.

In [5]: from decimal import Decimal

# Using Decimal for higher precision

result = Decimal('0.1') + Decimal('0.2') - Decimal('0.3')
print(f"Accurate result with Decimal: {result}") # This will print 0

Accurate result with Decimal: 0.0

HOW TO IMPROVE FLOATING-POINT ACCURACY?

Improving floating-point accuracy in Python can be achieved through several methods
depending on the context of your calculations. Floating-point numbers inherently have
precision limitations due to their binary representation. Here are some common
techniques to improve accuracy:

1. Using the decimal Module

The decimal module in Python provides higher precision and control over decimal
arithmetic compared to native floating-point numbers. This module is useful when
dealing with financial calculations or other high-precision requirements.

In [45]: from decimal import Decimal, getcontext

# Set precision (number of digits)

getcontext().prec = 50 # Increase the precision

# Example calculation using Decimal

x = Decimal('0.1')
y = Decimal('0.2')
z = x + y

print(f"Result with Decimal: {z}") # Exact result will be 0.3

Result with Decimal: 0.3

 Using decimal, you can avoid floating-point errors such as 0.1 + 0.2 != 0.3.

2. Using the fractions Module

The fractions module provides exact results for rational numbers by representing
numbers as fractions, avoiding the precision issues common with floating-point
arithmetic.

In [48]: from fractions import Fraction

# Use Fraction for exact arithmetic

x = Fraction(1, 10) # Represents 0.1
y = Fraction(2, 10) # Represents 0.2
result = x + y

print(f"Result with Fraction: {result}") # Result is exactly 3/10

print(f"As a float: {float(result)}") # Convert to float if necessary

Result with Fraction: 3/10

As a float: 0.3

The fractions module ensures exact arithmetic as long as the inputs are rational numbers.

3. Using SymPy for Symbolic Computations

If you are performing complex mathematical operations, symbolic computations using
sympy can maintain precision until you need a numerical result.

In [5]: from sympy import Rational, symbols

# Use Rational for precise fractions

x = Rational(1, 10) # 0.1
y = Rational(2, 10) # 0.2
z = x + y

# Perform symbolic computations

print(f"Result with SymPy: {z}") # Outputs 3/10

# Convert to floating-point only at the end, when needed

print(f"As a float: {float(z)}")

Result with SymPy: 3/10

As a float: 0.3

This approach is useful for high-precision calculations where symbolic manipulation is

needed until the final result.

4. Use numpy's Higher Precision Data Types

If you are working with arrays and numerical computations, numpy provides higher-
precision floating-point types like float128 or float64, which can reduce the rounding
errors typical with float32.

In [4]: import numpy as np

 # Use higher precision data type float128
x = np.float64(0.1)
y = np.float64(0.2)
z = x + y

print(f"Result with float64: {z}") # More precise result

Result with float64: 0.30000000000000004

Although the precision is not infinite, float128 offers higher accuracy than the standard
float64.

5. Using the math.fsum Function for Accurate Summation

Python's math.fsum() is a more accurate summation function for floating-point numbers
than the built-in sum().

In [13]: import math

numbers = [0.1] * 10 # Summing 0.1 ten times

print(f"Built-in sum: {sum(numbers)}") # May not be exactly 1.0
print(f"math.fsum: {math.fsum(numbers)}") # Guarantees a more accurate result

Built-in sum: 1.0

math.fsum: 1.0

math.fsum( ) uses a more precise algorithm to avoid accumulating rounding errors.

6. Context-Specific Solutions
For specific domains such as scientific computing, using specialized libraries like mpmath
(for arbitrary precision) or scipy can be beneficial. These libraries offer advanced
algorithms to maintain accuracy and improve floating-point precision in mathematical
and scientific computations.

Example: Using mpmath for Arbitrary Precision

In [19]: from mpmath import mp

# Set precision
mp.dps = 50 # Set precision to 50 decimal places

# Perform high-precision calculation

x = mp.mpf('0.1') + mp.mpf('0.2') - mp.mpf('0.3')
print(f"Result with mpmath: {x}") # Result will be exactly 0

Result with mpmath: 6.6819117752304891153513411678787046970379922002626e-52

mpmath allows for very high precision beyond what decimal offers.

Conclusion

To improve floating-point accuracy in Python:

1. Use decimal or fractions for exact arithmetic when dealing with decimals or rational
numbers.
 2. Use numpy's higher precision types like float128 for numerical operations.
3. Kahan Summation and similar algorithms can reduce accumulated errors in
summations.
4. Use mathematical reformulation to avoid subtracting nearly equal numbers.
5. math.fsum() provides more accurate summation than sum().
6. Use symbolic libraries like sympy or high-precision libraries like mpmath for more
complex or scientific calculations.
7. Each method improves accuracy in different ways, depending on the problem you
are tackling.

How to Determine the Machine Epsilon $\epsilon_m$??

You can find the machine epsilon using the sys module or the numpy module. The
machine epsilon is the smallest number that, when added to 1.0, results in a value
different from 1.00. $$1_c + \epsilon_m = 1_c$$

we should assume that all single-precision numbers contain an error in the sixth
decimal place and that all doubles have an error in

the 15th place.

Common ways to calculate the machine epsilon:

1. Using a manual iterative approach You can also calculate the machine epsilon
manually by successively halving a small number until adding it to 1.0 no longer gives a
value different from 1.0:

In [ ]: epsilon = 1.0
while (1.0 + epsilon) != 1.0:
epsilon /= 2

# Go one step back to get the machine epsilon

epsilon *= 2
print(epsilon)

2. Using sys.float_info.epsilon The simplest way is to use the sys module, which
provides system-specific parameters related to floating-point arithmetic:

In [ ]: import sys

epsilon = sys.float_info.epsilon
print(epsilon)

3. Using NumPy If you're using numpy, there's a built-in method to retrieve the machine
epsilon:

In [ ]: import numpy as np
epsilon = np.finfo(float).eps
print(epsilon)

Why is epsilon important?

 The machine epsilon ($\epsilon$) is an important concept in numerical computing
because it defines the precision limit of floating-point arithmetic. Here’s why it matters:

1. Understanding Floating-Point Precision

Floating-point representation: Computers use a limited number of bits to represent

real numbers in floating-point format. This representation is approximate, meaning
that not all numbers can be represented exactly, leading to rounding errors.
Machine epsilon gives us an idea of how close two numbers can be before the
computer considers them to be equal.
Smallest difference: Machine epsilon is the smallest positive number that, when
added to 1.0, results in a number distinguishable from 1.0. It helps quantify the
precision of floating-point arithmetic on a given system.

2. Error Analysis in Numerical Methods

When performing computations with floating-point numbers, tiny errors are

introduced due to rounding. Machine epsilon provides a benchmark for
understanding the magnitude of these errors.
In iterative numerical algorithms (e.g., solving linear systems, optimization), results
may need to be compared with epsilon to determine convergence. Knowing the
value of epsilon helps avoid situations where the algorithm falsely assumes that a
solution is exact.

3. Avoiding Over-Precision

In practical terms, trying to compute or compare values beyond machine epsilon can
lead to meaningless results or performance issues. For instance, asking whether two
floating-point numbers are exactly equal is often unreliable due to floating-point
rounding. Instead, comparisons are done within a tolerance range based on epsilon.

Example: When checking if two numbers are "equal," we often write:

In [ ]: if abs(a - b) < epsilon:

# a and b are considered equal

4. Guarding Against Numerical Instabilities

When working with operations like subtraction of nearly equal numbers (called
catastrophic cancellation) or matrix inversions, the precision limits defined by epsilon
can help detect potential numerical instability.
Small errors can grow disproportionately in some calculations (like dividing by a very
small number), so understanding machine epsilon allows programmers to design
algorithms that are more stable and avoid amplifying these small errors.

5. Floating-Point Arithmetic Limits

Knowing epsilon helps establish the limits of floating-point arithmetic, ensuring that
results are interpreted correctly. When performing large-scale simulations or
 calculations with high precision, understanding the bounds of the machine can
prevent misunderstandings or errors in results.

In summary, machine epsilon plays a critical role in managing and understanding the
limitations of numerical precision in floating-point arithmetic, ensuring that algorithms
are robust, accurate, and reliable within the limits of the system.

3.2 NUMERICAL ERROR

Rounding Error This error arises because digital computers cannot present some
quantities exactly. In Python, floating-point numbers may lead to precision errors due to
how numbers are represented in memory. $\pi=
3.1415926535897932384626433832795028841971693993751$, in Python
$\pi=3.141592653589793$. The Difference is $0.0000000000000002384626 . . .$ and is
called the rounding error on the number.

Accuracy It tells us how closely the computed value agrees with the true value.

Precision It tells us how closely individual computed values agree with each other.

Absolute Error $$|E_t| = |True - approx|$$

True Relative Error / Percentage Error $$|e_t| = 100 \% \times \frac{|True - approx|}
{True}$$

Relative Error /Percentage Error $$|e_a| = 100 \% \times \frac{|Present - Previous|}

{Present}$$ We iterate to a specified tolerance (stopping criterion) $e_s$
$$ e_a = \leq e_s$$

Truncation Errors It is an error caused by approximating a mathematical process. $$e^x

= \sum_{n=0}^{\infty} \frac{x^n}{n!}= 1 + \frac{x}{1!} + \frac{x^2}{2!} + \frac{x^3}{3!} +
...$$

EXAMPLE 3.1A: THE DIFFERENCE OF TWO

NUMBERS
In [82]: x = 1000000000000000
y = 1000000000000001.2345678901234
Tdif= 1.2345678901234
print(x, y, y-x, (y-x)/Tdif)

1000000000000000 1000000000000001.2 1.25 1.0125000091125467

if the difference between two numbers is very small, comparable with the error on the
numbers, i.e., with the accuracy of the computer, then the fractional error can become
large and you may have a problem.

EXAMPLE 3.2B: THE DIFFERENCE OF TWO

NUMBERS
 Consider the two numbers $x = 1$, $y = 1 + 10^{−15}\sqrt{2}$. Determine the
relative difference in $y -x$.

In [7]: from math import sqrt

x = 1.0
y = 1.0 + 1.0e-15*sqrt(2)
Truediff= 1.0e-15*sqrt(2)
diff= y-x
print('Truediff=', Truediff)
print('diff=', diff)
print('Relative Error=', (Truediff-diff)/Truediff)

Truediff= 1.4142135623730953e-15
diff= 1.3322676295501878e-15
Relative Error= 0.05794452478973511

EXAMPLE 3.3: Summing Series

*A classic numerical problem is the summation of a series to evaluate a function. As
an example, consider the infinite series for $\exp(x)$: $$\exp(x) =
\sum_{n=0}^{\infty} \frac{x^n}{n!}= 1 + \frac{x}{1!} + \frac{x^2}{2!} + \frac{x^3}
{3!} + ............. (\exact)$$

In [32]: import numpy as np

N=0
x=1
eps=1e-8
Tval=np.exp(x)
print('x', '\t', 'N','\t \t', 'exp(x)','\t \t', 'sum', '\t \t', '|exp(x) - sum|/
sum=1.
term=1.0
while abs(term/sum)> eps:
N+=1
term*=x**N/N
sum+=term
error=(Tval - sum)/Tval
print(x,'\t', N, '\t', Tval,'\t', sum, '\t', error)

x N exp(x) sum |exp(x) - sum|/e

xp(x)
1 11 2.718281828459045 2.718281826198493 8.31610676352332
7e-10

In [34]: import numpy as np

import math as m
N=0
x=1
eps=1e-8
Tval=np.exp(x)
print('x', '\t', 'N','\t \t', 'exp(x)','\t \t', 'sum', '\t \t', '|exp(x) - sum|/
sum=1.
term=1.0
while abs(term/sum)> eps:
N+=1
term=x**N/m.factorial(N)
sum=sum+term
 error=(Tval - sum)/Tval
print(x,'\t', N, '\t', Tval,'\t', sum, '\t', error)

x N exp(x) sum |exp(x) - sum|/e

xp(x)
1 11 2.718281828459045 2.718281826198493 8.31610676352332
7e-10

In [36]: import numpy as np

import math as m
N=11
x=1
Tval=np.exp(x)
print('x', '\t', 'N','\t \t', 'exp(x)','\t \t', 'sum', '\t \t', '|exp(x) - sum|/
sum=0.
for i in range (N):
sum+=x**i/m.factorial(i)
error=(Tval - sum)/Tval
print(x,'\t', N, '\t', Tval,'\t', sum, '\t', error)

x N exp(x) sum |exp(x) - sum|/e

xp(x)
1 11 2.718281828459045 2.7182818011463845 1.00477663102110
53e-08

EXAMPLE 3.4: Quantum harmonic Oscillator at

Finite Temperature
The quantum simple harmonic oscillator has energy levels $E_n = \hbar \omega(n
+1/2)$, where $n = 0, 1, 2, . . . , \infty$. As shown by Boltzmann and Gibbs, the
average energy of a simple harmonic oscillator at temperature $T$ is $$\left
<E\right> = \frac{1}{Z} \sum_{n=0}^{\infty} E_n e^{−\beta E_n}$$ where $\beta =
1/(k_BT)$ with $k_B$ being the Boltzmann constant, and $Z = \sum_{n=0}^{\infty}
e^{−\beta E_n}$. Suppose we want to calculate, approximately, the value of$\left <
E \right>$ when $k_BT = 100$. Since the terms in the sums for hEi and Z dwindle in
size quite quickly as n becomes large, we can get a reasonable approximation by
taking just the first 1000 terms in each sum. Working in units where $\hbar =
\omega = 1$.

here’s a program to do the calculation:

In [6]: #from math import exp

import numpy as np
N = 1000
beta = 1/100
S = 0.0
Z = 0.0
for n in range(N):
E = n + 0.5
weight = np.exp(-beta*E)
S += weight*E
Z += weight
print(S/Z)

99.95543134093475
 3.2 Accuracy Example: Solving Equations
Some algorithms may lose precision when solving certain mathematical problems. For
example, solving a quadratic equation where the discriminant is small.

In [7]: import math

# Quadratic equation ax^2 + bx + c = 0

a = 1
b = 100000000
c = 1

# Standard formula for solving quadratic equation

x1 = (-b + math.sqrt(b**2 - 4*a*c)) / (2*a)
x2 = (-b - math.sqrt(b**2 - 4*a*c)) / (2*a)

print(f"Roots (x1, x2): {x1}, {x2}")

Roots (x1, x2): -7.450580596923828e-09, -100000000.0

This example suffers from precision loss for the small discriminant case. You can rewrite it
using a numerically stable method to improve accuracy.

3.3 Speed Example: Loop Performance

Looping over a large list can be done using different methods, such as list
comprehensions, map(), and for loops. Each method has a different speed.

In [10]: import time

# Create a large list of numbers

large_list = list(range(1, 1000000))

# Using a for loop

start_time = time.time()
squares = []
for num in large_list:
squares.append(num ** 2)
end_time = time.time()
print(f"For loop time: {end_time - start_time:.5f} seconds")

# Using list comprehension

start_time = time.time()
squares = [num ** 2 for num in large_list]
end_time = time.time()
print(f"List comprehension time: {end_time - start_time:.5f} seconds")

# Using map()
start_time = time.time()
squares = list(map(lambda x: x ** 2, large_list))
end_time = time.time()
print(f"Map function time: {end_time - start_time:.5f} seconds")

For loop time: 0.15291 seconds

List comprehension time: 0.07359 seconds
Map function time: 0.10096 seconds
 This code compares the speed of three different looping methods over the same list.

4. Speed Example: Numpy vs Python List Operations

Python's numpy library is optimized for speed when working with large datasets
compared to native Python lists.

In [13]: import numpy as np

import time

# Create large arrays using Python lists and NumPy arrays

python_list = list(range(1, 1000000))
numpy_array = np.array(python_list)

# Using Python list

start_time = time.time()
python_squared = [x ** 2 for x in python_list]
end_time = time.time()
print(f"Python list time: {end_time - start_time:.5f} seconds")

# Using NumPy array

start_time = time.time()
numpy_squared = numpy_array ** 2
end_time = time.time()
print(f"NumPy array time: {end_time - start_time:.5f} seconds")

Python list time: 0.11895 seconds

NumPy array time: 0.00202 seconds

This example shows how numpy is significantly faster for vectorized operations
compared to Python lists.

Accuracy Example: Numerical Integration

When integrating functions numerically, Python’s built-in scipy library can be more
accurate than basic implementations.

In [16]: import numpy as np

from scipy.integrate import quad

# Define a function to integrate

def func(x):
return np.sin(x)

# Use numpy's trapz method for numerical integration

x = np.linspace(0, np.pi, 1000)
y = np.sin(x)
integral_trapz = np.trapz(y, x)
print(f"Trapezoidal integration result: {integral_trapz}")
# Use scipy's quad method (more accurate)
integral_quad, error = quad(func, 0, np.pi)
print(f"Quad integration result: {integral_quad}, with error: {error}")

Trapezoidal integration result: 1.999998351770852

Quad integration result: 2.0, with error: 2.220446049250313e-14
 The scipy.integrate.quad method is much more accurate than the trapezoidal rule for
certain types of functions.

These examples demonstrate how to measure both accuracy and speed in Python, using
different techniques like floating-point arithmetic, loops, vectorization with numpy, and
parallel computing with multiprocessing.

Optimized Kahan Summation Algorithm Inline the code for faster execution: If you don't
need a separate function call, inlining the Kahan summation directly into your main code
helps avoid function call overhead. Minimize operations: Reduce the number of variables
updated or operations repeated in each iteration. Optimized Code Example

In [23]: def kahan_summation_optimized(numbers):

sum_ = 0.0 # Initialize the sum
c = 0.0 # Compensation for lost low-order bits

for num in numbers:

# Calculate compensated value
y = num - c
t = sum_ + y

# New compensation (to avoid losing low-order bits)

c = (t - sum_) - y

# Update sum
sum_ = t

return sum_

# Example usage
numbers = [0.1] * 1000000 # A large list of numbers to sum
result = kahan_summation_optimized(numbers)
print(f"Optimized Kahan summation result: {result}")

Optimized Kahan summation result: 100000.0

Explanation of Optimization

1. Reduced Assignments: The algorithm has minimal assignments and doesn't create
new variables unnecessarily inside the loop.
2. Efficient Calculation: We directly update the sum and the compensation (c) in every
iteration, avoiding redundant computations.
3. Loop Efficiency: The loop remains tight, avoiding unnecessary operations outside of
essential floating-point additions and subtractions.

Speed and Accuracy

This optimized Kahan summation will maintain accuracy for summing floating-point
numbers while reducing any performance penalties typically associated with floating-
point arithmetic. It's faster than the unoptimized version, as unnecessary operations are
minimized while keeping the core logic intact.

In [ ]:
 5. Minimizing Rounding Errors in Large Calculations
When adding or subtracting many floating-point numbers, rounding errors can
accumulate. Use algorithms designed to minimize these errors, such as Kahan
Summation.

Kahan Summation Algorithm:

This algorithm reduces numerical errors when summing a sequence of floating-point

numbers.

In [9]: def kahan_summation(numbers):

sum_ = 0.0
c = 0.0 # A running compensation for lost low-order bits
for num in numbers:
y = num - c # Subtract the compensation
t = sum_ + y # Add the compensated value
c = (t - sum_) - y # Calculate the new compensation
sum_ = t
return sum_

# Example usage
numbers = [0.1, 0.2, 0.3, 0.4]
print(f"Kahan summation result: {kahan_summation(numbers)}")

Kahan summation result: 1.0

This method helps improve accuracy in large summations.

6. Avoiding Subtraction of Nearly Equal Numbers

When subtracting numbers that are close in value, significant digits can be lost, leading
to inaccuracies. Instead of directly subtracting, consider using mathematical identities to
transform the equation.

Example: Calculating a Difference of Squares

If you need to calculate $𝑎^2 − 𝑏^2$, instead of directly computing: $𝑎^2 − 𝑏^2 = (𝑎 −
𝑏)(𝑎 + 𝑏)$

This reformulation is numerically stable because it avoids subtracting two large numbers
that are nearly equal.

In [11]: def difference_of_squares(a, b):

return (a - b) * (a + b)

# Example usage
a = 1000000.001
b = 1000000.0
print(f"Result with reformulation: {difference_of_squares(a, b)}")

Result with reformulation: 2000.0000959949027

In [ ]:
In [ ]:

In [ ]: