Scientific Programming (in C)

Steve Summit

C was not used very much for floating-point work at first (i.e. back when it was still
being developed and refined), so its support for some aspects of floating-point and
scientific work is a little weak. (On the other hand, C's support for string processing and
other data structures can make up the difference in a program that does some of each.)
There are two significant areas of concern. One is specific to C, and concerns the
handling of multidimensional arrays. The other is endemic in floating-point work done on
computers, and concerns accuracy, precision, and round-off error.

Functions and Multi-dimensional Arrays

We've seen that when a function receives a one-dimensional array as a parameter:

f(double a[])
{
...
}
the size of the array need not be specified, because the function is not allocating the array
and does not need to know how big it is. The array is actually allocated in the caller (and
in fact, only a pointer to the beginning of the array is passed to a function like f). If, in
the function, we need to know the number of elements in the array, we have to pass it as a
second parameter:
f(double a[], int n)
{ ... }
If we wish to emphasize that it is really a pointer which is being passed, we can write it
like this:
f(double *a, int n)
{ ... }
(In fact, if we declare the parameter as an array, the compiler pretends we declared it as a
pointer, as in this last form.)

In the one-dimensional case, it is actually a bit of an advantage that we do not need to

specify the size of the array. If f does something generic like ``find the average of the
numbers in the array,'' it will clearly be useful to be able to call it with arrays of any size.
(If we had to write things like

f(double a[10])
{ ... }
then we might find it a real nuisance that f could only operate on arrays of size 10; we'd
hate to have to write several different versions of f, one for each size.)

Unfortunately, the situation is not nearly so straightforward for multi-dimensional arrays.

If you had an array

double a2[10][20];
which you wanted to pass to a function g, what would g receive? We could declare g as
g(double a[10][20])
{ ... }
and let the compiler worry about it--that is, let it ignore the dimensions it doesn't need,
and pretend that g really receives a pointer. But what pointer does it receive? It's easy to
imagine that g could also be declared as
g(double **a) /* WRONG */
{ ... }
but this is incorrect. Notice that the array a2 that we're trying to pass to g is not really a
multidimensional array; it is an array of arrays (more precisely, a one-dimensional array
of one-dimensional arrays). So in fact, what g receives is a pointer to the first element of
the array, but the first element of the array is itself an array! Therefore, g receives a
pointer to an array. Its actual declaration looks like this:
g(double (*a)[20])
{ ... }
(and this is what the compiler pretends we'd written if we instead write a[10][20] or a[]
[20]). This says that the parameter a is a pointer to an array of 20 doubles. We can still
use normal-looking subscripts: a[i][j] is the j'th element in the i'th array pointed to by
a. The problem is that no matter how we declare g and its array parameter a, we have to
specify the second dimension. If we write
g(double a[][])
the compiler complains that a needed dimension is missing. If we write
g(double (*a)[])
the compiler complains that a needed dimension is missing. We have to specify the
``width'' of the array, and if we've got various sizes of arrays running around in our
program, we're back with the problem of potentially needing different functions to deal
with different sizes of arrays.

Why wouldn't

g(double **a) /* WRONG */

work? That says that a is a pointer to a pointer, but since we're passing an array of arrays,
the function will receive a pointer to an array. A pointer to an array is not compatible
with a pointer to a pointer. (More on double **a below.)

A second problem arises if you need any temporary arrays inside a function. Often, a
temporary array needs to be the same size as the passed-in arrays being worked on.
Unfortunately, there's no way to declare an array and say ``just make it as big as the
passed-in array.''

It's easy to imagine a solution to all of the problems we've been discussing. To write a
function accepting a multidimensional array with the size specified by the caller, you
might want to write

g(double a[m][n], int m, int n) /* won't work */

where the array dimensions are taken from other function parameters. (This is, of course,
just how you do it in FORTRAN.) To declare a temporary array, you might try things like
f(double a[n], int n)
 {
double tmparray[n]; /* won't work */
...
}
or
g(double a[m][n], int m, int n)
{
double tmparray[m][n]; /* won't work */
...
}
But you can't do any of these things in standard C. Every array dimension must be a
compile-time constant. Array parameters can't take their dimensions from other
parameters, and local arrays can't take their dimensions from parameters.

(It is worth noting that at least one popular compiler, namely gcc, does let you do these
things: parameter and local arrays can take their dimensions from parameters. If you
make use of this extension, just be aware that your program won't work under other
compilers.)

How can we work around these limitations? The most powerful and flexible solution
involves abandoning multidimensional arrays and using pointers to pointers instead. This
approach has the advantage that all dimensions may be specified at run time, though it
also has a few disadvantages. (Namely: you may have to remember to free the arrays,
especially if they're local; the intermediate pointers take up more space, and accessing
``array'' elements via pointers-to-pointers may be slightly less efficient than true subscript
references. On the other hand, on some machines, pointer accesses are more efficient.
Another disadvantage is that our treatise on scientific programming in C is about to
digress into a treatise on memory allocation.)

To illustrate, here is a pair of functions for allocating simulated one- and two-dimensional
arrays of double. (By ``simulated'' I mean that these are not true arrays as the compiler
knows them, but because of the way arrays and pointers work in C, we are going to be
able to treat these pointers as if they were arrays, by using [] array subscript notation.)

#include <stdio.h>
#include <stdlib.h>

double *alloc1d(int n)
{
double *dp = (double *)malloc(n * sizeof(double));
if(dp == NULL) {
fprintf(stderr, "out of memory\n");
exit(1);
}
return dp;
}

double **alloc2d(int m, int n)

{
int i;
double **dpp = (double **)malloc(m * sizeof(double *));
 if(dpp == NULL) {
fprintf(stderr, "out of memory\n");
exit(1);
}
for(i = 0; i < m; i++) {
if((dpp[i] = (double *)malloc(n * sizeof(double))) ==
NULL) {
fprintf(stderr, "out of memory\n");
exit(1);
}
}
return dpp;
}

These routines print an error message and exit if they can't allocate the requested
memory. Another approach would be to have them return NULL, and have the caller check
the return value and take appropriate action on failure. (In this case, alloc2d would want
to ``back out'' by freeing any intermediate pointers it had successfully allocated.)

Note that these routines allocate arrays of type double. They could be trivially rewritten
to allocate arrays of other types. (Note, however, that it is impossible in portable,
standard C to write a routine which generically allocates a multidimensional array of
arbitrary type, or with an arbitrary number of dimensions.)

When a one-dimensional array allocated by alloc1d is no longer needed, it can be freed

by passing the pointer to the standard free routine. Freeing two-dimensional arrays
allocated by alloc2d is a bit trickier, because all the intermediate pointers need to be
freed, too. Here is a function to do it:

void free2d(double **dpp, int m)

{
int i;
for(i = 0; i < m; i++)
free((void *)dpp[i]);

free((void *)dpp);
}
(The casts in the malloc and free calls in these routines are not necessary under ANSI
C, but are included to make it easier to port this code to a pre-ANSI compiler, if
necessary.)

With these routines in place, we can dynamically allocate one- and two-dimensional
arrays of double to our heart's content. Here is an (artificial) example:

double *a1 = alloc1d(10);

double **a2 = alloc2d(10, 20);
a1[0] = 0;
a1[9] = 9;
a2[0][0] = 0.0;
a2[0][19] = 0.19;
a2[9][0] = 9.0;
 a2[9][19] = 9.19;

Now, when we pass a two-dimensional array allocated by alloc2d to a function, that

function will be declared as accepting a pointer-to-a-pointer:

g(double **a) /* okay, for alloc2d "arrays" */

When we need a temporary array within a function, we can allocate one with alloc1d or
alloc2d; however, we must remember to free it before the function returns. (Remember
that malloc'ed memory persists until explicitly freed, or until the program exits. Local
variables disappear when the function they're local to returns, but in the case of pointers
to malloc'ed memory, that just means that we lose the pointer, and not that the pointed-to
memory has been freed. We have to explicitly free it, before we lose the pointer.)

One last note: since

g1(double a[10][20])
is not compatible with
g2(double **a)
we will have to maintain a segregation between arrays-of-arrays and pointers-to-pointers.
We can't write a single function that operates on either a true multidimensional array or
one of our pointer-to-pointer, simulated multidimensional arrays. (Actually, there is one
way, but it's more elaborate, and not strictly portable.)

That's about enough (for now, at least) about multidimensional arrays. We now turn to
the subject of...

Floating-Point Precision

It's the essence of real numbers that there are infinitely many of them: not only infinitely
large (in both the positive and negative direction), but infinitely small and infinitely fine.
Between any two real numbers like 1.1 and 1.2 there are infinitely many more numbers:
1.11, 1.12, etc., and also 1.101, 1.102, etc., and also 1.1001, 1.1002, etc.; and etc. There is
no way in a finite amount of space to represent the infinite granularity of real numbers.
Most computers use an approximation called floating point. In a floating point
representation, we keep track of a base number or ``mantissa'' with some degree of
precision, and a magnitude or exponent. The parallel here with scientific notation is
close: The number actually represented is something like mantissa x
10^exponent. (Actually, most computers use mantissa x
2^exponent, and we'll soon see how this difference can be important.)

The advantage of a floating-point representation is that it lets us devote a finite amount of

space to each value but still do a decent job of approximating the real numbers we're
interested in. If we have six decimal digits of precision, we can represent numbers like
123.456 or 123456 or 0.123456 or 0.000000123456 or 123456000000. However, we can
not accurately represent numbers which require more than six significant digits: we can't
talk about numbers like 123456.123456 or 0.1234567 or even 1234567. If we try to
represent numbers like these (i.e. if they occur in our calculations), they'll be ``rounded
off'' somehow. (``Rounded off'' is in quotes because we don't always know whether the
unrepresentable digits will be properly rounded up or down, or randomly rounded up or
down, or simply discarded. High-quality floating-point processors and subroutine
libraries will try to do a good job of rounding when a result cannot be exactly
represented, but unfortunately there are some low-quality ones out there.)

In simple circumstances, the finite precision of floating-point numbers is not a problem.

Real-world inputs (i.e. anything you measure) never have infinite precision; in fact they
often have only two or three significant digits of precision. Almost everyone has a vague
understanding that floating-point representations have finite precision, so it's easy to
shrug off slight inaccuracies as ``roundoff error.'' If these errors always stayed confined
to the least-significant digit, they wouldn't be much of a problem.

The problem is that error can accumulate. In the worst case, you can lose one (or more
than one) significant digit in each step of a calculation. If a calculation involves six steps,
and you only have six significant digits to play with, by the end of the calculation, you've
got nothing of significance left. As a wise programmer once said, ``Floating point
numbers are like piles of sand. Every time you move them around, you lose a little sand.''

How can an error grow to involve more than the least-significant digit? Here is one
simple example. Suppose we are computing the sum 1+1+1+1+1+1+1+1+1+1+1000000,
and we have only six significant digits to play with. Clearly the answer is 1000010,
which has six significant digits, so we're okay. But suppose we compute it in the other
order, as 1000000+1+1+1+1+1+1+1+1+1+1. In the first step, 1000000+1, the
intermediate result is 1000001, which has seven significant digits and so cannot be
represented. It will probably be rounded back down to 1000000. So
1000000+1+1+1+1+1+1+1+1+1+1 = 1000000, and 1+1+1+1+1+1+1+1+1+1+1000000 !
= 1000000+1+1+1+1+1+1+1+1+1+1. In other words, the associative law does not hold
for floating-point addition. Elementary school students often wonder whether it makes a
difference which order you add the column of numbers up in (and sometimes notice that
it does, i.e. they sometimes make mistakes when they try it). But here we see that, on
modern digital computers, the folklore and superstition is true: it really can make a
difference which order you add the numbers up in.

If you don't believe the above, I invite you to try this little program:

#include <stdio.h>

#define BIG 1e8

main()
{
int i;
float f = 0;

for(i = 0; i < 20; i++)

f += 1;
f += BIG;
printf("%f\n", f);
 f = BIG;
for(i = 0; i < 20; i++)
f += 1;
printf("%f\n", f);

return 0;
}
You may have to play with the value of BIG (other values to try are 1e6, 1e7, or 1e9), but
you should probably be able to get a surprising result.

Because of the finite precision of floating point numbers on computers, you also have to
be careful when subtracting large numbers, as the result may not be significant at all. If
we try to compute the difference 100000789 - 100000123 (again using only six
significant digits), we are not likely to get 666, because neither 100000789 nor
100000123 are representable in six digits in the first place. This may not seem surprising,
but what if the difference we were really interested in, and thought was significant, was
789 - 123, with that bias of 100000000 being an artifact of the algorithm which we
assumed would cancel out? In this case, it does not cancel out, and in general we cannot
assume that

(a + b) - (c + b) = a - c

Another thing to know about floating point numbers on computers is that, as mentioned
above, they're usually represented internally in base 2. It's well known that in base 10,
fractions like 1/3 have infinitely-repeating decimal representations. (In fact, if you
truncate them, you can easily find anomalies like the one we discussed in the previous
paragraphs, such as that 0.333+0.333+0.333 is 0.999, not 1.) What's not so well known is
that in base 2, the fraction one tenth is an infinitely-repeating binary number: it's
0.00011001100110011₂, where the 0011 part keeps repeating. This means
that almost any number that you thought was a ``clean'' decimal fraction like 0.1 or 2.3 is
probably represented internally as a messy infinite binary fraction which has been
truncated or rounded and hence is not quite the number you thought it was. (In fact,
depending on how carefully routines like atof and printf have been written, it's
possible to have something like printf("%f", atof("2.3")) print an ugly number like
2.299999.) Where you have to watch out for this is that if you multiply a number by 0.1,
it is not quite the same as dividing it by 10. Furthermore, if you multiply a number by 0.1
three times, it may be different than if you'd multiplied it by 0.001 and different than if
you'd divided it by 1000.

My purpose in bringing up these anomalies is not to scare you off of doing floating point
work forever, but to drive home the point that floating point inaccuracies can be
significant, and cannot be brushed of as simple uncertainty about the least-significant
digit. With care, the inaccuracies can be confined to the least-significant digit, and the
care involves making sure that errors do not proliferate and compound themselves. (Also,
I should again point out that none of these problems are specific to C; they arise any time
floating-point work is done with finite precision.)
In practice, anomalies crop up when both large and small numbers participate in the same
calculation (as in the 1+1+1+1+1+1+1+1+1+1+1000000 example), and when large
numbers are subtracted, and when calculations are iterated many times. One thing to
beware of is that many common algorithms magnify the effects of these problems. For
example, the definition of the standard deviation is the square root of the mean variance,
namely

sqrt(sum(sqr(x[i] - mean)) / n)
where sum() and sqr() are summing and squaring operators (neither of which are
standard C library functions, of course) and where mean is
sum(x[i]) / n
(The statisticians out there know that the denominator in the first expression is not
necessarily n but should usually be n-1.)

Computing standard deviations from the definition can be a bit of a nuisance because you
have to walk over the numbers twice: once to compute the mean, and a second time to
subtract it from each input number, square the difference, and sum the squares. So it's
quite popular to rearrange the expression to

sqrt((sum(sqr(x[i])) - sqr(sum(x))/n) / n)
which involves only the sum of the x's and the sum of the squares of the x's, and hence
can be completed in one pass. However, it turns out that this method can in practice be
significantly less accurate. By squaring each individual x and accumulating
sum(sqr(x[i])), the big numbers get bigger and the small numbers get smaller, so the
small numbers are more likely to get lost in the underflow. When you use the definition,
it's only the differences that get squared, so less of significance gets lost. (However, even
working from the definition, if the individual variances sum to near zero they can
sometimes end up summing to a slightly negative number, due to accumulated error,
which is a problem when you try to take the square root.)

Other standard mathematical manipulations to watch out for are:

The quadratic formula:

If b² and 4ac are both large, there may be very few significant digits left in
the determinant b² - 4ac.

Solving systems of equations, inverting matrices, computing the determinant of matrices:

The standard mathematical techniques involve large numbers of compounded operations

which can lost a lot of accuracy.

Iterated simulations:

Obviously, any iteration is prone to compounded errors.

Trigonometric reductions
The only sane ways of computing trig functions numerically (i.e. the ways likely to be
used by library functions such as sin, cos, and tan) work only for the one period of the
function near 0. These functions will of course work on arguments outside the interval (-
pi, pi), but the function must first reduce the arguments by subtracting some multiple of
2*pi, and as we've seen, taking the difference of two large numbers (whether you do it or
the library function does it) can lose all significance.

Compounded interest

The standard formula is something like

principle x (1 + rate)ⁿ
but raising a number near 1 to a large power can be inaccurate.

Converting strings to numbers:

[okay, so this is a computer science problem, not a mathematical problem] If you handle
digits to the right of the decimal by repeatedly multiplying by 0.1, you'll lose accuracy.

For some of these problems, using double instead of single precision makes the
difference, but for others, great care and rearrangement of the algorithms may be required
for best accuracy.

Returning to C, you might like to know that type float is guaranteed to give you the
equivalent of at least 6 decimal digits of significance, and double (which is essentially
C's version of the ``double precision'' mentioned in the preceding paragraph) is
guaranteed to give you at least 10. Both float and double are guaranteed to handle
exponents in at least the range -37 to +37.

One other point relating to numerical work and specific to C is that C does not have a
built-in exponentiation operator. It does have the library function pow, which is the right
thing to use for general exponents, but don't use it for simple squares or cubes. (Use
simple multiplication instead.) pow is often implemented using the identity

x^y = e^{y ln(x)}

and the inevitable slight inaccuracies in the underlying natural log and e^x
routines can foil you. For example, if you compute (int)pow(2., 3.), on some
machines the answer comes out 7, because pow returns a number like 7.99999, and C's
floating-to-int conversion simply discards all of the fractional part.

The fact that a floating-to-int conversion truncates the fractional part has nothing to do,
by the way, with the problem mentioned earlier of a poor-quality floating-point processor
not rounding well. (Back there we were talking about rounding an intermediate result
which was still trying to be represented as a floating-point number.) When you do want to
round a number off to the nearest integer, the usual trick is
 (int)(x + 0.5)
By adding 0.5 before truncating the fractional part, you arrange that numbers with
fractions of .5 or above get rounded up. (This trick obviously doesn't handle negative
numbers correctly, nor does it do even/odd rounding.)

***

I am not a numerical analyst, so this presentation has been somewhat superficial. On the
other hand, I've tried to make it practical and accessible and to highlight the real-world
problems which do come up in practice but which too few programmers are aware of.
You can learn much more (including the traditional notations for describing and
analyzing floating point accuracy, and some clever algorithms for doing floating point
arithmetic without losing so much accuracy) in the following references.

David Goldberg, ``What Every Computer Scientist Should Know about Floating-Point
Arithmetic,'' ACM Computing Surveys, Vol. 23 #1, March, 1991, pp. 5-48.

Brian W. Kernighan and P.J. Plauger, The Elements of Programming Style, Second
Edition, McGraw-Hill, 1978, ISBN 0-07-034207-5, pp. 115-118.

Donald E. Knuth, The Art of Computer Programming, Volume 2: Seminumerical

Algorithms, second edition, Addison-Wesley, 1981, ISBN 0-201-03822-6, chapter 4.

William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery,

Numerical Recipes in C, 2nd edition, Cambridge University Press, 1992, ISBN 0-521-
43108-5.