11

Statistical Inferences

Once we have estimated the unknown parameters that appear in an algebraic

or ODE model, it is quite important to perform a few additional calculations to
establish estimates of the standard error in the parameters and in the expected re-
sponse variables. These additional computational steps are very valuable as they
provide us with a quantitative measure of the quality of the overall fit and inform
us how trustworthy the parameter estimates are.

11.1 INFERENCES ON THE PARAMETERS

When the Gauss-Newton method is used to estimate the unknown parame-
ters, we linearize the model equations and at each iteration we solve the corre-
sponding linear least squares problem. As a result, the estimated parameter values
have linear least squares properties. Namely, the parameter estimates are normally
distributed, unbiased (i.e., E(k*)=k) and their covariance matrix is given by

COV(k*) = Og [A*]~' (11.1)

where A* is matrix A evaluated at k*. It should be noted that for linear least
squares matrix A is independent of the parameters while this is clearly not the case
for nonlinear least squares problems. The required estimate of the variance Og is
obtained from
„ 2 S(k*) S(k*)
(11.2)
(d.f.) N m- p
177

Copyright © 2001 by Taylor & Francis Group, LLC

 178 Chapter 11

where (d.f.)= Nm—p are the degrees of freedom, namely the total number of meas-
urements minus the number of unknown parameters.
The above expressions for the C(3F(k*) and ojr are valid, if the statistically
correct choice of the weighting matrix Q;, (i=l,...,N) is used in the formulation of
the problem. Namely, if the errors in the response variables (EJ, i=l,...,N) are nor-
mally distributed with zero mean and covariance matrix,

COF(£]) = a;: M, (1L3)

we should use [Mj]~ 1-1 as the weighting matrix Q. where the matrices M j; i=l,...,N
are known whereas the scaling factor, a^ , could be unknown. Based on the struc-
ture of MJ we arrive at the various cases of least squares estimation (Simple LS,
Weighted LS or Generalized LS) as described in detail in Chapter 2.
Although the computation of COK(k*) is a simple extra step after conver-
gence of the Gauss-Newton method, we are not obliged to use the Gauss-Newton
method for the search of the best parameter values. Once k* has been obtained
using any search method, one can proceed and compute the sensitivity coefficients
by setting up matrix A and thus quantify the uncertainty in the estimated parame-
ter values by estimating COK(k*).
Approximate inference regions for nonlinear models are defined by analogy
to the linear models. In particular, the (1 -a) 100% joint confidence region for the
parameter vector k is described by the ellipsoid,

or

[ k - k f [A*r 1 [k-k*] = F« Nm ™ P (1Mb)

N m- p

where a is the selected probability level in Fisher's F-distribution and ^Nm-p is

obtained from the F-distribution tables with Vj=p and v2=(N/w-/?) degrees of free-
dom. The corresponding (I -a) 100% marginal confidence interval for each pa-
rameter, kj, i=l,2,...,p, is given by

k * - t * / 2 o k i < kj <k* +t*/2oki (11.5)

where t^ 7 2 is obtained from the tables of Student's T-distribution with v=(Nw-p)

degrees of freedom. The standard error of parameter kj, a k . , is obtained as the

Copyright © 2001 by Taylor & Francis Group, LLC

 Statistical Inferences 179

square root of the corresponding diagonal element of the inverse of matrix A*

multiplied by 6 g , i.e.,

°kj = °E

It is reminded that for v>30 the approximation t^ / 2 ~ z^ can be used

where zaj2 is obtained from the standard normal distribution tables. Simply put,
when we have many data points, we can simply take as the 95% confidence inter-
val twice the standard error (recall that zom$=\ .96 whereas t 0 025=2.042).
The linear approximation employed by the Gauss-Newton method in solv-
ing the nonlinear least squares problem enables us to obtain inference regions for
the parameters very easily. However, these regions are only approximate and
practice has shown that in many cases they can be very misleading (Watts and
Bates, 1988). Nonetheless, even if the regions are not exact, we obtain a very good
idea of the correlation among the parameters.
We can compute the exact (1 -a) 100% joint parameter likelihood region
using the equation given below

S(k) = S(k*) 1h + ^ _ _ ^HN H

(11.7)
Nw-_p

The computation of the above surface in the parameter space is not trivial.
For the two-parameter case (p=2), the joint confidence region on the k r k 2 plane
can be determined by using any contouring method. The contour line is approxi-
mated from many function evaluations of S(k) over a dense grid of (k b k2) values.

11.2 INFERENCES ON THE EXPECTED RESPONSE VARIABLES

Having determined the uncertainty in the parameter estimates, we can pro-
ceed and obtain confidence intervals for the expected mean response. Let us first
consider models described by a set of nonlinear algebraic equations, y=f(x,k). The
100(1-a)% confidence interval of the expected mean response of the variable y. at
x0 is given by

f x
( o,k*)-C2<i < u < f ( x 0 , k * ) + tj; / 2 a (11.8)

Copyright © 2001 by Taylor & Francis Group, LLC

 180 Chapter 11

where t^ / 2 is obtained from the tables of Student's T-distribution with v=(Nw-p)

degrees of freedom. Based on the linear approximation of the model equations, we
have

of1
y0(k) = f(x0,k*) [k - k *] (11.9)
5k

with the partial derivative (5fT/5k)r evaluated at x0 and k*. Taking variances from
both sides we have

\T

(ll.lOa)
5k dk

Substitution of the expression for COK(k*) yields,

T
\T T
5f
(ll.lOb)
5k 5k

The standard prediction error of yjo, o y .Q , is the square root of the jth diago-
nal element of COF(y0), namely,

5k 5k

Equation 11.8 represents the confidence interval for the mean expected re-
sponse rather than a. future observation (future measurement) of the response vari-
able, y 0 . In this case, besides the uncertainty in the estimated parameters, we must
include the uncertainty due to the measurement error (EO). The (l~a)100% confi-
dence interval of y JQ is

(II.12)

where the standard prediction error of y JQ is given by

Copyright © 2001 by Taylor & Francis Group, LLC
 Statistical Inferences 181

°y.,o =
-'
5f
ok
,1
T

[Af
'*>\ok
(11.13)
V ) V /

Next let us turn our attention to models described by a set of ordinary differ-
ential equations. We are interested in establishing confidence intervals for each of
the response variables y j , j=l,...,w at any time t=t0. The linear approximation of
the output vector at time to,

y(t 0 ,k) = Cx(t 0 ,k*) + CG(t 0 )[k-k*] (11.14)

yields the expression for COK(y(t0)),

CW(y 0 ) = CG(t0)COF(k*)CTGT(t0) (11.15)

which can be rewritten as

CW(y 0 ) = d2 (11.16)

with the sensitivity coefficients matrix G(to) evaluated at k*. The estimated stan-
dard prediction error of yj(to) is obtained as the square root of the j* diagonal ele-
ment of CO F(y(t0)).

A*]- 1 C T G T (t 0 )J i j (11.17)

Based on the latter, we can compute the (l-a)100% confidence interval of

the expected mean response of y, at t=t0,

y j (t 0 ,k*)-t^ / 2 d < u < y ( t 0 , k * ) + C26 (11.18)

If on the other hand we wish to compute lhe(l-a)100% confidence interval

of the response of y, at t=t0, we must include the error term (EO) in the calculation
of the standard error, namely we have

v
y j (t 0 ,k*)-t a / 2 o y j o < y j (t 0 ) < (11.19)
with

Copyright © 2001 by Taylor & Francis Group, LLC

 182 Chapter 11

+ {CG(t 0 )[A*]- 1 C T G T (t 0 )J (11.20)

11.3 MODEL ADEQUACY TESTS

There is a plethora of model adequacy tests that the user can employ to de-
cide whether the assumed mathematical model is indeed adequate. Generally
speaking these tests are based on the comparison of the experimental error vari-
ance estimated by the model to that obtained experimentally or through other
means.

1 1 .3. 1 Single Response Models

Let us first consider models that have only one measured variable (m=\).
We shall consider two cases. One where we know precisely the value of the ex-
perimental error variance and the other when we have an estimate of it. Namely,
there is quantifiable uncertainty in our estimate of the experimental error variance.

CASE 1 : <r£ is known precisely:

In this case we assume that we know precisely the value of the standard
experimental error in the measurements (oe). Using Equation 1 1.2 we obtain an
estimate of the experimental error variance under the assumption that the model is
adequate. Therefore, to test whether the model is adequate we simply need to test
the hypothesis

H : c =
«
« model

at any desirable level of significance, e.g., a=0.05. Here with o , , we denote

the error variance estimated by the model equations (Equation 11.2); namely,
dg is an estimate of
Since o g is known exactly (i.e., there is no uncertainty in its value, it is a
given number) the above hypothesis test is done through a x -test. Namely,

=>
If Xdata > Xv=(Nm-p),l-a Reject H0

where

Copyright © 2001 by Taylor & Francis Group, LLC

 Statistical Inferences 183

and Xv=(Nm-p),l-a 's obtained from the tables of the jf-distribution with degrees
of freedom v=(N/w-p).

CASE 2: o£ is known approximately:

Let us assume that o £ is not known exactly, however, we have performed

n repeated measurements of the response variable. From this small sample of
multiple measurements we can determine the sample mean and sample variance. If
s g is the sample estimate of oe , estimated from the n repeated measurements it
is given by

S7E = —\ ^
/ ( y j - y )j (11.22)
n-1 £

where the sample mean is obtained from

(11.23)

Again, we test the hypothesis at any desirable level of significance, for ex-
ample ce=0.05

H : = a
o <* model £

In this case, since o g is known only approximately, the above hypothesis

is tested using an F-test, i.e.,

v Nm w v22= n ~l
iIff Fr data >
-^ Fj_'
i- l=( ~P)>
' ^
=> n • *H
Reject n
a o

where
Copyright © 2001 by Taylor & Francis Group, LLC
 184 Chapter 11

(H-24)

and F v j = d f >v ' 2 = n ~' is obtained from the tables of the F-distribution.

11.3.2 Multivariate Models

Let us now consider models that have only more than one measured variable
(m>\). The previously described model adequacy tests have multivariate exten-
sions that can be found in several advanced statistics textbooks. For example, the
book Introduction to Applied Multivariate Statistics by Srivastava and Carter
(1983) presents several tests on covariance matrices.
In many engineering applications, however, we can easily reduce the prob-
lem to the univariate tests presented in the previous section by assuming that the
covariance matrix of the errors can be written as

CWfe) = cjgM, ; i=l,...,N (11.25)

where M; are known matrices. Actually quite often we can further assume that the
matrices IM,, i=l,...,N are the same and equal to matrix M.
An independent estimate of COV(z), E , that is required for the adequacy
tests can be obtained by performing NK repeated experiments as

or for the case of univariate tests, s £ , the sample estimate of O E in Equation

11.25, can be obtained from

NR

(N R -i " ' "••" "J u

"' "' (11.27)

Copyright © 2001 by Taylor & Francis Group, LLC