716 IEEE TRANSACTI ONS ON AUTOMATI C CONTROL, VOL. AC-19, KO.

6, DECEMBER 1974
A New Look at the Statistical Model Identification
HIROTUGU AI(AIKE, JIEJIBER, IEEE
Abstract-The history of the development of statistical hypothesis
testing in time series analysis is reviewed briefly and it is pointed
out that the hypothesis testing procedure is not adequately defined
as the procedure for statistical model identilication. The classical
maximum likelihood estimation procedure is reviewed and a new
estimate minimum information theoretical criterion (AIC) estimate
(MAICE) which is designed for the purpose of statistical identifica-
tion is introduced. When there are several competing models the
MAICE is defined by the model and the maximum likelihood esti-
mates of the parameters which give the minimum of AIC defined by
AIC = (-2)log(maximum likelihood) +2(number of
independently adjusted parameters within the model).
MAICE provides a versatile procedure for statistical model identi-
fication which is free from the ambiguities inherent in the application
of conventional hypothesis testing procedure. The practical utility of
MAICE in time series analysis is demonstrated with some numerical
examples.
I
I. IXTRODUCTION
X spite of the recent, development of t.he use of statis-
tical concepts and models in almost, every field of engi-
neering and science it seems as if the difficulty of con-
structing an adequate model based on the information
provided by a finite number of observations is not fully
recognized. Undoubtedly the subject of statistical model
construction or ident.ification is heavily dependent on the
results of theoret.ica1 analyses of the object. under observa-
tion. Yet. it must be realized that there is usually a big gap
betn-een the theoretical results and the pract,ical proce-
dures of identification. A typical example is the gap between
the results of the theory of minimal realizations of a linear
system and the identifichon of a Markovian representa-
tion of a stochastic process based on a record of finite
duration. A minimal realization of a linear system is
usually defined through t.he analysis of the rank or the
dependence relation of the rows or columns of some
Hankel matrix [l]. I n a practical situation, even if the
Hankel matrix is theoretically given! the rounding errors
will always make the matrix of full rank. If the matrix is
obtained from a record of obserrat.ions of a real object the
sampling variabilities of the elements of the matrix nil1 be
by far the greater than the rounding errors and also the
system n-ill always be infinite dimensional. Thus it can be
seen that the subject of statistical identification is essen-
tially concerned with the art of approximation n-hich is a
basic element of human intellectual activity.
As was noticed by Lehman 12, p. viii], hypothesis
t,esting procedures arc traditionally applied to the situ-
ations where actually multiple decision procedures are
3Iannscript received February 12, 1974; revised I I arch 2, 1974.
The author is with the Institute of Statistical Mathen1atie,
AIinato-ku, Tokyo, J apan.
required. If the statistical identification procedure is con-
sidered as a decision procedure the very basic problem is
the appropriate choice of t,he loss function. I n the Se y -
man-Pearson theory of stat.istica1 hypothesis testing only
the probabilit.ies of rejecting and accepting the correct
and incorrect hypotheses, respectively, are considered to
define the loss caused by the decision. I n practical situ-
ations the assumed null hypotheses are only approxima-
tions and they are almost ah-ays different from the
reality. Thus the choice of the loss function in the test.
theory makes its practical application logically contra-
dictory. The rwognit,ion of this point that the hypothesis
testing procedure is not adequa.tely formulated as a pro-
cedure of approximation is very important for the de-
velopment of pracbically useful identification procedures.
A nen- perspective of the problem of identification is
obtained by the analysis of t,he very practical and success-
ful method of maximum likelihood. The fact. t.hat the
maximum likelihood estimates are. under certain regu-
larity conditions, asymptot.ically efficient shom that the
likelihood function tends to bc a quantity which is most.
sensitive to the small variations of the parameters around
the true values. This observation suggests thc use of
S(g;f(. ! e) ) =J g ( . ~ ) hgf(.@j d . ~
as a criterion of fit of a model with thc probabilist.ic.
structure defined by the probability density function
j (@) to the structure defined hy the density function g( x) .
Contrary to the assumption of a single family of density
f ( x 0) in the classical maximum likelihood estimation
procedure, several alternative models or families defined
by the densities n-ith different forms and/or with one and
the same form but with different restrictions on the
parameter vector e arc contemplated in the usual situation
of ident.ification. A detailed analysis of the maximum
likelihood estimate (MLE) leads naturally to a definition
of a nen- estimate x\-hich is useful for this type of multiple
model situation. The new estimate is called the minimum
information theoretic criterion (AIC) estimate (IIAICE),
where -$IC stands for an information theoretic criterion
recently introduced by the present author [3] and is an
estimate of a measurp of fit of the model. XI I CE is de-
fined by the model and its parameter valucs which give the
minimum of AI C. B - the introduction of I LUCE the
problem of statistical identification is explicitly formulated
as a problem of estimation and the need of the subjective
judgement required in the hypothesis testing procedure
for the decision on the levels of significance is completely
eliminated. To give an explicit definition of IIdICE and to
discuss its characteristics by comparison with the con-
ventional identification procedure based on estimation
Authorized licensed use limited to: University of North Texas. Downloaded on April 02,2010 at 10:56:42 EDT from IEEE Xplore. Restrictions apply.
AEAIKE: STATISTICAL MODEL IDENTIFICATION 717
and hypothesis testing form the main objectives of the
present pa.per.
Although MAICE provides a versatile method of
identification which can be used in every field of st.atist,ica.l
model building, its pract.ica1 utility in time series analysis
is quit,e significant. Some numerical examples are given to
show how MAICE ca.n give objectively defined ansm-ers
to the problems of time series analysis in contrast with t.he
conventional approach by hypothesis testing which can
only give subject.ive and often inconclusive answers.
11. HYPOTHESIS TESTING IN TIME SERIES ANALYSIS
The study of t,he t,esting procedure of time series start,ed
with t.he investigation of the test. of a. simple hypot,hesis
t,hat a. single serial correlation coefficient. is equal t.o 0.
The utilit,y of this t.ype of t.est, is certa,inly t.oo limit,ed to
make it a generally useful procedure for model identifica-
tion. I n 1%7 Quenouille 143 int.roduced a test for the
goodness of fit of a.utoregressive (AR) models. The idea of
the Quenouilles test was extended by Wold [5] to a test of
goodness of fit of moving average (31-4) models. Several
refinements and generalizations of these test. procedures
followed [GI-[9] but a most, significant contribut,ion to trhe
subject, of hypothesis testing in time series analysis was
made by Whittle [lo], [ l l ] by a systematic application of
the 7Veyma.n-Pearson likelihood ratio t.est. procedure to
t.he time series situation.
A very basic test of t,ime series is the test, of whiteness.
I n many situations of model identification the whiteness
of the residual series after fitt,ing a model is required as a
proof of adequacy of the model and the test of whitreness
is widely used in practical applications [12]-[15]. For the
test of whiteness the analysis of t,he periodgram provides a
general solution.
,4 good exposition of the classical hypothesis testing
procedures including the test.s based on the periodgrams is
given in Hannan [16].
The fitting of AR or RIA models is essentially a. subject
of multiple decision procedure rather than t,hat, of hy-
pothesis testing. Anderson [17] discussed the determination
of the order of a Gaussian AR process explicit,ly as a
multiple decision procedure. The procedure takes a form
of a sequence of tests of the models starting a.t the highest
order and successively down t.0 the lowest. order. To
apply t:he procedure t.0 a real problem one has to specify the
level of signifimnce of the t.est for each order of the model.
Although t.he procedure is designed to satisfy certain
clearly defined condition of optimalitg, the essential
difficulty of the problem of order determination remains
as the difficulty in choosing the levels of significance.
Also the loss function of t.he decision procedure is defined
by the probability of making incorrect, decisions and thus
the procedure is not free from the 1ogica.l cont.radict.ion
that in practical applicat,ions the order of the true struc-
t.ure will always be infinite. This difficulty can only be
avoided b- reformulat,ing t.he problem explicitly as a
problem of approximation of the true st,ructure by the
model.
111. DIRECT APPROACH TO MODEL ERROR CONTROL
I n the field of nont,ime series regression analysis Mal-
lows introduced a statist.ic C, for the selection of variables
for regression 1181. C, is defined by
6, =(&*)-I (residua.1 sum of squares) - N + 2p,
where z2 is a properly chosen estimat.e of u2, t,he unknown
mriance of t,he true residual, N is t,he number of observa-
tions, and p is the number of variables in regression. The
expected value of C, is roughly p if the fitt,ed model is
exact, and greater ot,hemise. C, is an estimate of the
expected sum of squares of the prediction, scaled by u*,
when the estimated regression coefficient.s are used for
prediction and has a. clearly defined meaning as a measure
of adequacy of the adopted model. Defined with this
clearly defined criterion of fit, C, attract.ed serious atten-
tion of the people who were concerned with the <<ion
analyses of practical data. See t,he references of [ H I . Un-
fortunately some subjective judgement is required for the
choice of 62 in the definition of C, .
At almost the same t.ime when C, was introduced,
Da.visson [19] analyzed the mean-square prediction error
of st,ationary Gaussia.n process when the est.imat.ed co-
ef5cient.s of the predict.or were used for prediction and
discussed the mean-square error of an adaptive smoot.hing
filter [20]. The observed t.ime series xi is the sum of signal
si and additive whit,e noise ni. The filtered output, Zi is
given by
L
B* = pj xi +j , (i =1,2,.. . J7)
j =- Si
where p j is determined from t.he sample xi (i =1,2,. . . , X) .
The probiem is how t,o define L and M so that the mean-
square smoothing error over the N samples E[ ( l / N)
cy=l (si - a,)] is minimized. Under appropriate assump-
tions of si and n i Davisson [20] a.rrived at an estin1at.e of
this error which is defined by
&x L, I - W, L] =s2 + 2?(M + L + l ) / N,
where s* is an estimate of the error variance and E is the
slope of the curve of s2 as a funct.ion of ( X + L)/A7 at
1a.rger values of ( L + X ) / N . This result is in close
correspondence with Mallows C,, and suggests the im-
portance of t.his type of statistics in the field of model
identificat,ion for prediction. Like t.he choice of k2 in
Mallows C, the choice of E in the present st.atistic 6.v2 [ U,
L] becomes a. diffcult problcm in pract.ical application.
In 1969, without knowing t,he close relat,ionship with
the above two procedures, the present author introduced
a fitting procedure of the unirariate AR. model defined by
[21]. I n this procedure t.he mean-square error of the one-
stepahea.d prediction obt.ained by using the least squares
estimates of t.hc coefficients is controlled. The mean-
square error is called t,he final prediction error (FPE) and
when the data ; y i (i =1,2; . . , X) are given its estima.t,e is
Yi =alyi-1 + . . . + aPyi-, + xi, where xi is a white noise
Authorized licensed use limited to: University of North Texas. Downloaded on April 02,2010 at 10:56:42 EDT from IEEE Xplore. Restrictions apply.
defined by
FPECP) =I <n: + P ) / W - d l
. (eo - 6 e - ... - 6 C )
P l 1 PP P ?
where the mean of yi is assumed to be 0, PI =(l!X)x>=;'
yi+!yi and are obtained by solving the Tule-Walker
equation defined by cr?s. By scanning p successively from
0 to some upper limit L the identified modcl is given by
the p and the corresponding Bpi's n-hich give the nlinimum
of FPE(p) ( p =O.l;..,L). I n this procedure no sub-
jective element is left in the definition of FPE(p). Only
the determination of the upper limit L requires judgerncnt.
The characteristics of the procedure n-as further analyzed
[ E ] and the procedure worked remarlcablp 1 ~1 1 with
practical data [33], 1341. Gersch and Sharp [25] discussed
their experience of the use of the procedure. Bhansali [?GI
reports vcrp disappointing results, claiming that they
were obtained b- dkailre's method. Actually the dis-
appointing results are due to his incorrect definition of
the related statistic and have nothing to do with the
present minimum FPE procedure. The procedure was
extended to the case of multivariate AR nlodcl fitting
[ X ] . A successful result of implementation of a computer
control of cement kiln processes based on the results
obtained by this identification procedure was reported bJ -
Otomo and others [as].
One common characteristic of the three procedures
discussed in this section is that the analvsis of the sta-
tistics has to be extended to the order of 1 / S of the main
t,erm.
IV. IIEAK LOGLIKELIHOOD AS A ~\IEASCRE OF FIT
The well known fact that the MLE is, under regularity
conditions, asymptotically efficient [29] shows that the
likelihood function tends to be a most sensitive criterion of
the deviation of the model parameters from the true values.
Consider the situation where ~ 1 . x ~ . . . . ,x.\- are obtained
as the rcsults of 12: independent observations of a random
variable with probability density function g(r). I f n
parametric famil- of density function is given by f(.&)
with a vector parameter 0. the average log-likelihood. or
the log-likelihood divided by X , is given by
..
(1/1V) log f(Zj18), (1)
I = 1
where. as in the sequel of the present paper, log denotes the
natural logarithms. As X is increased indefinitely, this
average tends, with probabilit- 1. to
s(g;f(+)) =&(x) logf(.+) d.r,
where the esistence of the integral is assumed. From the
efficiency of I I LE it can be seen that the (average) mean
log-likelihood S(g;f(. ! e) ) must be a most sensitive criterion
to the small deviation of f(x)8) from g(.r). The difference
k f ( - l e ) ) =s(g;g) - s(g;f(.Io))
is kno-m as the Iiullback-Leibler mean information for
discrimination between g( x) and f ( . l e ) and takes positive
value, unless f (~i 8) =g(r) holds alnlost everyhere [30].
These observations show that S(g;f(.le)) n-ill be a reason-
able criterion for defining a best fitting model by its
maximization or, from the analogy to the concept of
entropy? by minimizing -S(g:f(.18)). I t should be men-
tioned here that in 1950 this last quantity was adopted as a
definition of information function b?. Bartlctt [31]. One
of the most important characteristics of S(g;f(. l e) ) is that
its natural estimate, the average log-likelihood (1). can be
obtained without the knowledge of ~(s). Xhen only one
family f(s(0) is given. maximizing the cstinlatc (1) of
S(g;f(. : e) ) n-ith respect to 0 leads to the NLE 4.
In the case of statistical idrntification. usually several
families of f(s,e), with different forms off(.+) and/or with
one and the samc form off(.+) but with different restric-
tions on the paranwtcr vector 8. are given and it is re-
quired to decide on the best choice of j ( s i 0 ) . Thc classical
nlaxinlunl likelihood principle can not provide useful
solution to this type of problems. -4 solution can be
obtained by incorporating thc basic idea underlying the
statistics discussed in the prcceding section with the
masimum likelihood principle.
Considcr the situation where ~( s ) =j(r'8,). For this
case I ( g: f ( - , e) ) and S( q; f ( . 10)) will simply be dcnotcd by
l ( eo: e) and S(e,;e), respectively. When e is sufficiently
close to eo, I(8o;O) admit5 an approximation [SO]
I(Oo;e, + A8) =( + ) l , A O / i , ? :
=-here 1!A8;1J2 =Ae'JAf3 and J is the Fisher information
matrix which is positive definite and defined by
where J i j denotes the (i.j)th element of J and ei the ith
component of 8. Thus n-hen the J I LE 4 of eo lies very
close to eo the deviation of the distribution defined by
f(xl8) from the true distribution j(. rbo) in terms of the
variation of S(c/:f(. le)) will be measured by (f) I e - e&?.
Consider the situation n-here the variation of 8 for maxi-
mizing the lildihood is restricted to a lower dimensional
subspace e of 0 v,-hich does not include 0,. For thc MLE 6
of eo restricted in 8: if 6 which is in e and gives the
maximum of S(e,;e) is sufficiently close to eo. it can be
shon-n that the distribution of X! ; d - 6/ i Je for sufficiently
large l \ 7 is approsinlated under certain regularity conditions
by n chi-square distribution thc degree of freedom equal
to thc dimension of the restricted parameter space. See,
for example. [32]. Thus it holds that
E-3.\71(eo;4) = - eo ~f + X.,
(2)
where E, denotes the mean of the approsinlate distribu-
tion and X. is the dimension of 8 or the number of param-
eters independently adjusted for thc masimization of the
likelihood. Relation (2) is a generalization of the expected
prediction error underlping the statistic5 discussed in the
preceding section. When there are several models it will
Authorized licensed use limited to: University of North Texas. Downloaded on April 02,2010 at 10:56:42 EDT from IEEE Xplore. Restrictions apply.
. 4 l & i ~ ~ : STATISTICAL MODEL I DENTI FI CATI ON 719
be natural to adopt, t,he one which will give the ninimum
of EI(Oo;8). For this purpose, considering the situation
where t,hese models have t,heir 8s very close to Bo, it. be-
comes necessa,ry to develop some estimate of NllO -
Oo11J2 of (2). The relation (2) is based on the fact. t,hat the
asymptotic distribut,ion of dhT(8 - 8) is approxima,t,ed
by a Gaussian distribution with mean zero and variance
mat,rix J-l. From tahk fact. if
is used as an estimate of RTl18 - OO/lJ2 it needs a correctmion
for the downward bias int.roduced by replacing 8 by 8.
This correct,ion is simply realized by adding k to (3).
For the purpose of ident.ification only t.he comparison of
the values of t,he estimates of EI(Oo;8) for various models
is necessary and thus the conlmon term in (3) which
includes eo can be discarded.
v. DEFINITION OF AN I hTORMATI OS CRITERION
Based on the observations of the preceding sect,ion an
informat,ion criterion AIC of 13is defined by
AIC(8) =(-2) log (maximum likelihood) + 2 ,
where, as is defined before, k is t.he number of indepen-
dent.ly adjusted parameters to get. 8. (l/N)AIC(8) may be
considered as a.n estimate of -2ES(Oo;8). I C st,ands for
infornlation criterion and A is added so that similar &a-
tistics, BIC, DIC etc., ma>- follow. When there are several
specificat.ions of f(z)O) corresponding to several models, the
MAICE is defined by the f(z(8) which gives the minimum
of AIC(8). When there is only one unrestricted family of
f(zlO), the J l dI CE is defined by f(z(8) with 8 identical to
the classical MLE. It should be not,iced t.hat an arbitrary
a.ddit,ive constant can be int,roduced int.0 t.he definition of
AIC(8) when the comparison of the results for different
sets of observations is not intended. The present, definition
.of MAI CE gives a mat,hematical formulat,ion of the prin-
ciple of parsimony in model building. When the maxinlum
likelihood is identical for two models t.he MAICE, is t,he
one defined witlth the smaller number of parameters.
I n t,ime series analysis, even under the Gaussia.n assump-
tion, the exact definition of likelihood is usually too com-
plicated for practical use and some approximation is
necessary. For the applicat.ion of J I AI CE t.here is a
subtle problem in defining the approximat.ion to the likeli-
hood funct.ion. This is due to the fact that. for the definit,ion
of AIC the log-likelihoods must. be defined consistent.ly to
the order of magnit,ude of 1. For the fitting of a shtionary
Gaussia.n process model a. measure of the deviation of a
model from a true structure can be defined as the limit of
t.he average mean log-likelihood when the number of
obserrat,ions A T is increased indefinitely. This quantity is
ident.ica1 to the mean log-likelihood of innovation defined
by the fitted model. Thus a natural procedure for t,he
fitting of a st.ationary zero-mean Gaussian process model
to t.he sequence of observations yl,y?,. . . , yay is t.o define a.
primitive sta.t,ionarg Gaussian model with t.he Z-lag CO-
variance matrices R@), which a.re defined by
and fit a model by ma.ximizing t,he mean log-likelihood of
innovation or equivalently, if the elements of t,he co-
variance matrix of innovation a.re within the paramet,er
set, by minimizing the log-det,erminant of the variance
matrix of innovation, hT times of which is to be used in
place of the log-likelihood in the definit,ion of AIC. The
adoption of the divisor N in the definition of R(1) is im-
port.ant t.o keep t.he sequence of the covariance matrices
positive definite. The present procedure of fitting a
Gaussian model t.hrough the primitive model is discussed
in detail i n [33]. It leads naturally to t,he concept of
Gaussian estimate developed by 1Vhitt.le [31]. When the
asympt.ot.ic distribution of t.he norma.lized correlation
coefficient,s of yn is identical to that of a Gaussian process
the asympt,otic dist,ribution of the stat.istics defined as
funct.ions of these coefficie11t.s will a.lso be independent of
the assumption of Gaussia.n process. This point and the
asymptotic behavior of the related statist.ics n:hich is re-
quired for the justification of the present, definition of AIC
is discussed in det<ail in the above paper by Whittle.
For the fitt.ing of a univariat.e Gaussian AR model t,he
MAICE defined with t,he present definit.ion of AIC is
asympt.otically identical t.0 t,he est.imat.e obtainrd by the
minimum FPE procedure.
AIC and a primitive definition of AIA1C.E were first
introduced by t,he present, author in 1971 [3]. Some early
successful results of applications are reported in [ 3] , 13.51,
[36 1.
VI. NUMERICAL EXAMPLES
Before going into t,he discussion of t,he characteristics
of MAICE it,s practical utilit,y is demonst,rated in this
section.
For the convenience of t,he readers who might wish to
check the results by t,hemelves Gaussian AR models were
fitted t,o t.he data given in Andersons book on t.ime series
analysis [37]. To t.he 1+olds three series artificially gen-
erat.ed by the second-order AR schemes models up t,o t.he
50th order were fitted. I n two cases t.he MAICES mere
the second-order models. I n the case where the XAICE
was the first-order model, the second-order coefficient of
the generating equation had a very small absolute value
compared with its sampling variability and the one-step-
ahead prediction error variance was smaller for the
MAICE than for t.he second-order model defined tvith the
MLEs of t,he coefficients. To t.he classical serirs of Wolfers
sunspot numbers with N =176 AR models up to t,he
35th order were fitted and the MAICE was the eight.h-
order model. AIC att.ained a local minimum at. the second
order. I n the case of the series of Beveridges wheat price
index wit.h N =370 the MAICE among t.he AR model up
to the 50th order was again of the eight,h order. XIC
Authorized licensed use limited to: University of North Texas. Downloaded on April 02,2010 at 10:56:42 EDT from IEEE Xplore. Restrictions apply.
720 IEEE TRAXSACTIONS ON AUM?+MTIC CONTROL, DECEYBER 1974
attained a local minimum at the second order which was
adopted by Sargan [38]. I n the light of the discussions of
these series by Anderson, t,he choice of eight-order models
for these two series looks reasonable.
Two examples of applicat.ion of the minimum FPE pro-
cedure, which produces est.imates asymptotically equiva-
lent, to I\IAICEa, are report.ed in [3]. I n tJ he example
taken from the book by J enkins and Watts 139, sec.tion
5.4.31 the estimate was identical to the one chosen by t,he
authors of the book after a careful analysis. I n t>he case of
the seiche record treat.ed by Whit.t.le [40] t.he minimum
FPE procedure clearly suggested t.he need of a very high-
order AIR model. The difficu1t.y of fitting AR models to
this set of data was discussed by Whittle [41, p. 381.
The procedure was also applied to the series E and F
given in the book by Box and J enkins [E]. Second- or
third-order AR model was suggested by the authors for the
series E which is a part of the Wolfers sunspot number
series with A 7 =100. The MAICE among the AR models
up to the 20th order was the second-order model. Among
the AR models up to t.he 10th order fitt,ed to the series F
with S =70the I\LUCE was the second-order model,
m-hich agrees with the suggestion made by the authors of
t,he book.
To test the ability of discriminating between AR and
MA models ten series of yn ( n =1,. . . ,1600) were gener-
ated by the relation y, =x, + O.~X,-~ - O.lx,-?, where
x, was generated from a physica.1 noise source and was
supposed to be a Gaussian white noise. AR models were
fitted to the first W points of each series for N =50, 100,
200, 400, 800: 1600. The sample averages of the J I AI CE
AR order were 3.1, 4.1, 6.5, 6.8, 8.2, and 9.3 for the SUC-
cessirely increasing values of N . An approximate J I AI CE
procedure which is designed to get. an initial est.imate of
1\IA41CE for the fitting of 3Iarkovian models, described in
[33]. nas applied to the data. With only a few exceptions
t.he approximate IIAICEs were of the second order. This
corresponds to the AR-MA model with a second-order
AR and a first-order M A . The second- and third-order AI A
models xere then fitted to the dat.a with h: = 1600.
Among the AR and MA models fitted to the dat,a the
second-order X I model mas chosen nine times as the
3L4ICE and the t.hird-order J1A was chosen once. The
average difference of the minimum of llIC between AH.
and JI-4 models was 7.7, which roughly mea.ns that the
expected likelihood ratio of a pair of t,wo fitt.ed models
will be about 47 for a set of data with N =1600 in favor of
M A model.
Another test was made with the example discussed by
Gersch and Sharp [%I. Eight series of length K =800
were generated by an AR-MA scheme described in the
paper. The average of the AIAICE AR orders was 17.9
which is in good agreement n-ith the value reported b?-
Gersch and Sharp. The approximate J IAICE procedure
was applied to determine the order or t.he dimension of the
Markovian representation of the process. For the eight,
cases the procedure identically picked the correct order
four. AR-X4 models of various orders were fitted to one
set of dah and t.he corresponding values of AIC(p,q) were
computed, where AIC(p,q) is the value of .$IC for the
model with AR order p and I I A order p and was defined by
AIC(p,q) =N log (3ILE of innovation variance)
+ X P + $9.
The results are AIC(3,2) =192.72, AIC(1,3) =66.54,
and AIC(5,4) = 69.43. The minimum is attained at
p =4 and q =3 which correspond to the true structure.
Fig. 1 illustrates the estimates of the power spectral
demit.y obtained by applying various procedures to this
set of data. It should be mentioned that, in this example
the Hessian of the mean log-likelihood function becomes
singular at the true values of the parameters for the
models with p and q simultaneously greater than 4 and 3,
respectively. The detailed discussion of the difficulty
connected with this singularity is beyond the scope of the
present paper. Fig. 2 shows the results of application of
the same type of procedure to a record of brain wave with
N =1120. I n this case only one AR-JIA model with AR
order 4 and ITA order 3 was fitted. The value of AIC of
this model is 1145.6 and that of the 1IAICE AR model is
1120.9. This suggests that, the 13th order MAICE AR
model is a better choice, a conclusion which seems in
good agreement. with the impression obtained from the
inspection of Fig. 2.
AIC(4>4) =67.44, AIC(5,3) =67.18 ;2IC(6,3) =67.65,
VII. DISCCSSIOXS
When f(al0) is very far from g(x), S(g;f(. je)) is only a
subjective measure of deviation of f(.r(e) from g(r). Thus
the general discussion of the characteristics of 3IAICE
nil1 only be possible under the assumption that for at
least one family f(&) is sufficiently closed to g(r) com-
pared with the expected deviation of f(sJ6) from f(rI0).
The detailed analysis of thc statistical characteristics of
XXI CE is only necessary when there are several families
which sa.tisfy this condition. As a single estimate of
-2A7ES(g;f(.)8)), -2 times the log-mitximum likelihood
will be sufficient but for the present purpose of estimating
the difference of -3XES(g;f(. 18)) the introduction of the
term +2k into the definition of AIC is crucial. The dis-
appointing results reported by Bhansali [%I were due to
his incorrect use of the statistic. equivalent to using +X.
in place of +?X- in AIC.
When the models are specified by a successive increase
of restrictions on the parameter e of f(rl0) the AIAICE
procedure takes a form of repeated applications of con-
ventional log-likelihood ratio test of goodness of fit with
automatically adjusted lcvels of significance defined by
the terms +4k. Nhen there are different families approxi-
mating the true likelihood equally well the situation will
at least locally be approximated b?- the different para-
metrizations of one and the same family. For these cases
the significance of the diffelence of AICs bet\\-cen two
models will be evaluated by comparing it u-ith the vari-
abi1it.y of a chi-square variable with the degree of freedom
Authorized licensed use limited to: University of North Texas. Downloaded on April 02,2010 at 10:56:42 EDT from IEEE Xplore. Restrictions apply.
AKAIEE : STA"IS"IC.AL NODEL IDENTIFICATION
Fig. 1. Estimates of an AR-MA spectrum: theoretical spectrum (solid t.hin line with dots), AR-MA estimate (thick
line), AR estimate (solid thin line), and Hanning windowed estimate with maximum lag 80 (crosses).
'iw 5.00 10.00 15.00
Fig. 2. &timates of brain wave spectrum: =-MA est.imate (thickline), AR estimat,e (solid thin line), and Hanning
windowed estimate with maximum lag 150 (crosses).
721
Authorized licensed use limited to: University of North Texas. Downloaded on April 02,2010 at 10:56:42 EDT from IEEE Xplore. Restrictions apply.
722 IEEE TR-I NSACTI ONS ON AUTOMATI C COSTROL, DECEMBER 1974
equal to the difference of the ks of the two models. When
the two models form separate families in t.he sense of Cos
[ E ] ! [43] the procedure developed by Cos and extended
by Walker [a] to time series situation may be useful for
the detailed evaluation of the difference of XIC.
I t must be clearly recognized that I\IXICE can not be
compared with a. hypothesis testing procedure unless the
latter is defined as a decision procedure with required
levels of significance. The use of a Lxed level of Significance
for the comparison of models with various number of
parameters is wrong since this dors not takc into account
the increase of the variability of the estimates \dm1 thc
number of parameters is increased. As will be seen bp the
work of Kennedy and Bancroft [45] the theory of model
building based on a sequence of significance tests is not
sufficient.ly developed to provide a practically useful
procedure.
Although the present author has no proof of optimalit>-
of MAICE it is at present. the only procedure applicable to
every situation where the likelihood can be properly
defined and it is actually producing very reasonable results
n-ithout very muchamount of help of subjective judgement.
The successful results of numerical experiments suggest
almost unlimited applicabilitv of MAICE in the fields of
modeling, prediction, signal detection, pattern recognition.
and adaptation. Further improvements of definition and
use of .AIC and numerical comparisons of J LUCE with
other procedures in various specific applications will be
the subjects of further stud>-.
VIII. COWLUSION
The practical utility of the hypothesis testing procedure
as a method of statistical model building or identification
must be considered quite limited. To develop useful
procedures of identification more direct approach to the
control of the error or loss caused by the use of the identi-
fied model is necessary. From the success of thc classical
maximum likelihood procedures t he mean log-likelihood
seems to be a natural choice as the criterion of fit of a
statistical model. The XUCE procedure based on -4IC
which is an estimate of the mean log-likelihood proyides a
versatile procedure for the statistical model identification.
It also provides a mathematical formulation of the prin-
ciple of parsimony in the field of model construction.
Since a procedure based on AIAICE can be implemented
without the aid of subjective judgement, thc successful
numerical results of applications suggest that the implc-
mentations of many statistical identification proccdurcs
for prediction, signal detection? pattern recognition, and
adaptation will be made practical 11-ith AIBICE.
ACKSOTTLEDGNEST
The author is grateful to Prof. T. Kailath, Stanford
University: for encouraging him to mi te the present
paper. Thanks are also duc to Prof. I<. Sato. Sagasaki
University, for providing the brain wave data treated in
Section V.
REFERESCES
[l ] H. Aikaike, Stochastic theory of minimal rea.lization, this
[ 2] E. L. Lehntan, Tcstitrg Statistical Hypothesis. Ken- York:
131 H. .lkaike. Inforn~stion theory and an extension of the maxi-
iswe, pp. 667-6i4.
IViley, 1959.
[4] 11. H. Qaenouille. A large-sample test for the goodness of fit
of antoregressive schemes, J . Xo!y. Statist. Soc., vol. 110, pp.
12:3-120, 1947.
1.51 H. JVold. -4 Inrae-snmole test for moving averages. J . Rou.
. .
Statist. soc., H , &I . I I , pp. 297-30>. 1949. I
[6] 31. P. Barlert and P. 1. I)ianandn, I<stensionz of Quenouilles
test for autoregressive scheme, J . f?oy. Sfati d. Soc., H , vol.
12, pp. 10S-115! 19.50.
[ i ) 31. S. Rnrtlett and I ). X-. llajalabrhmsn. Goodness of fit test
for sirnnltxneous autoregrewive series, J . Roy. Stati st. Soc., B,
[SI -4. 11. \Vnlker? Sote on a generalization of the large aanlple
vol . 15, pp. 10-124, I9.X.
goodness of fit test for linear antoregresive scheme, J . Roy.
St af i sf . Soc.. IZ, 1-01. 12, pp. 102-107, 1950.
191 -. The existence of Bnrtl ett-l (ni al ak~hnl n~~ eoodnesa of fit
. .
Authorized licensed use limited to: University of North Texas. Downloaded on April 02,2010 at 10:56:42 EDT from IEEE Xplore. Restrictions apply.
[33] H. Akaike, Markovian representation of stochastic processes
and its application to t.he analysis of autoregressive moving
[34] P. Whittle, Gawian est.imation in stationary time series,
average processes, Ann. Inst. Statist. Xath., t.o be published.
[33] H. Akaike, Use of an information theoretic quantity for
Bull. Int. Statist. I,&., vol. 39, pp. 105-129, 1962.
st.at,istical model identification, in Proc. 5tji Ha.waii Ink. c072j.
System Sciences, pp. 249-230, 1972.
[36] H. Akaike, Aut.omat.ic data structure search by the maximum
likelihood, in Computer i n Biomedicim? Suppl. to Proc. 5th
[37] T. W. Anderson, The Statistical Analysis of Ti me Series.
Hawaii Int. Conf. on. SystCmz Scienas, pp. 99-101, 1972.
[38] J . D. Sargan, An approximate treatment of t.he propert.ies of
New York: Wiley, 1971.
the correlomam and oeriodwam, J. Roy. Statist. SOC. B, vol.
- .
[39] G. h.1. J enkins and D. G. Watts, Spectral A d y s i s and its
[40] P. Whittle, The statistical analysis of a seiche record, J .
15, pp. 140-152,1953:
San Francisco, Calif.: Holden-Day, 1968.
Marine Res., vol. 13, pp. 76-100, 1954.
[41] P.. Whittle, Prediction. a.n.d Regulation. London, England:
English Univ. Pres, 1963.
[42] D. R. Cos, Tests of separate families of hypothses, in Proc.
4th Berkeley Symp. Mathernatical Statistics a.nd Probability, vol.
1. 1961. VD. 105-123.
[43] ~. R. CGx, Further results on tests of separate families of
hvpotheses, J. Eoy. Statist. SOC., I?, vol. 24, pp. 4064. 5, . 1962.
[44] A.M . Walker, Some test.s of separate families of hypotheses in
time series analysis, Biometriku, vol. 34, pp. 39-68, 1967.
[45] W. J. Kennedy and T. A. Bancroft, Model building for pre-
An.?%. Math. Statist., vol. 12, pp. 1273-1284, 1971.
diction in regrai on based upon repeated significance tests,
Hiortugu Akaike (XI%), for a photograph and biography see page
674 of this issue.
Some Recent Advances in Time Series Modeling
EMANUEL PARZEN
Absfract-The ai mof this paper is to describe some of the impor-
tant concepts and techniques which seem to help provide a solution
of the stationary time series problem (prediction and model iden-
acati on). Section 1 reviews models. Section Il reviews predic-
tion theory and develops criteria of closeaess of a fitted model to
a true model. The cential role of the infinite autoregressive trans-
fer function g, is developed, and the time series modeling problem
is defined t o be the estimation of 9,. Section In reviews estimation
theory. Section IV describes autoregressive estimators of 9,.
I t introduces a ciiterion for selecting the order of an autoregressive
estimator which can be regarded as determining the order of an
AR scheme when in fact the time series is generated by an AR
scheme of hi te order.
T
I . INTRODUCTION
HE a.im of this paper is t.0 describe some of the im-
portant. concepts and techniques which seem to me
to help provide realistic models for the processes generating
observed time series.
Section I1 reviexi-s t,he types of models (model concep-
t>ions) which statisticians have developed for time series
analysis and indicates the value of signal plus noise de-
compositions as compared x7it.h simply an autoregressive-
moving average (ARMA) represent,ation.
Section I11 reviem prediction theory and develops
criteria. of closeness of a fitted model t.0 a %we model.
The central role of the infinit.e autoregressive transfer
function y, is developed, and the t>ime series modeling
problem is defined to be t,he est.imat.ion of y, .
Section I11 review the estimation theory of autore-
This work was supported in paft by the Officepf .Naval Research.
Universit.y of Ke-s York, Buffalo, N.Y.
Manuscript, received J anuary 19, 1974; revised May 2, 1974.
The author is wit.h the Dlvision of Statlstlcal Science, State
gressive (AR) schemes and
equat,ions. It develops an
the basic role of Yule-Wa.lker
anaIogous t,heory for moving
average (ALA) schemes, based on the duality betu-eenf(w),
the spect,ral density and inverse-spectral density, and
R(v) and Ri ( v ) ! the cova.riance and covarinverse. The
estimation of Ri(v) is shown t.0 be a consequence of t.he
estimation of y,.
Section V describes autoregressive estimators of y, . It.
introduccs a. criterion for selecting the order of an auto-
regressive estimator which can be regarded a.s determining
t.he order of an AR scheme when in fact t.he time series is
generat,ed by an AR scheme of finite order.
11. TINE SERIES MODELS
Given observed data, st.atistics is concerned with in-
ference from what zms observed to what might have beet1
observed; More precisely, one postulates a proba.bi1it.y
model for the process genemting t.he data. in n-hich some
parameters are unknown and are to be inferred from the
data. Stat.istics is t,hen concerned v-it.h parameter inference
or parameter identifica,tion (determinat,ion of parameter
values by estimation and hypothesis test.ing procedures).
A model for data is called structural if its paramekrs
have a natural or structural interpretation; such models
provide expl anat i on and control of t,he process generating
the da,ta.
When no models are available for a. data set from theory
or experience, it is st,ill possible to fit models u-hich suflice
for si mul at i on (from what has been observed, generate
more data similar t.o that observed), predictio??. (from what.
has been observed, forecast t.he data that will be observed),
and pa.ttern recognition (from what has been observed, infer
Authorized licensed use limited to: University of North Texas. Downloaded on April 02,2010 at 10:56:42 EDT from IEEE Xplore. Restrictions apply.