✬ ✩

STAT 520 Linear Stationary and Nonstationary Models 1

General Linear Process

Consider a general linear process of the form

P∞ P∞
zt = at + j=1 ψj at−j = (1 + j=1 ψj B j )at = ψ(B)at ,
where at is a white noise process with var[at ] = σa2 , B is the
backward shift operator, Bzt = zt−1 , B j zt = zt−j , and ψ(B) is
called the transfer function. Alternatively, one may write
P∞
(1 − j=1 πj B j )zt = π(B)zt = at ,
where the current value of zt is “regressed” on the past values zt−j .
It is easily seen that π(B)ψ(B) = 1.

It is known that any zero-mean stationary Gaussian process can be

P∞
written in the MA form zt = ψ(B)at with j=1 |ψj | < ∞.

The transfer function ψ(B) defines a linear filter that transforms

✫ ✪
P∞
the input at to output zt . The filter is stable with j=1 |ψj | < ∞.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 2

Autocovariance and Spectrum

Set ψ0 = 1 and ψh = 0, h < 0. It is easy to calculate

P∞ P∞
γk = σa j=0 ψj ψj+k . Write γ(B) = k=−∞ γk B k as the
2

autocovariance generating function. It follows that

γ(B) = σa k j ψj ψj+k B k
2
P P

2 −j j+k 2 −1
P P
= σ a j ψj B k ψ j+k B = σ a ψ(B )ψ(B),

where B −1 = F is the forward shift operator.

Recall the definition of the power spectrum,

P∞
p(ω) = k=−∞ γk e−i2πkω .
Substituting e−i2πω for B in ψ(B), one has
p(ω) = σa2 ψ(ei2πω )ψ(e−i2πω ) = σa2 |ψ(e−i2πω )|2 .
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 3

Stationarity and Invertibility

For the linear process zt = ψ(B)at to be a valid stationary process,

−i2πω
P∞
ψ(e ) = j=0 ψj e−i2πωj must be convergent, i.e., ψ(B) be
P∞
convergent for |B| ≤ 1. It suffices to have j=0 |ψj | < ∞.

A process is invertible if π(B) is convergent for |B| ≤ 1. It suffices

P∞
to have j=0 |πj | < ∞. To illustrate the idea, consider the MA(1)
Pk j
process zt = (1 − θB)at . Since 0 x (1 − x) = 1 − xk+1 , one has
Pk
zt = − j=1 θj zt−j + at − θk+1 at−k−1 .
For |θ| < 1, one may let k → ∞ and “invert” the process into an
AR(∞) process, with πj = θj dying out as j → 0. For |θ| ≥ 1,
θk+1 at−k−1 6→ 0. Also note that ρ1 = −1/(θ + θ−1 ), so θ = b±1 are
not identifiable. Invertibility removes the ambiguity and assures

✫ ✪
practical sensibility.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 4

AR(p) Process: Stationarity

An autoregressive process of order p (i.e., AR(p)) is defined by

zt = φ1 zt−1 + · · · + φp zt−p + at ,
or (1 − φ1 B − · · · − φp B p )zt = φ(B)zt = at . The transfer function
is given by ψ(B) = φ−1 (B). AR(p) is invertible by definition.
Qp
Write φ(B) = j=1 (1 − Gj B), where G−1 j are the roots of
φ(B) = 0. One has (assuming distinctive roots),
Qp 1
Pp K
ψ(B) = j=1 1−Gj B = j=1 1−Gjj B ,
so one must have |Gi | < 1 for ψ(B) to be convergent for all |B| ≤ 1.
In other words, one needs the roots of φ(B) to lie outside of the
unit circle for zt = φ−1 (B)at to be stationary.

To get the roots of 1 + .6x + .5x2 , use polyroot(c(1,.6,.5)) in R.

✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 5

Examples: AR(1) and AR(2)

Stationarity condition
For AR(1), one needs |φ1 | < 1.
For AR(2), one needs |gj | < 1 in the expression
φ(B) = (1 − g1 B)(1 − g2 B) = 1 − (g1 + g2 )B − (−g1 g2 )B 2 .
With gj real, (φ1 , φ2 ) = (g1 + g2 , −g1 g2 ) over g1 , g2 ∈ (−1, 1). With
gj a conjugate pair Ae±i2πω , one has (φ1 , φ2 ) = (2A cos 2πω, −A2 )
over ω ∈ (−1/2, 1/2), A ∈ (0, 1).

Autocorrelation
For AR(1), ρk = φk1 , k ≥ 0.
For AR(2), ρk = φ1 ρk−1 + φ2 ρk−2 , k > 0; ρ0 = 1, ρ1 = φ1 /(1 − φ2 ).
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 6

Examples: AR(1) and AR(2)

Variance
For AR(1), γ0 = φ1 γ1 + σa2 , so γ0 = σa2 /(1 − φ1 ρ1 ) = σa2 /(1 − φ21 ).
For AR(2), γ0 = φ1 γ1 + φ2 γ2 + σa2 , so
σa2 1 − φ2 σa2
γ0 = = 2 2 .
1 − φ 1 ρ1 − φ 2 ρ2 1 + φ2 {(1 − φ2 ) − φ1 }
Power spectrum
For AR(1), p(ω) = σa2 /|1 − φ1 e−i2πω |2 = σa2 /(1 + φ21 − 2φ1 cos 2πω).
For AR(2),
σa2
p(ω) =
|1 − φ1 e−i2πω − φ2 e−i4πω |2
σa2
= .
1 + φ21 + φ22 − 2φ1 (1 − φ2 ) cos 2πω − 2φ2 cos 4πω
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 7

AR(p) Process: Yule-Walker Equations

Taking expectations of the expression,

zt−k zt = φ1 zt−k zt−1 + · · · + φp zt−k zt−p + zt−k at ,
one has, after dividing by γ0 ,
ρk = φ1 ρk−1 + · · · + φp ρk−p , k > 0.
Substituting k = 1, . . . , p, one obtains the Yule-Walker equations
    
1 ρ1 ρ2 ... ρp−1 φ1 ρ1
    
 ρ1 1 ρ1 ... ρp−2 
  φ 2   ρ2 
   
 .  =  . ,

 .. .. .. ..  .  .

 . . . ... . 

. .
ρp−1 ρp−2 ρp−3 ... 1 φp ρp

or Pp φ = ρp , and φ = P−1
p ρp expresses φj ’s in terms of ACF’s.
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 8

Partial Autocorrelation Function

Consider a Gaussian stationary process. The partial autocorrelation

function at lag k is defined by
αk = corr(zk , z0 |z1 , . . . , zk−1 ).
It can be shown that αk equals the kth element of φk = P−1k ρk ,
φkk . Replacing ρv by rv in the Yule-Walker equations, one gets the
sample PACF φ̂kk as the kth element of φ̂k = P̂−1
k ρ̂k .

For AR(p) processes at lag k > p, one has φkk = 0, and it can be
shown that, asymptotically, φ̂kk ∼ N (0, N1 ). Sample PACF’s are
available in R via acf with type="partial", or via pacf.

For non-Gaussian processes, one may still calculate φkk via ACFs
as diagnostics for AR(p) models, though they may no longer be

✫ ✪
perceived as conditional correlations.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 9

Recursive Yule-Walker Solutions and PACF

P ρ̃h
Let h = k − 1. Partition Pk = ( ρ̃Th 1
), where ρ̃h is ρh in reverse order,
h
and write d = 1 − ρ̃Th P−1
h ρ̃ h = 1 − ρTh P−1
h ρh= 1 − φ T
h ρh . One has
 −1 −1 −1 
Ph +d Ph ρ̃h ρ̃T P
h h
−1
−d −1 −1
Ph
ρ̃ h
P−1
k = −1 T −1
.
−d ρ̃h Ph −1
d

Write φ̃h = P−1

h ρ̃h . Straightforward algebra yields,
     
T
ρ h φ h − d −1
(ρ k − φ h ρ̃h )φ̃h  φh − φkk φ̃h
φ k = Pk −1   =  =  ,
T
ρk d (ρk − φh ρ̃h )
−1
φkk
which gives the recursive formulas for Yule-Walker solutions.

Consider Gaussian process with γ0 = 1. The conditional covariance

matrix of (z0 , zk )|(z1 , . . . , zk−1 ) is given by
   T
1 ρk ρh
ρ 1 − T P h (ρh , ρ̃h ).
−1
k h ρ̃

It follows that αk = (ρk − ρTh P−1 T −1

✫ ✪
h ρ̃h )/(1 − ρh Ph ρh ) = φkk .

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 10

Yule-Walker (Moment) Estimates for AR(p)

Pp
Since γ0 = j=1 φj γj + σa2 , so σa2 = γ0 (1 − φTp ρp ). Substituting
T
ρ̂j = rj , γ̂0 = c0 , one has φ̂p = P̂−1
p ρ̂p , σ̂a2 = c0 (1 − φ̂p ρ̂p ) = v̂p .

Recall the recursive Yule-Walker solutions, and verify that

(1 − φTk ρk ) = (1 − φTk−1 ρk−1 )(1 − φ2kk ), one has
ρk − ρTk−1 φ̃k−1
φkk = = γ0 (ρk − ρTk−1 φ̃k−1 )/vk−1 ,
1 − φTk−1 ρk−1
φk,k−1 = φk−1 − φkk φ̃k−1 ,
vk = γ0 (1 − φTk ρk ) = vk−1 (1 − φ2kk ),

where φTk = (φTk,k−1 , φkk ). Putting hats on the parameters and

starting with φ̂11 = r1 and v̂1 = c0 (1 − r12 ), one obtains the

✫ ✪
Durbin-Levinson algorithm for fitting AR models.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 11

Examples: AR(1), AR(2), and AR(3)

The Y-W equations for AR(1), AR(2), and AR(3) are φ1 = ρ1 ,

    
! ! ! 1 ρ1 ρ2 φ1 ρ1
1 ρ1 φ1 ρ1      
= , and ρ1 1 ρ1  φ2  = ρ2 
     .
ρ 1 φ ρ

1 2 2
ρ2 ρ1 1 φ3 ρ3

The Durbin-Levinson algorithm proceeds as follows:

1. φ11 = r1 ; v1 = c0 (1 − r12 ).
2. φ22 = c0 (r2 − r1 φ11 )/v1 ; φ21 = φ11 − φ22 φ11 ; v2 = v1 (1 − φ222 ).
3. φ33 = c0 (r3 − r1 φ22 − r2 φ21 )/v2 ;
(φ31 , φ32 ) = (φ21 , φ22 ) − φ33 (φ22 , φ21 ); v3 = v2 (1 − φ233 ).

✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 12

MA(q) Process: Invertibility

A moving average process of order q (i.e., MA(q)) is defined by

zt = at − θ1 at−1 − · · · − θq at−q ,
or zt = (1 − θ1 B − · · · − θq B q )at = θ(B)at . The transfer function is
given by ψ(B) = θ(B). MA(q) is stationary by definition.

Similar to the stationarity condition for AR(p), one needs the roots
of θ(B) to lie outside of the unit circle for zt = θ(B)at to be
invertible. Let G−1
j be the roots of θ(B) and consider the spectrum
Qq
p(ω) = σa2 j=1 |1 − Gj e−i2πω |2 . For Gj real,
|1 − Gj e−i2πω |2 ∝ (Gj + G−1
j − 2 cos 2πω),

so G±1
j are exchangeable. Similar arguments can be made for
conjugate pairs of complex roots. Hence, MA(q) models come in
“2q -plet”, of which only one is invertible, barring |Gj | = 1.
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 13

Examples: MA(1) and MA(2)

Invertibility condition
The invertibility of MA(1) and MA(2) is dual to the stationarity of
AR(1) and AR(2).

Variance and autocorrelation

For MA(1), γ0 = σa2 (1 + θ12 ); ρ1 = −θ1 /(1 + θ12 ), ρk = 0, k > 1.
For MA(2), γ0 = σa2 (1 + θ12 + θ22 );
−θ1 (1 − θ2 ) −θ2
ρ1 = , ρ2 = , ρk = 0, k > 2.
1 + θ12 + θ22 1 + θ12 + θ22
Power spectrum
Replacing φ1 by θ1 and φ2 by θ2 in the power spectrums of AR(1)
and AR(2), and move the denominators to the numerators, one

✫ ✪
gets the power spectrums of MA(1) and MA(2).

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 14

Multiplicity: MA(1) and MA(2)

Consider zt = (1 − 2B)at , which has the same autocorrelation

function as the invertible zt = (1 − 0.5B)at .

Consider zt = (1 − B − B 2 )at = (1 + 0.618B)(1 − 1.618B)at , which

has the same autocorrelation function as the invertible
zt = (1 − 0.382B 2 )at = (1 + 0.618B)(1 − 0.618B)at , where
1/1.618 = 0.618. The other two members of the family are
zt = (1 − 2.618B 2 )at = (1 + 1.618B)(1 − 1.618B)at and
zt = (1 + B − B 2 )at = (1 + 1.618B)(1 − 0.618B)at .

The at in different expressions are independent but may have

different variances.

✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 15

ARMA(p, q) Process

An ARMA(p,q) model is of the form

zt − φ1 zt−1 − · · · − φp zt−p = at − θ1 at−1 − · · · − θq zt−q ,
or φ(B)zt = θ(B)at , where φ(B) and θ(B) are polynomials of
degree p and q in B. The stationarity and invertibility are governed
by the roots of φ(B) and θ(B).

For k > q, since φ(B)zt = θ(B)at is uncorrelated with zt−k , one has
cov[φ(B)zt , zt−k ] = φ(B)ρk = 0, or more explicitly,
ρk = φ1 ρk−1 + · · · + φp ρk−p , k > q.

With the transfer function ψ(B) = φ−1 (B)θ(B), the power

spectrum of ARMA(p,q) is seen to be
p(ω) = σa2 |θ(e−i2πω )|2 /|φ(e−i2πω )|2 .
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 16

Example: ARMA(1,1)

Stationarity and invertibility condition

For stationarity, one needs |φ1 | < 1, for invertibility, |θ1 | < 1.

Variance and autocorrelation

Note that E[zt at ] = E[(φ1 zt−1 + at − θ1 at−1 )at ] = σa2 , one has
γ0 = E[(φ1 zt−1 + at − θ1 at−1 )2 ] = φ21 γ0 + σa2 + θ12 σa2 − 2φ1 θ1 σa2 ,
so γ0 = σa2 (1 + θ12 − 2φ1 θ1 )/(1 − φ21 ). Similarly, one has
(φ1 −θ1 )(1−φ1 θ1 )
ρ1 = φ1 − θ1 σa2 /γ0 = 1+θ12 −2φ1 θ1
,
ρk = φ1k−1 ρ1 , k > 1.

Power spectrum
|1 − θ1 e−i2πω |2 2
2 1 + θ1 − 2θ1 cos 2πω
p(ω) = σa2 = σa .
✫ ✪
|1 − φ1 e −i2πω | 2 2
1 + φ1 − 2φ1 cos 2πω

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 17

Moment Estimates for MA(q) and ARMA(p,q)

Pq
For an MA(q) model, one has γ0 = σa2 (1 + j=1 θj2 ) and
Pq−k
γk = σa2 (−θk + j=1 θj θj+k ), k ≥ 1. The moment estimates of σa2 ,
θq , . . . , θ1 can be obtained through a simple iteration,
σa2 = c0 /(1 +
Pq
j=1 θj2 ),
Pq−k
θk = −(ck /σa2 − j=1 θj θj+k ), k = q, . . . , 1.

Remember that the solutions of θj and σa2 are not unique.

For an ARMA(p,q) model, one needs to use cj , j = 0, . . . , p + q.

One can solve φj from the equations,
γk = φ1 γk−1 + · · · + φp γk−p , k = q + 1, . . . , q + p.

Note that wt = φ(B)zt = θ(B)at , and the ACF of wt is

γk′ = φT Γk φ, where φT = (1, −φ1 , . . . , −φp ) and Γk has (i, j)th
T

✫ ✪
entry γk+j−i . Use c′k = φ̂ Γ̂k φ̂ in the MA iteration above to get θj .

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 18

Moment Estimates: ARMA(1,1)

When applied to an ARMA(1,1) process (1 − φB)zt = (1 − θB)at ,

the algorithm for moment estimates proceeds as follows:
1. Solve φ from r2 = φ r1 .
2. Calculate
! ! ! !
c0 c1 1 c1 c2 1
c′0 = (1, −φ) , c′1 = (1, −φ) .
c1 c0 −φ c0 c1 −φ

3. Solve θ, σa2 from equations

′
c 0 c′1
σa2 = , θ = − 2.
1 + θ2 σa

✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 19

Estimation of Mean

Consider φ(B)zt = µ + θ(B)at . It is easily seen that

E[zt ] = µ/(1 − φ1 − · · · − φp ). Recall the “large sample” variance of
the sample mean z̄,
∞
1 X γ(1) σa2 2 σa2 θ2 (1) p(0)
var[z̄] = γk = = ψ (1) = = ,
n n n n φ2 (1) n
k=−∞

where γ(B) = σa2 ψ(B)ψ(B −1 ) is the covariance generating function

and p(ω) is the power spectrum. The moment estimate of µ is thus
√
µ̂ = φ̂(1)z̄ with approximate standard error σ̂a |θ̂(1)|/ n.

Fitting an ARMA(1,1) model to Series A, one has φ̂ = .8683,

θ̂ = .4804, and σ̂a2 = .09842. Further, z̄ = 17.06 with
p
s.e.[z̄] ≈ p̂(0)/n = .0882, and µ̂ = (1 − .8683)(17.06) = 2.25 with
√
s.e.[µ̂] = σ̂a (1 − θ̂)/ n = .0116.
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 20

Linear Difference Equation and ACF

From the linear difference equation φ(B)ρk = 0, k > q, one can

obtain a general expression for ρk .
Qp
Write φ(B) = j=1 (1 − Gj B), where G−1 j are the roots of φ(B). It
is easy to verify that (1 − Gj B)Gtj = 0, so ρk has a term Aj Gkj .

For a double root G−1 2 t

j , one also has (1 − Gj B) (tGj ) = 0, so ρk has
terms (Aj,0 + Aj,1 k)Gkj . In general, a root G−1
j of multiplicity m
Pm−1
contributes terms v=0 Aj,v k v Gkj .

For pairs of conjugate complex roots |Gj |−1 e±iγj , one has terms
|Gj |k (Aj eiγj k + Āj e−iγj k ) = 2|Gj |k |Aj | cos(kγj + αj ).
Pp
Assuming distinct roots, one has ρk = j=1 Aj Gkj , where Aj ’s are
determined by the initial values ρq , . . . , ρq−p+1 .
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 21

Examples: AR(2) and ARMA(2,1)

For (1 − 0.4B − 0.21B 2 )zt = (1 − 0.7B)(1 + 0.3B)zt = at ,

ρk = A1 0.7k + A2 (−0.3)k , k > 0, as φ(B)ρk = 0, k > 0. A1 and A2
can be fixed via ρ0 = 1 and ρ−1 = ρ1 = φ1 /(1 − φ2 ).

For (1 − 0.8eiπ/3 B)(1 − 0.8e−iπ/3 B)zt = (1 − 0.5B)at ,

ρk = A(0.8eiπ/3 )k + Ā(0.8e−iπ/3 )k
= |A|(0.8)k ei(α+kπ/3) + |A|(0.8)k e−i(α+kπ/3)
= (0.8)k 2|A| cos(kπ/3 + α)
= (0.8)k {B cos(kπ/3) + C sin(kπ/3)}, k > 1,

where B and C can be fixed from ρ0 = 1 and ρ1 ; ρ1 and κ = σa2 /γ0

satisfy equations 1 = φ21 + φ22 + 2φ1 φ2 ρ1 + (1 + θ2 − 2φ1 θ)κ and

✫ ✪
ρ1 = φ1 + φ2 ρ1 − θκ, where φ1 = 0.8, φ2 = −0.64, and θ = 0.5.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 22

Reverse Time Stationary Models

A stationary process is characterized by its autocovariance and

mean, independent of the time direction. In particular, models
assuming forward or reverse time are mathematically equivalent.

Recall the autocovariance generating function of zt = ψ(B)at ,

γ(B) = σa2 ψ(B)ψ(B −1 ). It is clear that zt = ψ(F )at has the same
autocovariance, where F = B −1 is the forward shift operator. For
ARMA(p,q), let G−1 j be the roots of θ(B) and H −1
j those of φ(B).
The same autocovariance is shared by all processes of the form
Qp ±1
Qq
j=1 (1 − Hj B )zt = j=1 (1 − Gj B ±1 )at .

Consider an MA(1) process zt = at − θat−1 . For |θ| > 1, one has

zt = at − θat−1 = (−θ)(−θ−1 at + at−1 ) = ãt − θ−1 ãt+1 ,

✫ ✪
an invertible reverse time MA(1) model, where ãt = −θat−1 .

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 23

Model Identification via ACF/PACF

For k > q with an MA(q) process, ρk = 0, E[rk ] ≈ 0, and

Pq
var[rk ] ≈ (1 + 2 j=1 ρ2j )/N .

For k > p with an AR(p) process, φkk = 0, E[φ̂kk ] ≈ 0, and

var[φ̂kk ] ≈ 1/N , where φ̂kk is the Yule-Walker estimate of φkk .

For an stationary ARMA(p,q) process, ρk damps out exponentially.

If φ(B) has a near unit root G−1 i = (1 − δ i ) −1
, ρk has a term
Ai (1 − δi )k ≈ Ai (1 − kδi ), damping out at a much slower linear
rate. A slowly damping ρk signifies nonstationarity.

In practice, one inspect rk for stationarity, take differences if

nonstationary, and repeat the process. The order identification of
mixed ARMA model is not as straightforward.
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 24

Model Selection via AIC or BIC

To each observed series, one usually can fit several different models
with similar goodness-of-fit. For example, suppose the ARMA(1,1)
model (1 + .2B)zt = (1 − .8B)at is a good fit to the data. Since
(1 + .2B)−1 (1 − .8B) = 1 − B + .2B 2 − .04B 3 + · · · ≈ 1 − B + .2B 2 ,

so an MA(2) fit zt = (1 − B + .2B 2 )at is also likely a good fit.

AIC and BIC can be of assistance in the selection of competing

models. Let l(γ|z) be the log likelihood of the model and γ̂ be the
MLE of γ, where γ consists of all model parameters including φj ,
θk , and σa2 . AIC and BIC are defined by
AIC = −2l(γ̂|z) + 2r, BIC = −2l(γ̂|z) + r log n,
where r is the number of parameters and n is the sample size.

✫ ✪
Models with smaller AIC or BIC are considered better ones.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 25

ARIMA(p,d,q) Processes

To model nonstationary yet nonexplosive series, a popular device is

the autoregressive integrated moving average (ARIMA) model,
φ(B)∇d zt = ϕ(B)zt = θ(B)at ,
where ϕ(B) = φ(B)∇d is a generalized AR operator. Note that
∇d = (1 − B)d has roots on the unit circle.

A process with roots of ϕ(B) inside the unit circle is explosive.

Assume stationarity and invertibility for ∇d zt . An ARIMA model

can be written in the AR(∞) form π(B)zt = at , where
P∞
π(B) = 1 − j=1 πj B j = θ−1 (B)φ(B)(1 − B)d .
P∞
For d > 0, since π(1) = 0, one has j=1 πj = 1.
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 26

MA form of ARIMA Processes

Symbolically, an ARIMA process can be written in a MA(∞) form

P∞
zt = ψ(B)at , ψ(B) = 1 + i=1 ψi B i , although {zt } is nonstationary
and the filter unstable. From ϕ(B)ψ(B) = θ(B), one has
ψj = ϕ1 ψj−1 + · · · + ϕp+d ψj−p−d − θj , j > 0,
where ψ0 = 1, ψj = 0, j < 0. For j > q, ϕ(B)ψj = 0.

Take a time origin k < t and write zt = Ik (t − k) + Ck (t − k), where

Pt−k−1
Ik (t − k) = j=0 ψj at−j . For t − k > q, ϕ(B)Ik (t − k) = θ(B)at ,
so ϕ(B)Ck (t − k) = 0. Ck (t − k) is called the complementary
function, and is seen to be determined by the history up to time k.
It follows that E[zt |zk , zk−1 , . . . ] = Ck (t − k).

Note that Ck (t − k) = Ck−1 (t − (k − 1)) + ψt−k ak .

✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 27

ψ Weights and π Weights

(1 − ϕ1 B − ϕ2 B 2 − · · · )(ψ0 + ψ1 B + ψ2 B 2 + · · · ) = (1 − θ1 B − θ2 B 2 − · · · ).
Matching coefficients, one has

ψ1 − ϕ1 ψ0 = −θ1 ,
ψ2 − ϕ1 ψ1 − ϕ2 ψ0 = −θ2 ,
ψ3 − ϕ1 ψ2 − ϕ2 ψ1 − ϕ3 ψ0 = −θ3 ,
......

Likewise, ϕ(B) = θ(B)(−π0 − π1 B − π2 B 2 − · · · ), for π0 = −1, so

π1 − θ 1 π 0 = ϕ 1 ,
π2 − θ 1 π1 − θ 2 π 0 = ϕ 2 ,
π 3 − θ 1 π2 − θ 2 π1 − θ 3 π 0 = ϕ 3 ,
......

✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 28

MA Form of ARIMA: Some Details

For l > 0, I2 (l) uses a3 , a4 , . . . to represent updates to z2+l after z2 .

I2 (t − 2) = at + ψ1 at−1 + ψ2 at−2 + · · · + ψt−3 a3 ,

I2 (t − 3) = at−1 + ψ1 at−2 + ψ2 at−3 + · · · + ψt−4 a3 .

ϕ(B)I2 (t − 2) = I2 (t − 2) − ϕ1 I2 (t − 3) − ϕ2 I2 (t − 4) − · · · = θ(B)at , for

t − 2 > q, is shown below
1 : at + ψ1 at−1 + ψ2 at−2 + · · · + ψt−3 a3

−ϕ1 : at−1 + ψ1 at−2 + · · · + ψt−4 a3

−ϕ2 : at−2 + · · · + ψt−5 a3

......

with coefficients of at , at−1 , . . . given by

at : 1

at−1 : ψ1 − ϕ1 ψ0 = −θ1

at−2 : ψ2 − ϕ1 ψ1 − ϕ2 ψ0 = −θ2

......

a3 : ψt−3 − ϕ1 ψt−4 − · · · − ϕt−3 ψ0 = −θt−3

✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 29

Example: ARIMA(1,1,1)

Consider p = d = q = 1 with |φ|, |θ| < 1. ϕ(B) = (1 − φB)(1 − B).

Since ϕ(B)ψj = 0, j > 1, one has ψj = A0 + A1 φj , where

A0 = (1 − θ)/(1 − φ) and A1 = (θ − φ)/(1 − φ) are determined from
A0 + A1 = ψ0 = 1 and A0 + A1 φ = ψ1 = ϕ1 − θ = 1 + φ − θ.
(k) (k)
Since Ck (t − k) = b0 + b1 φt−k for t − k > 1, one has
Pt−k−1 (k) (k)
zt = j=0 (A0 + A1 φj )at−j + (b0 + b1 φt−k ),
(k) (k) (k) (k)
where b0 , b1 satisfy the initial conditions b0 + b1 = zk and
(k) (k)
b0 + b1 φ + ak+1 = zk+1 = (1 + φ)zk − φzk−1 + ak+1 − θak . Solving for
(k) (k) (k)
b0 , b1 from the equations, one has b0 = (zk − φzk−1 − θak )/(1 − φ),
(k)
b1 = (−φ(zk − zk−1 ) + θak )/(1 − φ).

With π(B) = (1 − θB)−1 (1 − φB)(1 − B), it is easy to verify that

π1 = 1 + φ − θ, πj = (1 − θ)(θ − φ)θj−2 , j > 1.
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 30

Example: IMA(0,2,2)

Consider p = 0, d = q = 2 with θ(B) invertible. ϕ(B) = (1 − B)2 .

Since ϕ(B)ψj = 0, j > 2, one has ψj = A0 + A1 j, where A0 = 1 + θ2 and

A1 = 1 − θ1 − θ2 are solved from A0 + A1 = ψ1 = ϕ1 − θ1 = 2 − θ1 and
A0 + 2A1 = ψ2 = ϕ1 ψ1 + ϕ2 − θ2 = 2(2 − θ1 ) − (1 + θ2 ).
(k) (k)
Since Ck (t − k) = b0 + b1 (t − k) for t − k > 2, one has
Pt−k−1 (k) (k)
zt = at + j=1 (A0 + A1 j)at−j + (b0 + b1 (t − k)),
(k) (k) (k) (k)
where b0 , b1 satisfy the initial conditions b0 + b1 = zk+1 − ak+1 and
(k) (k)
b0 + 2b1 = zk+2 − ak+2 − ψ1 ak+1 . It follows that
(k)
b1 = zk+2 − zk+1 − ak+2 − (1 − θ1 )ak+1 = zk − zk−1 − (θ1 + θ2 )ak − θ2 ak−1
(k) (k) (k)
and b0 = zk+1 − ak+1 − b1 = zk + θ2 ak . Note that Ck (0) = zk 6= b0 .

Since θ(B)π(B) = ϕ(B), one has π1 = 2 − θ1 ,

✫ ✪
π2 = π1 θ1 − (1 + θ2 ) = θ1 (2 − θ1 ) − (1 + θ2 ), and θ(B)πj = 0, j > 2.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 31

ARIMA Processes with Added Noise

The sum of independent MA processes of orders q and q1 is itself

an MA process of order max(q, q1 ).

Suppose one observes Zt = zt + bt , where φ(B)∇d zt = θ(B)at and

φ1 (B)bt = θ1 (B)αt with at , αt being two independent white noise
processes. It follows that
φ1 (B)φ(B)∇d Zt = φ1 (B)θ(B)at + φ(B)θ1 (B)∇d αt ,
so Zt is of order (p1 + p, d, max(p1 + q, p + d + q1 )). In particular, an
IMA process with added white noise is of order (0, d, max(q, d)).

If φ(B) and φ1 (B) share some common roots, the orders will be
lower. In general, an ARIMA model of form ϕ(B)zt = θ(B)at is
over-parameterized if ϕ(B) and θ(B) have common roots.
✫ ✪
C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 32

Example: IMA(0,1,1) and Random Walk

Consider Zt = zt + bt , where ∇zt = at − θat−1 and at , bt are

independent white noise with variances σa2 , σb2 .

For the autocovariance of ∇Zt = (1 − θB)at + (1 − B)bt , one has

γ0 = σa2 (1 + θ2 ) + 2σb2 , γ1 = −θσa2 − σb2 , γk = 0, k > 1.
Write ∇Zt = ut − Θut−1 and equate γ0 = σu2 (1 + Θ2 ), γ1 = −Θσu2 ,
2
√ 2
r(1+θ )+2− 4r(1−θ)2 +r 2 (1−θ 2 )2 2 θσa +σb2
Θ= 2(1+rθ) , σu = Θ ,

where r = σa2 /σb2 . Consider a random walk with θ = 0. One has

√
Θ = (r + 2 − 4r + r2 )/2, σu2 = σb2 /Θ.
Hence, an IMA(0,1,1) process with Θ > 0 is seen to be a random

✫ ✪
walk buried in a white noise.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 33

Testing for Unit Root

Consider an AR(1) process zt = φzt−1 + at . Observing z0 , . . . , zn

Pn
and minimizing the LS criterion t=1 (zt − φzt−1 )2 , one has
Pn Pn 2
Pn Pn 2
φ̂ = t=1 zt zt−1 / t=1 zt−1 = φ + t=1 zt−1 at / t=1 zt−1 .
It can be shown through conditioning arguments that
Pn Pn 2
Pn 2
E[ t=1 zt−1 at ] = 0, var[ t=1 zt−1 at ] = σa E[ t=1 zt−1 ].

For |φ| < 1, zt is stationary with γ0 = var[zt ] = σa2 /(1 − φ2 ), so

p
n/(1 − φ2 )(φ̂ − φ) = Op (1).
Pn 2
For φ = 1, E[ t=1 zt−1 ] = σa2 n(n + 1)/2, thus n(φ̂ − 1) = Op (1).
√ Pn
A test based on the “t-statistic”, τ̂ = (φ̂ − 1)/ s2 / t=1 zt−1 2
,
Pn
where s = t=1 (zt − φ̂zt−1 )2 /(n − 1), was proposed by Dickey and
2

✫ ✪
Fuller, who derived its asymptotic null distribution under φ = 1.

C. Gu Spring 2024
 ✬ ✩
STAT 520 Linear Stationary and Nonstationary Models 34

Testing for Unit Root

Allowing for a constant, a linear trend, and possibly dependent but

stationary innovations ut with autocovariance γk , one has
zt = β0 + β1 (t − n/2) + φzt−1 + ut .
The asymptotic distribution of the “t-statistic”, τ̂ = (φ̂ − 1)/s.e.[φ̂],
was derived by Phillips and Perron under φ = 1, which depends on
P∞
2
γ0 and σ = pu (0) = k=−∞ γk . Consistent estimates of γ0 and σ 2
Pn
are γ̂0 = t=1 û2t /(n − 3) and the Newey-West estimate,
2 −1
Pn 2 −1
Pl Pn
σ̂ = n t=1 ût + 2n s=1 wsl t=s+1 ût ût−s ,

where ût are the residuals from the LS fit, wsl = 1 − s/(l + 1), and
l → ∞, l4 /n → 0 as n → ∞. The test is implemented in PP.test.
Pp
For φ(B)∇zt = θ(B)at , zt = zt−1 + j=1 φj wt−j +θ(B)at = zt−1 +ut ,
where wt = ∇zt . The process {ut } is stationary when {wt } is.
✫ ✪
C. Gu Spring 2024