AR, MA and ARMA models

• The autoregressive process of order p or AR(p) is defined

by the equation
p
X
Xt = φj Xt−j + ωt
j=1

where ωt ∼ N (0, σ 2 )
• φ = (φ1 , φ2 , . . . , φp ) is the vector of model coefficients and
p is a non-negative integer.
• The AR model establishes that a realization at time t is a
linear combination of the p previous realization plus some
noise term.
• For p = 0, Xt = ωt and there is no autoregression term.

53
• The lag operator is denoted by B and used to express
lagged values of the process so BXt = Xt−1 ,
B 2 Xt = Xt−2 , B 3 Xt = Xt−3 ,. . ., B d Xt−d .

• If we define
p
X
Φ(B) = 1 − φj B j = 1 − φ 1 B − φ 2 B 2 − . . . − φ p B p
j=1

the AR(p) process is given by the equation

Φ(B)Xt = ωt ; t = 1, . . . , n.

• Φ(B) is known as the characteristic polynomial of the

process and its roots determine when the process is
stationary or not.

• The moving average process of order q or M A(q) is

54
 defined as
q
X
Xt = ω t + θj ωt−j
j=1

• Under this model, the observed process depends on

previous ωt0 s
• M A(q) can define correlated noise structure in our data
and goes beyond the traditional assumption where errors
are iid.
• In lag operator notation, the M A(q) process is given by
Pq
the equation Xt = Θ(B)ωt where Θ(B) = 1 + j=1 θj B j .
• The general autoregressive moving average process of
orders p and q or ARM A(p, q) combines both AR and
M A models into a unique representation.

55
• The ARMA process of orders p and q is defined as
p
X q
X
Xt = φj Xt−j + θj ωt−j + ωt
j=1 j=1

• In lag operator notation, the ARM A(p, q) process is

given by Φ(B)Xt = Θ(B)ωt , t = 1, . . . , n
• Lets focus on the AR process and its characteristic
polynomial.
• The characteristic polynomial can be expressed as:
p
Y
Φ(B) = (1 − αi B)
i=1

where the α0 s are the reciprocal roots.

• If β1 , β2 , . . . , βp are such that Φ(βi ) = 0 (roots of the

56
 polynomial) then β1 = 1/α1 , β2 = 1/α2 , . . . , βp = 1/αp

• Theorem: If Xt ∼ AR(p), Xt is a stationary process if and

only if the modulus of all the roots of the characteristic
polynomial are greater than one, i.e. if ||βi || > 1 for all
i = 1, 2, . . . , p or equivalently if ||αi || < 1, i = 1, 2, . . . p.

• The αi0 s are also known as the poles of the AR process.

• This theorem follows from the general linear process

theory.

• Some of the poles or reciprocal roots can be real number

and some can be complex numbers and we need to
distinguish between the 2 cases.

57
• For the complex case, we will use the representation

αi = ri exp(±ωi i), i = 1, . . . , C

so C is the total number of conjugate pairs and 2C is the

total number of complex poles.
• ri is the modulus of αi and ωi its frequency.
• The real reciprocal roots are denoted as

αi = ri , i = 1, 2, . . . R

• Example. Consider the AR(1) process Xt = φXt−1 + ωt .

• In lag-operator notation this process is (1 − φB)Xt = ωt
and the characteristic polynomial is Φ(B) = (1 − φB).
• If Φ(B) = (1 − φB) = 0, the only characteristic root is

58
 β = 1/φ (assuming φ 6= 0).
• The AR(1) process is stationary if only if |φ| < 1 or
−1 < φ < 1.
• The case where φ = 1 corresponds to a Random Walk
process with a zero drift, Xt = Xt−1 + ωt
• This is a non-stationary explosive process.
• If we recursive apply the AR(1) equation, the Random
Walk process can be expressed as
Xt = ωt + ωt−1 + ωt−2 + . . .. Then,
P∞ 2
V ar(Xt ) = t=0 σ = ∞.
• Example. AR(2) process Xt = φ1 Xt−1 + φ2 Xt−2 + ωt
• The characteristic polynomial is now

59
 Φ(B) = (1 − φ1 B − φ2 B 2 )
• The solutions to Φ(B) = 0 are
q q
−φ1 + φ21 + 4φ2 −φ1 − φ21 + 4φ2
β1 = ; β2 =
2φ2 2φ2
• The reciprocal roots are
q q
φ1 + φ21 + 4φ2 φ1 − φ21 + 4φ2
α1 = ; α2 =
2 2
• The AR(2) is stationary if and only if ||α1 || < 1 and
||α2 || < 1
• These two conditions imply that ||α1 α2 || = |φ2 | < 1 and
||α1 + α2 || = |φ1 | < 2 which means −1 < φ2 < 1 and
−2 < φ1 < 2.

60
• For α1 and α2 real numbers, φ21 + 4φ2 ≥ 0 which implies
−1 < α2 ≤ α1 < 1 and after some algebra φ1 + φ2 < 1;
φ2 − φ1 < 1
φ21
• In the complex case φ21 + 4φ2 < 0 or −4 > φ2
• If we combine all the inequalities we obtain a region
bounded by the lines φ2 = 1 + φ1 ; φ2 = 1 − φ1 ; φ2 = −1.
• This is the region where the AR(2) process is stationary.
• For an AR(p) where p ≥ 3, the region where the process
is stationary is quite abstract.
• For the stationarity condition of the MA(q) process, we
need to rely on the general linear process.
• A general linear process is a random sequence Xt of the

61
 form,
∞
X
Xt = aj ωt−j
j=0

where ωt is a white noise sequence with variance σ 2 .

• In lag operator notation, the general linear is given by the

expression Xt = Φ(B) ωt where Φ(B) = j=0 aj B j .
−1 −1 P∞

• Note firstly that by the definition of the linear process,

E(Xt ) = 0.

• Then, the covariance between Xt and Xs is

∞ X
X ∞
E[Xt Xs ] = aj al E[Xt−j Xs−l ]
j=0 l=0

62
 ∞
X
= σ2 aj aj+s−t ; (s ≥ t)
j=0

• The last expression depends on t and s only through the

difference s − t. Therefore, the process is stationary if
P∞
j=0 aj aj+k is finite for all non-negative integers k
P∞ 2
• Setting k = 0 we require that j=0 j < ∞
a
• Given that a correlation is always between −1 and 1,

|γk | ≤ γ0
P∞
so if γ0 < ∞ then j=0 aj aj+k < ∞.
P∞ 2
• Then Xt is stationary if and only if j=0 j < ∞
a
• The MA(q) process can be written as a general linear

63
 P∞
process of the form Xt = j=0 aj ωt−j where

 θj j=0,. . . ,q
aj =
 0 j=q+1,q+2,. . .
with θ0 = 1.
2 Pq
= j=0 θj2 < ∞ so a
P∞
• For the MA(q) process j=0 aj
moving average process is always stationary.
• For the ARMA(p,q) process given by Φ(B)Xt = Θ(B)ωt
Xt is stationary if only if the roots of Φ(B) = 0 have all
modulus greater than 1 or all the reciprocal roots have a
modulus less than one.
• A related concept to stationary linear process is
invertible process.

64
• Definition: A process Xt is invertible if
∞
X
Xt = aj Xt−j + ωt
j=1
P∞ 2
with the restriction that j=1 aj <∞

• Basically, an invertible process is an infinite

autoregression.

• By definition the AR(p) is invertible. We can set

a1 = φ1 , a2 = φ2 , . . . ap = φp and aj = 0, j > p. Then
P∞ 2 Pp 2 which is finite.
a
j=1 j = φ
=1 j

• For an MA(q) process we have Xt = Θ(B)ωt . If we find a

polynomial Θ(B)−1 such that Θ(B)Θ(B)−1 = 1 then we
can invert the process since Θ(B)−1 Xt = ωt

65
• The MA(q) process is invertible if and only if the roots of
Θ(B) have all modulus greater than one.
• To illustrate this last point consider the MA(1) process
Xt = (1 − θB)ωt
• If |θ| < 1 then
∞
1 X
Θ(B)−1 = = θj B j
(1 − θB) j=0

• Since |θ| < 1 then j=0 θj < ∞ and so the process is

P∞

invertible and has the representation

∞
X
Xt = θj Xt−j + ωt
j=1

• The ARMA(p,q) process is invertible whenever the MA

66
 part of the process is invertible, i.e. when Θ(B) has
reciprocal roots with modulus less than one.

Autocorrelation and Partial Autocorrelation

• The partial autocorrelation function (PACF) of a process

Zt is defined as
Pk = Corr(Zt , Zt+k |Zt+1 , . . . , Zt+k−1 ); k = 0, 1, 2, 3, . . .

• This PACF is equal to the ordinary correlation between

ˆ where Ẑt and Zt+k
Zt − Ẑt and Zt+k − Zt+k ˆ are the
“best” linear estimators for Zt and Zt+k respectively.

• This PACF can also be derived through an autoregressive

model of order k

Zt+k = φk1 Zt+k−1 + φk2 Zt+k−2 + . . . + φkk Zt + ωt+k

67
• The coefficients φk1 , φk2 , . . . , φkk define the PACF.

• We have a set of linear equations for which the solution

can be obtained via Cramer’s Rule.

• R/S-plus include an option to compute the PACF.

> acf(x,type=’’partial’’)

68
• The partial autocorrelation can be derived as follows.
Suppose that Zt is zero mean stationary process.

• Consider a regression model where Zt+k is regressed on k

lagged variables Zt+k−1 , Zt+k−2 , . . . , Zt , i.e.,

Zt+k = φk1 Zt+k−1 + φk2 Zt+k−2 + . . . + φkk Zt + ωt+k

• φki denotes the i-th regression parameter and ωt+k is a

normal error term uncorrelated with Zt+k−j for j ≥ 1.

• Multiplying Zt+k−j on both sides of the above regression

equation and taking the expectation, we get

γj = φk1 γj−1 + φk2 γj−2 + . . . + φkk γj−k

69
• If we divide by γ0 we get,

ρj = φk1 ρj−1 + φk2 ρj−2 + . . . + φkk ρj−k

• For j = 1, 2, . . . , k, we have the following system of

equations:

ρ1 = φk1 ρ0 + φk2 ρ1 + . . . + φkk ρk−1

ρ2 = φk1 ρ1 + φk2 ρ0 + . . . + φkk ρk−2
..
.
ρk = φk1 ρk−1 + φk2 ρk−2 + . . . + φkk ρ0

• Using Cramer’s rule successively for k = 1, 2, . . ., we have

φ11 = ρ1

70
  
1 ρ1
 
 
 
ρ1 ρ2
 
 
φ22 =  
1 ρ1
 
 
 
ρ1 ρ1
 
 
 

 1 ρ1 ρ1 

 

 ρ1 1 ρ2 

 
 ρ2 ρ1 ρ3 
φ33 =  

 1 ρ1 ρ2 

 

 ρ1 1 ρ1 

 
 ρ2 ρ1 1 

71
  
1 ρ1 ρ2 . . . ρk−2 ρ1 
 

 
ρ1 1 ρ1 ... ρk−3 ρ2 
 

.. .. .. .. .. .. 
 


 . . . . . . 
ρk−1 ρk−2 ρk−3 . . . ρ1 ρk 
 

φkk =  
1 ρ1 ρ2 . . . ρk−2 ρk−1
 
 
 
ρ1 1 ρ1 ... ρk−3 ρk−2
 
 
.. .. .. .. .. ..
 
 

 . . . . . . 

ρk−1 ρk−2 ρk−3 . . . ρ1 1
 
 

• As a function of k, φkk is usually referred to as the

partial autocorrelation function (PACF).
• A computer package will produce an estimate of φk,k
using ρˆk

72
• Example. Consider the stationary AR(1) process ,
Xt = αXt−1 + ωt with −1 < α < 1. (change φ to α)

• Previously, we establish that the autocorrelation function

for an AR(1) is ρk = αk .

• If we apply Cramer’s rule φ11 = ρ1 = α.

• Also
 
 1 α
 

 
 α α2
 

φ22 =   =0
 1 α
 

 
 α 1
 


• In fact, it can be checked that φkk = 0 for any k ≥ 2.

73
• The result is that for the AR(1)

 α k=1
φkk =
 0 k≥2

so the partial autocorrelation of an AR(1) cuts down to

zero after lag 1.

• Examples. Simulation of an AR(1) process with

coefficient α = 0.9 and α = −0.5. 1000 observations in
each case. In the second case σ 2 = 9.

74
alpha=.9
xt=arima.sim(1000,model=list(ar=alpha))
par(mfrow=c(3,1),oma=c(2,2,2,2))
ts.plot(ts(xt))
acf(xt,lag=20)
acf(xt,type="partial",lag=20)
mtext("AR(1) process with alpha =.9,
sigma^2=1",outer=T,cex=1.1)

75
 AR(1) process with alpha =.9, sigma^2=1

6
4
2
ts(xt)

−2 0
−6

0 200 400 600 800 1000

Time

Series xt
0.8
ACF

0.4
0.0

0 5 10 15 20

Lag

Series xt
0.8
Partial ACF

0.4
0.0

5 10 15 20

Lag

76
# Now with a variance different to one
epsilon=rnorm(500,mean=0,sd=sqrt(3))
alpha=-.5
xt=arima.sim(500,model=list(ar=alpha),innov=epsilon)
par(mfrow=c(3,1),oma=c(2,2,2,2))
ts.plot(ts(xt))
acf(xt,lag=50)
acf(xt,type="partial",lag=50)
mtext("AR(1) process with alpha=-.5,
sigma^2=3",outer=T,cex=1.1)

77
 AR(1) process with alpha=−.5, sigma^2=3

6
4
2
ts(xt)

0
−4

0 100 200 300 400 500

Time

Series xt
1.0
0.5
ACF

0.0
−0.5

0 10 20 30 40 50

Lag

Series xt
0.1
−0.1
Partial ACF

−0.3
−0.5

0 10 20 30 40 50

Lag

78
• Example. The AR(2) model

Xt = φ1 Xt−1 + φ2 Xt−2 + ωt

• First we need to find the autocorrelation function of the

process. That will allow us to find the PACF.
• By multiplying the AR(2) equation by Xt−k (both sides)
and taking the expected value, we get

γk = φ1 γk−1 + φ2 γk−2 , k = 1, 2, . . .

• Dividing by γ0 gives,

ρk = φ1 ρk−1 + φ2 ρk−2 , k = 1, 2, . . .

which defines a linear difference equation for ρk .

• These difference equations can be difficult to solve.

79
• For this type of equations, it is recommended (Diggle) to
explore a solution of the form ρk = λk and try to
determine the value of λ.
• If we substitute λk in the difference equation, we get

λk = φ1 λk−1 + φ2 λk−2

• Which gives the equation

λ2 − φ 1 λ − φ 2 = 0

• The solution to this equation is

q
φ1 ± φ21 + 4φ2
λ=
2
• This expression gives α1 and α2 , the reciprocal roots of

80
 the characteristic polynomial.
• Since α1 and α2 are solutions of the equation, then a
linear combination is also a solution.
• Then, the general solution to the difference equation
takes the form,

ρk = aα1k + bα2k , k = 0, 1, 2, . . .

where a and b are constants to be determined.

• Given this form, the ρ0k s will have an exponential
behavior.
• To find values for a and b, note that for k = 0, 1, the
difference equations are:

ρ0 = 1 = a + b

81
 ρ1 = φ1 /(1 − φ2 ) = aα1 + bα2

• Assuming the AR(2) satisfies the stationarity conditions,

we can find the values of ρk recursively.
• For the PACF, following Cramer’s rule (HW exercise) it
can be shown that the first two partial correlations are:
φ1
P1 =
1 − φ2
P2 = φ2

and the remaining Pk ’s are zero.

• Consider an AR(2) with complex reciprocal roots
α1,2 = rexp(±ωi)
• The characteristic polynomial is

82
 Φ(B) = (1 − re−iω B)(1 − reiω B) = (1 − 2rcos(ω)B + r 2 B 2 )

• The AR coefficients are given by φ1 = 2rcos(ω), φ2 = −r2

• Lets look at 500 simulated observations of an AR(2)

process with r = 0.95 (r = 0.75) and ω = 2π/15. Here is
the R code to obtain this.

83
r=0.95
w=2*pi/15
phi1=2*r*cos(w)
phi2=-r^2
xt=arima.sim(500,model=list(ar=c(phi1,phi2)))
par(mfrow=c(3,1),oma=c(2,2,2,2))
ts.plot(as.ts(xt))
acf(xt,lag=50)
acf(xt,lag=50,type="partial")

84
 AR(2) process with Phi1= 1.74 and Ph2= −0.9

15
as.ts(xt)

5
−5
−15

0 100 200 300 400 500

Time

Series xt
1.0
0.5
ACF

0.0
−0.5

0 10 20 30 40 50

Lag

Series xt
0.5
Partial ACF

0.0
−0.5

0 10 20 30 40 50

Lag

85
 AR(2) process with Phi1= 1.37 and Ph2= −0.56

0 2 4 6
as.ts(xt)

−4
−8

0 100 200 300 400 500

Time

Series xt
1.0
0.6
ACF

0.2
−0.2

0 10 20 30 40 50

Lag

Series xt
0.8
Partial ACF

0.4
0.0
−0.6

0 10 20 30 40 50

Lag

86
 • For the case of an AR(2) with two real roots r1 and r2 ,
the characteristic polynomial is
Φ(B) = (1 − r1 B)(1 − r2 B) = (1 − (r1 + r2 )B + r1 r2 B 2 )

• The AR(2) coefficients are φ1 = r1 + r2 and φ2 = −r1 r2

• Lets look at a simulated process with r1 = 0.9 and

r2 = 0.5.

• Then we will consider the case r1 = −0.9 and r2 = −.0.5

r1=0.9
r2=0.5
phi1=r1+r2
phi2=-r1*r2
xt=arima.sim(500,model=list(ar=c(phi1,phi2)))

87
 AR(2) process with Phi1= 1.4 and Ph2= −0.45

10
5
ts(xt)

0
−10

0 100 200 300 400 500

Time

Series xt
1.0
0.6
ACF

0.2
−0.2

0 10 20 30 40 50

Lag

Series xt
1.0
Partial ACF

0.5
0.0
−0.5

0 10 20 30 40 50

Lag

88
 AR(2) process with Phi1= −1.4 and Ph2= −0.45

5 10
ts(xt)

−5 0
−15

0 100 200 300 400 500

Time

Series xt
1.0
0.5
ACF

0.0
−1.0

0 10 20 30 40 50

Lag

Series xt
−0.2
Partial ACF

−0.6
−1.0

0 10 20 30 40 50

Lag

89