TechnicalNoteAutoregressiveModel 1 SpiderFinancialCorp,2014

Technical Note: AutoRegressive Model

Weoriginallycomposedthesetechnicalnotesaftersittinginonatimeseriesanalysisclass.Overthe
years,wevemaintainedthesenotesandaddednewinsights,empiricalobservationsandintuitions
acquired.Weoftengobacktothesenotesforresolvingdevelopmentissuesand/ortoproperlyaddress
aproductsupportmatter.
Inthispaper,wellgooveranothersimple,yetfundamental,econometricmodel:theautoregressive
model.Makesureyouhavelookedoverourpriorpaperonthemovingaveragemodel,aswebuildon
manyoftheconceptspresentedinthatpaper.
ThismodelservesasacornerstoneforanyseriousapplicationofARMA/ARIMAmodels.
Background
Theautoregressivemodeloforder p (i.e. ( ) AR p )isdefinedasfollows:

1 1 2 2 2
...
~ i.i.d ~ (0,1)
t o t t p t t
t t
t
x x x x a
a
N
| | | |
c o
c

= + + + + +
=
Where
-
t
a istheinnovationsorshocksforourprocess
- o istheconditionalstandarddeviation(akavolatility)
Essentially,the ( ) AR p ismerelyamultiplelinearregressionmodelwheretheindependent
(explanatory)variablesarethelaggededitionsoftheoutput(i.e.
1 2
, ,...,
t t t p
x x x

).Keepinmindthat
1 2
, ,...,
t t t p
x x x

maybehighlycorrelatedwitheachother.
Why do we need another model?
First,wecanthinkofanARmodelasaspecial(i.e.restricted)representationofa MA( ) process.Lets
considerthefollowingstationaryAR(1)process:

1 1
1 1 1 1
1 1
(1 )( )
t o t t
t o t t
t o t
x x a
x x a
L x a
| |
| | | |
| | |

= + +
= + + +
= + +

Now,bysubtractingthelongrunmeanfromtheresponsevariable(
t
x ),theprocessnowhaszerolong
run(unconditional/marginal)mean.

TechnicalNoteAutoregressiveModel 2 SpiderFinancialCorp,2014

1
1
1
0
1
1
o
o
| |
|

|
|
+ =
=

Next,theprocesscanbefurthersimplifiedasfollows:

1 1
1
(1 )( ) (1 )
1
t t t
t
t
L x L z a
a
z
L
| |
|
= =
=

Forastationaryprocess,the
1
1 | <

2 2 3 3
1 1 1 1
1
(1 ... ...)
1
N N t
t t
a
z L L L L a
L
| | | |
|
= = + + + + + +

Insum,usingtheAR(1)model,weareabletorepresentthis MA( ) modelusingasmallerstorage
requirement.
WecangeneralizetheprocedureforastationaryAR(p)model,andassumingan MA( ) representation
exists,theMAcoefficientsvaluesaresolelydeterminedbytheARcoefficientvalues:

1 1 2 2
1 2 1 1 1 2 2 2
2
1 2 1 2
...
... ...
(1 ... )( ) ..
t o t t p t p t
t o p t t p t p p t
p
p t o p t t
x x x x a
x x x x a
L L L x a a
| | | |
| | | | | | | | | |
| | | | | | |


= + + + + +
= + + + + + + + + +
= + + + + + =

Onceagain,bydesign,thelongrunmeanoftherevisedmodeliszero.

1 2
1 2
1
.. 0
1 ...
1
o p
o
p
p
i
i
| | | |
|

| | |
|
=
+ + + + =
=

=

Hence,theprocesscanberepresentedasfollows:

2
1 2
2
1 2 1 2
(1 ... )
( )
1 ... (1 L)(1 L)..(1 L)
p
p t t
t t
t t p
p p
L L L z a
a a
x z
L L L
| | |

| | |
=
= = =

TechnicalNoteAutoregressiveModel 3 SpiderFinancialCorp,2014

Byhaving 1, {1, 2,.., }

i
i p < e ,wecanusethepartialfractiondecompositionandthegeometric
seriesrepresentation;wethenconstructthealgebraicequivalentofthe MA( ) representation.
Hint:Bynow,thisformulationlooksenoughlikewhatwehavedoneearlierintheMAtechnicalnote,
sinceweinvertedafiniteorderMAprocessintoanequivalentrepresentationof ( ) AR .
Thekeypointisbeingabletoconvertastationary,finiteorderARprocessintoanalgebraically
equivalent MA( ) representation.Thispropertyisreferredtoascausality.
Causality
Definition:Alinearprocess{ }
t
X iscausal(strictly,acausalfunctionof{ }
t
a )ifthereisanequivalent
MA( ) representation.

0
( )
i
t t i t
i
X L a La

=
= + =

Where:

1
i
i

=
<

Causalityisapropertyofboth{ }
t
X and{ }
t
a .
Inplainwords,thevalueof{ }
t
X issolelydependentonthepastvaluesof{ }
t
a .
IMPORTANT:AnAR(p)processiscausal(withrespectto{ }
t
a )ifandonlyifthecharacteristicsroots(i.e.
1
i

)falloutsidetheunitcircle(i.e.
1
1 1
i
i

> < ).
Letsconsiderthefollowingexample:

1
(1 )( ) (1 ) z
1
z z
t t t
t t t
L x L a
a
| |
|
|

= =
>
= +

Now,letsreorganizethetermsinthismodel:

1
"
1
1
z (z )
z z 1
t t t
t t t
a
a
|

=
= + <

TechnicalNoteAutoregressiveModel 4 SpiderFinancialCorp,2014

" " 2 " "

2 2 1 2 2 1
3 2 " " "
3 3 2 1
1 " 2 " " "
1 2 1
" " 2 " "
1 2 3 1
z ( z ) z
z z
z z ...
z ... ...
t t t t t t t
t t t t t
N N N
t t N t N t N t t
N
t t t t t N
a a a a
a a a
a a a a
a a a a




+ + + + + +
+ + + +

+ + + + +
+ + + + +
= + + = + +
= + + +
= + + + + +
= + + + + +

Theprocessaboveisnoncausal,asitsvaluesdependonfuturevaluesof
"
{ }
t
a observations.However,it
isalsostationary.
Goingforward,foranAR(andARMA)process,stationarityisnotsufficientbyitself;theprocessmustbe
causalaswell.Forallourfuturediscussionsandapplication,weshallonlyconsiderstationarycausal
processes.
Stability
Similartowhatwedidinthemovingaveragemodelpaper,wewillnowexaminethelongrunmarginal
(unconditional)meanandvariance.
(1) Letsassumethelongrunmean( )exists,and:

1
[ ] [ ] ... [ ]
t t t p
E x E x E x

= = = =
Now,subtractthelongrunmeanfromalloutputvariables:

1 1 1 2 2 2
1 1 2 2
1 2
( ) ( ) ... ( )
( ) ( ) ( ) ... ( )
+ (1 ... )
t o t t p t p p t
t t t p t p t
o p
x x x x a
x x x x a
| | | | | | |
| | |
| | | |


+ = + + + + + + + +
= + + + +


Taketheexpectationfrombothsides:

1 1 2 2
1 2
1 2
1 2
1
[ ] [ ( ) ( ) ... ( ) ]
+ (1 ... )
0 (1 ... )
1 ...
1
t t t p t p t
o p
o p
o
p
p
i
i
E x E x x x a | | |
| | | |
| | | |
|

| | |
|

=
= + + + +

=
=

=

Insum,forthelongrunmeantoexist,thesumofvaluesoftheARcoefficientscantbeequalto
one.

TechnicalNoteAutoregressiveModel 5 SpiderFinancialCorp,2014

(2) ToexaminethelongrunvarianceofanARprocess,wellusetheequivalent MA( )

representationandexamineitslongrunvariance.

1 1 2 2 3 3
2
1 2
2
1 2
...
(1 ... )
1 ...
t t t t t p t p t
p
p t t
t
t p
p
x y y y y y a
L L L y a
a
y
L L L
| | | |
| | |
| | |

= = + + + + +
=
=

Usingpartialfractiondecomposition:

1 2
1 2
...
1 1 1
p
t t
p
c
c c
y a
L L L
(
= + + +
(

(


ForastableMAprocess,allcharacteristicsroots(i.e.
1
i

)mustfalloutsidetheunitcircle(i.e.
1
i
< ):

2 2 2 2
1 2 1 1 2 2 1 1 2 2
( ... ) ( ... ) L ( ... ) L ...
t p p p p p t
y c c c c c c c c c a ( = + + + + + + + + + + + +


Next,letsexaminetheconvergencepropertyoftheMArepresentation:

1 1 2 2
lim ... 0
k k k
p p
k
c c c

+ + + =
Finally,thelongrunvarianceofaninfiniteMAprocessexistsifthesumofitssquared
coefficientsisfinite.

2 2
1 2 1 1 2 2
2 2
1 1 2 2
2 2
1 1 2 2
1 1 1
Var[ ] (1 ( ... ) ( ... ) ...
+ ( ... ) ...)
( ... ) ( )
T k p p p
k
k k k
p p
p
i i i i
p p j j
i i j
x c c c c c c
c c c
c c c c

o

+

= = =
= + + + + + + + + +
+ + + +
+ + + = <


Furthermore,fortheAR(p)processtobecausal,thesumofabsolutecoefficientvaluesisfinite
aswell.

1 1 1
p
i
k j j
k i j
c

= = =
= <

TechnicalNoteAutoregressiveModel 6 SpiderFinancialCorp,2014

Example: AR(1)

2 2
2
2 4 6 2
2
(1 )
(1 ...)
1
Var[ ] (1 ...)
1
t t
t
t t
t
L y a
a
y L L a
L
y
|
| |
|
o
| | | o
|
=
= = + + +

= + + + + =

Assumingallcharacteristicroots(
1
i

)falloutsidetheunitcircle,theAR(p)processcanbeviewedasa
weightedsumofpstableMAprocesses,soafinitelongrunvariancemustexit.
Impulse Response Function (IRF)
Earlier,weusedAR(p)characteristicsrootsandpartialfractiondecompositiontoderivetheequivalent
ofaninfiniteordermovingaveragerepresentation.Alternatively,wecancomputetheimpulseresponse
function(IRF)andfindtheMAcoefficientsvalues.
Theimpulseresponsefunctiondescribesthemodeloutputtriggeredbyasingleshockattimet.

0 1
1 1
t
t
a
t
=
=

=

1 1
2 1 1 1
3 1 2 2 1
4 1 3 2 2 3 1
5 1 4 2 3 3 2 4 1
1 1 2 1 3 2 1
2 1 1 2 3 1 2
1 1 2 2 3 3
1
...
...
...
...
...
p p p p p
p p p p p
p k p k p k p k p k
y a
y y
y y y
y y y y
y y y y y
y y y y y
y y y y y
y y y y y
| |
| |
| | |
| | | |
| | | |
| | | |
| | | |
+
+ +
+ + + +
= =
= =
= + =
= + +
= + + +
= + + + +
= + + + +
= + + + +

Theprocedureaboveisrelativelysimple(computationally)toperform,andcanbecarriedonforany
arbitraryorder(i.e.k).
Note:Recallthepartialfractiondecompositionwedidearlier:

1 2
1 2
...
1 1 1
p
t t
p
c
c c
y a
L L L
(
= + + +
(

(

TechnicalNoteAutoregressiveModel 7 SpiderFinancialCorp,2014

WederivedthevaluesfortheMAcoefficientsasfollows:
2 2 2 2
1 2 1 1 2 2 1 1 2 2
( ... ) ( ... ) L ( ... ) L ...
t p p p p p t
y c c c c c c c c c a ( = + + + + + + + + + + + +


Inprinciple,theIRFvaluesmustmatchtheMAcoefficientsvalues.Sowecanconclude:
(1) Thesumofdenominators(i.e.
i
c )ofthepartialfractionsequalstoone(i.e.
1
1
1
p
i
i
c y
=
= =

).
(2) Theweightedsumofthecharacteristicsrootsequalsto
1
| (i.e.
2 1
1
p
i i
i
c y |
=
= =

).
(3) Theweightedsumofthesquaredcharacteristicsrootsequalsto
2
1 2
| | + (i.e.
2 2
3 1 2
1
p
i i
i
c y | |
=
= = +

).
Forecasting
Givenaninputdatasample
1 2
{ , ,..., }
T
x x x ,wecancalculatevaluesofthemovingaverageprocessfor
future(i.e.outofsample)valuesasfollows:

1 1 2 2
...
T T T p T p T
y y y y a | | |

= + + + +

1 1 2 1 1
2 1 1 2 2
2
1 2 1 2 3 1 1 1 2 2
[ ] ...
[ ] [ ] ...
= ( ) ( ) ... ( )
T T T p T p
T T T p T p
T T p p T p p T p
E y y y y
E y E y y y
y y y y
| | |
| | |
| | || | || | |
+ +
+ + +
+ +
= + + +
= + + +
+ + + + + + +

Wecancarrythiscalculationtoanynumberofstepswewish.
Next,fortheforecasterror:

2
1 1 2 1 1 1
2 2
2 1 1 2 2 2 1
3 1 2 2 1 3 3
1 1 1 2 2 2
Var[ ] Var[ ... ]
Var[ ] Var[ ... ] (1 )
Var[ ] Var[ ... ]
Var[ ( ...
T T T p T p T
T T T p T p T
T T T p T p T
T T p T p T
y y y y a
y y y y a
y y y y a
y y y a
| | | o
| | | | o
| | |
| | | |
+ + +
+ + + +
+ + + + +
+ + +
= + + + + =
= + + + + = +
= + + + +
= + + + +
2 1 3 3
2 2 2 2 2
1 2 1 1 2 3 1 1 2
) ... ]
Var[( ) .... .... ] (1 ( ) )
T p T p T
T T T
y y a
y a a
| |
| | | | | | o
+ + +
+ + +
+ + + +
= + + + + + = + + +

Asthenumberofstepsincrease,theformulasbecomemorecumbersome.Alternately,wecanusethe
MA( ) equivalentrepresentationandcomputetheforecasterror.

TechnicalNoteAutoregressiveModel 8 SpiderFinancialCorp,2014

2
1 2
IRF={ } (1 ...)
t t t
z z L L a u u = + + +
Andtheforecasterrorisexpressedasfollows:

2
1
2 2
2 1
2 2 2
3 1 2
2 2 2 2
1 2 1
2 2 2
1 2
Var[y ]
Var[y ] (1 )
Var[y ] (1 )
....
Var[y ] (1 ... )
....
Var[y ] (1 ...)
T
T
T
T k k
T k
k
o
u o
u u o
u u u o
u u o
+
+
+
+
+

=
= +
= + +
= + + + +
= + + +

Note:Theconditionalvariancegrowscumulativelyoveraninfinitenumberofstepstoreachitslongrun
(unconditional)variance.
Correlogram
Whatdotheautoregressive(AR)correlogramplotslooklike?HowcanweidentifyanARprocess(and
itsorder)usingonlyACForPACFplots?
First,letsexaminetheACFforanARprocess:
ACF(k)
k
k
o

= =
Where:

2
[( )( )] (covariance for lag j)
[( ) ] (long-run variance)
j t t j
o t
E x x
E x



=
=

Letsfirstcomputetheautocovariancefunction
j
.

1 1 1
1 1 1 2 2 1 1 2 1 3 2 1
2 1 3 2 1 1
[( )( )] [ ]
[( .. ) ] ...
(1 ) ...
t t t t
t t p t p t t o p p
p p o
E x x E z z
E z z z a z

| | | | | | |
| | | |

= =
= + + + + = + + + +
=

2 1 1 2 2 2 1 3 1 2 4 2 2
1 3 1 4 2 5 3 2 2
[( .. ) ] ( ) ...
( ) ( 1) ...
t t p t p t t o p p
p p o
E z z z a z | | | | | | | |
| | | | | |

= + + + + = + + + + +
+ =

TechnicalNoteAutoregressiveModel 9 SpiderFinancialCorp,2014

Next,forthe3
rd
lagcovariance;

3 1 1 2 2 3
3 1 2 2 1 3 4 1 5 2 6 3 2
2 4 1 1 5 2 6 3 7 4 2 3
[( .. ) ]
...
( ) ( ) ( 1) ...
t t p t p t t
o p p
p p o
E z z z a z | | |
| | | | | | |
| | | | | | | |

= + + + +
= + + + + + + +
+ + =

Insum,foranAR(p)process,weneedtoconstructandsolvep1linearsystemstocomputethevalues
ofthefirstp1autocovariances.
2 3 4 5 6 1
1 3 4 5 6 7
2 4 1 5 6 7 8
3 5 2 6 1 7 8 9
4 6 3 7 2 8 1 9 10
3 1 4 5 4 5
1 .
( ) 1 . 0
( ) ( ) 1 . 0 0
( ) ( ) ( ) 1 . 0 0
( ) ( ) ( ) ( ) 1 . 0 0
. . . . . . . .
( ) ( ) .
p p
p
p p p p p p p

+
+ +
+ + +
+ + + +
+ +
1 1
2 2
3 3
4 4
5 5
2 2
2 3 4 5 6 1 1 1
. .
0 0
( ) . 1
o
p p
p p p p p p p p

( ( (
( ( (
( ( (
( ( (
( ( (
( ( (
=
( ( (
( ( (
( ( (
( ( (
( ( (
+
( ( (

Theautocovarianceforlagsgreaterthanp1iscomputediterativelyasfollows:

1 1 2 2 1 1
1 1 2 1 1 2 1
2 1 1 2 1 3 2
1 1 2 2 1 1
...
...
...
...
...
p p p p p o
p p p p p
p p p p p
p k p k p k p k p k
| | | |
| | | |
| | | |
| | | |

+
+ +
+ + + +
= + + + +
= + + + +
= + + + +
= + + + +

Example:ForanAR(5)process,thelinearsystemofequationsoftheautocovariancefunctionsis
expressedbelow:

2 3 4 5 1 1
1 3 4 5 2 2
2 4 1 5 6 3 3
3 5 2 1 4 4
1
( ) 1 0
( ) ( ) 1 0
( ) 1
o

( ( (
( ( (
+
( ( (
=
( ( ( + +
( ( (
+

Q:whatdotheylooklikeintheACFplot?
Duetothecausalityeffect,ACFvaluesofatrueARprocessdontdroptozeroatanylag
number,butrathertailexponentially.
ThispropertyhelpsustoqualitativelyidentifytheAR/ARMA(vs.MA)processintheACFplot.
Determiningtheactualorder(i.e.p)oftheunderlyingARprocessis,inmostcases,difficult.

TechnicalNoteAutoregressiveModel 10 SpiderFinancialCorp,2014

Example:LetsconsidertheAR(1)process:

1 t t t
z z a

= +

1 1
2
2 2 1
3
3 2
1
[ ]
[ ]
...
t t o
t t o
o
k
k k o
E z z
E z z

= =
= = =
= =
= =

TheACFforanAR(1)processcanbeexpressedasfollows:
ACF(k)
k k
o

= =
TheACFvaluesdontdroptozeroatanylagnumber,butratherdeclineexponentially.
Q:WhataboutahigherorderARprocess?
TheACFplotcangetincreasinglymorecomplex,butitwillalwaystailexponentially.Thisisduetothe
modelscausalproperty.WecantellthedifferencebetweenanMAprocessandanAR/ARMAprocess
bythisqualitativedifference.
WeneedadifferenttoolorplottohelpidentifytheexactorderoftheARprocessanditsorder:aplot
thatdropstozeroafterthepthlagswhenthetruemodelisAR(p).Thistoolorplotisthepartialauto
correlationplot(PACF).
Partial autocorrelation function (PACF)
Thepartialautocorrelationfunction(PACF)isinterpretedasthecorrelationbetween
t
x and
t h
x

,
wherethelineardependencyoftheinterveninglags(
1 2 1
, ,...,
t t t h
x x x
+
)hasbeenremoved.

1 2 1
PACF( ) ( , | , ,..., )
t t h t t t h
h Corr x x x x x
+
=
Notethatthisisalsohowtheparametersofamultiplelinearregression(MLR)modelsareinterpreted.
Example:

2
1
2
1 2
t o
t o
x t
x t t
| |
| | |
= +
= + +

TechnicalNoteAutoregressiveModel 11 SpiderFinancialCorp,2014

Inthefirstmodel,
1
| isinterpretedasthelineardependencybetween
2
t and
t
x .Inthesecondmodel,
the
2
| isinterpretedasthelineardependencybetween
2
t and
t
x ,butwiththedependencybetween t
and
t
x alreadyaccountedfor.
Insum,thePACFhasaverysimilarinterpretationasthecoefficientsinthemultipleregressionsituations
andthePACFvaluesareestimatedusingthosecoefficientvalues.
(1) Constructaseriesofregressionmodelsandestimatetheparametersvalues:

0,1 1,1 1
0,2 1,2 1 2,2 2
0,3 1,3 1 2,3 2 3,3 3
0,4 1,4 1 2,4 2 3,4 3 4,4 4
0, 1, 1 2, 2 3, 3 ,
...
...
t t t
t t t t
t t t t t
t t t t t t
t k k t k t k t k k t k t
x x a
x x x a
x x x x a
x x x x x a
x x x x x a
| |
| | |
| | | |
| | | | |
| | | | |

= + +
= + + +
= + + + +
= + + + + +
= + + + + + +

(2) ThePACF(k)isestimatedby
, k k
| .
Notes:
(1) ToestimatethePACFofthefirstklags,wedneedtosolvekregressionmodels,whichcanbe
slowforlargerdatasets.Anumberofalgorithms(e.g.DurbinLevensonalgorithmandYule
Walkerestimations)canbeemployedtoexpeditethecalculations.
(2) ThePACFcanbecalculatedfromthesampleautocovariance.Forexample,toestimatethe
PACF(2),wesolvethefollowingsystem:

1,2 1 1
2,2 1 2
o
o
|
|

( ( (
=
( ( (


ForPACF(3),wesolvethefollowingsystem:

1 2 1,3 1
1 1 2,3 2
2 1 3,3 3
o
o
o
|
|
|

(
( (
(
( (
=
(
( (
(
( (



UsingtheDurbinLevensonalgorithmimprovesthecalculationspeeddramaticallybyreusingprior
calculationstoestimatecurrentones.
[( )( )]
j t t j
E x x

TechnicalNoteAutoregressiveModel 12 SpiderFinancialCorp,2014

Bydefinition,theautocovarianceoflagorderzero(
o
)istheunconditional(marginal)variance.
Bydesign,foratrueAR(p)process,thecorrespondingPACFplotdropstozeroafterplags.Ontheother
hand,theACFplottails(declines)exponentially.
UsingonlythePACFplot,IshouldbeabletoconstructanARmodelforanyprocess,right?No.
ThePACFplotmainlyexamineswhethertheunderlyingprocessisatrueARprocessandidentifiesthe
orderofthemodel.
Conclusion
Torecap,inthispaper,welaidthefoundationforaslightlymorecomplexmodel:theautoregressive
model(AR).First,wepresentedtheARprocessasarestrictedformofaninfiniteorderMAprocess.
Next,armedwithafewmathematicaltricks(i.e.IRF,partialfractiondecompositionandgeometric
series),wetackledmanymorecomplexcharacteristicsofthisprocess(e.g.forecasting,longrun
variance,etc.)byrepresentingitasanMAprocess.
Lateron,weintroducedanewconcept:Causality.Aprocessisdefinedascausalifandonlyifitsvalues
{ }
t
X aredependentontheprocessspastshocks/innovations
1 2
{ , , ,...}
t t t
a a a

.Weshowedthat
stationarityisnotasufficientconditionforourmodels;theymustbecausalaswell.
Finally,wedelvedintoARprocessidentificationusingcorrelogram(i.e.ACFandPACF)plots.Weshowed
thattheACFofanARprocessdoesnotdroptozero,butrathertailsexponentiallyinallcases.
Furthermore,welookedintoPACFplotsandoutlinedthatfactthatPACF,bydesign,dropstozeroafter
plagsforatrueARprocess.
Aswegoontodiscussmoreadvancedmodelsinfuturetechnicalnotes,wewilloftenrefertotheMA
andARprocessesandthematerialpresentedhere.



References
- Hamilton, J .D.; Time Series Analysis , Princeton University Press (1994), ISBN 0-691-04289-6
- D. S.G. Pollock,; Handbook of Time Series Analysis, Signal Processing, and Dynamics , Academic Press (1999),
ISBN: 0125609906
- Box, J enkins and Reisel; Time Series Analysis: Forecasting and Control , J ohn Wiley & SONS. (2008) 4th
edition, ISBN:0470272848