UNIVERSITY OF LIMPOPO

TURFLOOP CAMPUS

FACULTY OF SCIENCE AND AGRICULTURE

SCHOOL OF MATHEMATICS AND COMPUTER SCIENCES

DEPARTMENT OF STATISTICS & OPERATIONS RESEARCH

SSTA031
TIME SERIES ANALYSIS LECTURE NOTES
 TIME SERIES ANALYSIS LECTURE NOTES 2023

COURSE DESCRIPTIVE
In this course we look at the applied issues of Time Series Analysis. We will combine theoretical work
with the use of computer techniques for model solution. The purpose of this series of lectures is to
provide students with sufficient background in modern Time Series Analysis to choose techniques
suited both to the data-source and the Time Series model. The course places some emphasis on the
link between Time Series theory and forecasting estimation and deals explicitly with interpretation and
critical appraisal of forecasting estimates.

LEARNING OUTCOMES
After successful completion of the module, the student should be able to :

i. Understand the basic theory of time series analysis and forecasting approaches;
ii. Synthesize the relevant statistical knowledge and techniques for forecasting;
iii. Use procedures in popular statistical software for the analysis of time series and forecasting;
iv. Interpret analysis results and make recommendations for the choice of forecasting methods;
v. Produce and evaluate forecasts for given time series;
vi. Present analysis results of forecasting problems.

Lecturer : Mr K.N Maswanganyi ; Mr R.J Lebogo

Office Number : 2004D; 2027 (Maths Building)
Tel Number : 015 268 3680; 015 268 3587
Lecturing Venue : Check the Main Timetable

Lecture Notes : A copy of Lecture notes will be available on UL blackboard. The notes do
not include proofs of stationarity/invertibility , so students are advised to
take additional notes in class.

2|Page
 TIME SERIES ANALYSIS LECTURE NOTES 2023

READING LIST

Although lecture notes are provided, it is important that you reinforce this material by referring to more
detailed texts.

RECOMMENDED TEXTS

• Chris Chatfield (2004).The Analysis of Time Series, An Introduction, Six Edition, Chapman
& Hall/CRC.
• Cryer, D. C. and Chan, K (2008). Time Series Analysis with Application in R, 2nd Edition,
Springer.
• Wei W.W.S (2006) Time Series Analysis, Univariate and Multivariate Methods, Second
Edition, Pearson Addison Wesley.

SUPPLEMENTARY TEXTS

• Cryer, D. J. (1986) Time Series Analysis, Duxbury Press

• Abraham B. and Ledolter J. (1983) Statistical Methods for Forecasting
• Wiley Series, New York
• Peter J. Brockwell and Richard A.Davis (2002), Introduction to Time Series and Forecasting
,Second Edition,Springer.

TIME SCHEDULE
The lecture topics within the semester are as in the following schedule:

Week Dates Topics Chapters

1 30 Jan-03 Feb Introduction to Time Series 1
2 06-10 Feb Introduction to Time Series 1
3 13-17 Feb The Model Building Strategy 2
4 20 -24 Feb Models for Stationary Time Series 3
5 27 Feb-03 Mar Models for Stationary Time Series 3
6 06 -10 Mar Models for Non-Stationary Time Series 4
7 13-17 Mar Models for Non-Stationary Time Series 4
AUTUMN RECESS :20-24 March
9 27-31 Mar Models for Non-Stationary Time Series 4
10 03 -07 Apr Parameter Estimation 5
11 10-14 Apr Parameter Estimation 5
12 17 -21 Apr Model Specification and Diagnostics 6
13 24 -28 Apr Forecasting 7
14 01-05 May Forecasting 7
15 08 -12 May Forecasting 7
16 15-19 May REVISION WEEK

3|Page
 TIME SERIES ANALYSIS LECTURE NOTES 2023

NOTATIONS

Symbol Description

xt Observed time series at time t

{xt } Observed time series for all t

∇k k -th order differencing

∇ ks Seasonal differencing

Ε( x ) Mean of x

γ k and ACVF Autocovariance function

ρ k and ACF Autocorrelation function

Φ kk and PACF Partial autocorrelation function

WN 0, σ 2( ) White noise with mean 0 and variance σ 2

(
N 1, σ 2 ) Normally distributed with mean 1 and variance σ 2

IID Independent,identically distributed

AR( p ) Autoregressive model of order p

MA( q ) Moving average model of order q

ARMA( p, q ) Autoregressive moving average model of order ( p, q )

ARIMA( p, d , q ) Intergrated ARMA( p, q ) model

SARIMA Seasonal ARIMA

B Backward shift operator.

4|Page
 TIME SERIES ANALYSIS LECTURE NOTES 2023

CONTENTS
_____________________________________________________
Course Descriptive 2

Reading list & Time schedule 3

Notations 4

1 Introduction to Time Series 9

1.1 Introduction 9

1.2 Why do we need Time Series? 9

1.3 Time Series plots 10

1.4 General Approach to Time Series Modelling 10

1.5 Component of a Time Series 10

1.5.1 Trend (T) 10

1.5.2 Seasonal variation (S) 11

1.5.3 Cyclical variation (C) 11

1.5.4 Irregular variation (I) 11

1.6 Decomposition of a time series 12

1.7 Smoothing Methods 12

1.7.1 Moving Averages 12

1.7.2 Running Median 12

1.7.3 Exponential Smoothing 12

1.8 Trend Analysis 13

1.8.1 Methods for Trend Isolation 13

1.8.2 Calculating Moving Average 13

1.8.3 Regression Analysis 14

5|Page
 TIME SERIES ANALYSIS LECTURE NOTES 2023

1.9 Seasonal Analysis 15

1.9.1 Ratio-to-Moving Average Method 16

2 The Model Building Strategy 18

2.1 Introduction 18

2.2 The Box-Jenkins Technique 18

2.2.1 Model Specification 18

2.2.2 Model Fitting 19

2.2.3 Model Diagnostic 19

2.3 Stationary Time Series 19

2.3.1 Transformations 20

2.3.2 Stationary through differencing 21

2.4 Analyzing Series Which Contains a Trend 21

2.4.1 Filtering 21

2.5 Stochastic Processes 22

2.6 Mean,Variance & Covariances 22

3 Models for Stationary Time Series 23

3.1 Introduction 23

3.1.1 Strictly Stationary Processes 23

3.1.2 Weakly Stationary Processes 23

3.2 Autocorrelation Function of Stationary Processes 24

3.3 Purely Random Process 24

3.4 Random Walk 25

3.5 Moving Average Processes 26

3.5.1 MA(1) 26

3.5.2 MA(2) 26

3.5.3 MA( q ) 27

3.6 Partial Autocorrelation Function 28

6|Page
 TIME SERIES ANALYSIS LECTURE NOTES 2023

3.7 Autoregressive Processes 28

3.7.1 AR(1) 28

3.7.2 AR(2) 29

3.7.3 AR( p ) 30

3.8 The Dual Relationship between AR and MA Processes 30

3.9 Yule-Walker Equation 32

3.10 Mixed ARMA Models 33

3.10.1 ARMA(1,1) 34

3.10.2 Weights (ψ , π ) 34

3.11 Seasonal ARMA Models 35

4 Model for Non Stationary series 35

4.1 ARIMA Models 35

4.2 Non Stationary Seasonal Process 36

4.2.1 SARIMA Model 36

5 Parameter Estimation 37

5.1 Introduction 37

5.2 Methods of Moments 37

5.2.1 Mean 37

5.2.2 Autoregressive Model 39

5.2.3 Moving Average Models 39

5.3 The Least Square Estimation (LSE) 39

5.3.1 Autoregressive Models 40

5.4 Confidence Interval for Mean 40

6 Model Diagnostics 41

6.1 Introduction 41

6.1.1 Residual Analysis 41

7|Page
 TIME SERIES ANALYSIS LECTURE NOTES 2023

6.1.2 Test of Independence 42

6.1.3 Test for Normality 42

6.1.4 Test of Constant Variance 42

6.2 Autocorrelation of Residuals 42

6.2.1 Test for combined residual ACF 42

6.3 Over Fitting 43

7 Forecasting 44

7.1 MME 44

7.1.1 ARMA Model Forecast 44

7.2 Computation of Forecasting 44

7.2.1 Forecast Error and Forecast Error Variance 44

7.3 Prediction Interval 45

7.4 Forecasting AR( p ) and MA( q ) Models 45

7.5 Forecasting ARMA( p, q ) Models 46

7.6 Forecasting ARIMA ( p, d , q ) Models 46

9 APPENDIX A 47

8|Page
 TIME SERIES ANALYSIS LECTURE NOTES 2023

CHAPTER 1
INTRODUCTION TO TIME SERIES

1.1 Introduction
What is time series? A time series may be defined as a set of observations of a random variable
arranged in chronological (time) order. We also say it’s a series of observations recorded sequentially
at equally spaced intervals of time. Let us look at a few examples in order to appreciate what we
mean by a time series.

Example: 1.1.1
The daily temperature recorded over a period of a year. There are a time series because they are
recorded at equally spaced intervals of time and they are recorded regularly.

Example: 1.1.2
The hourly temperature readings of a machine in a factory constitutes a time series. The fact that the
temperature readings are taken every hour makes the temperature readings a time series.

1.2 Why do we need time series?

The aim of time series is “ to identify any recurring patterns which could be useful in estimating future
values of the time series”. Time series analysis assumes that the actual values of a random variable
in a time series are influenced by a variety of enviromentaly forces operating over time.

Time series analysis attempts to isolate and quantify the influence of these different environmental
forces operating on the time series into a number of different components. This is achieved through a
process known as decomposition of the time series.

Once identified and quantified, these components are used to estimate future values of the time
series. An important assumption in time series analysis is the continuation of past patterns into the
future ( i.e the environment in which the time series occurs is stable.)

Notation:

• The time series is denoted by {xt , t ∈ T }where T is the index.

• If T is continuous,we have a continuous time series.

• If T is discrete,we have a discrete time series,and T = ℜ ,the set of all integers. The time series
is some times written as ..., x − 2 , x −1 , x0 , x1 x 2 ,...

9|Page
 TIME SERIES ANALYSIS LECTURE NOTES 2023

• For simplicity, we will drop the index set and write {xt } or {x1 , x 2 , x3 ,...} to indicate that they are
observation.

• In practice, the time series interval for collection of time series could be
seconds,minutes,hours,days,weeks,months,years or any reasonable regular time intervals.

1.3 Time Series plots

The most important step in time series analysis is to plot the observations against time. This graph
should show up important features of the series such as a trend, seasonality, outliers and
discontinuities. The plot is vital, both to describe the data and to help in formulating a sensible model.
This is basically a plot of the response or variable of interest ( x ) against time (t ) , denoted xt .

1.4 General Approach to Time Series Modeling

• Plot the series and examine the main features: This is usually done with the aid of some
computer package e.g. SPSS, SAS, etc.

• Reform a transformation of the data if necessary.

• Remove the components to get stationary residuals by differencing the data.(i.e Replacing the
original series {xt }by y t = xt − xt −1 .

• Choose a model to fit the residuals.

• Do the forecasting

1.5 Component of a time series

One way to examine a time series is to break it into components. A standard approach is to find
components corresponding to a long –term trend, any cyclic behavior, seasonal behavior and a
residual, irregular part.

1.5.1 Trend (T)

A smooth or regular underlying movement of a series over a fairly long period of time. A gradual and
consistent pattern of changes.

10 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Example: 1.5.1 (Trend component)

y = -0.0864x + 381.6
Clothing and softgoods sales
$(m illion)
500 Clothing
1999
450 dollars
Estimated
400 trend
350 Linear
(Estimated
300 trend)
250
Mar Mar Mar Mar Mar Mar Mar
1991 1993 1995 1997 1999 2001 2003

1.5.2 Seasonal variation (S)

Movement in a time series which recur year after in some months or quarters with more less the
same intensity.

Example: 1.5.2 (Seasonal component)

1.5.3 Cyclical variation (C)

Period variations extending over a long period of time, caused by different factors such as cycles,
recession, depression, recovery, etc.

1.5.4 Irregular variation (I)

Variations caused by readily identifiable special events such as elections, wars, floods, earthquakes,
strikes, etc.

11 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Example: 1.5.3 (Irregular component)

1.6 Decomposition of a time series

The main time series analysis is to isolate the influence of each of the four components of the actual
series. The multiplicative time series model is used to analyse the influence of each of these four
components. The multiplicative model is based on the idea that the actual values of a time series xt
can be found by multiplying the trend component (T ) by cyclical component (C ) , by seasonal index
(S ) and by irregular component (I ) . Thus, the multiplicative time series is defined as: x = T × C × S × I .

Another model we can use is the additive model given by: x = T + C + S + I .

1.7 Smoothing methods

Smoothing methods are used in attempting to get rid of the irregular, random component of the
series.

1.7.1 Moving averages:

A moving average (ma) of order M is produced by calculating the average value of a variable over a
set of M values of the series.

1.7.2 Running median:

A running median of order M is produced by calculating the median value of a variable over a set of
M values of the series.

1.7.3 Exponential smoothing:

xˆ t +1 = β xt + (1 − β ) xˆ t

Where xˆ t +1 = the exponential smoothing forecast at time t.

12 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

x̂t = the old forecast.

xt = the actual value (observation) at time t.

1.8 Trend Analysis

The trend in a time series can be identified by averaging out the short term fluctuations in the
series.This will result in either a smooth curve or a straight line.

1.8.1 Methods for trend isolation

(a) The moving average method: Produces a smooth curve.

(b) Regression analysis method: Involves fitting a straight line.

1.8.2 Calculating moving average

• The three –year moving total for an observation x would be the sum of the observation
immediately before x , x itself and the observation immediately after x .

• The three- year moving average would be each of these moving totals divided by 3.

• The five-year moving total for an observation x would be the sum of the two observations
immediately before x , x itself and the two observations immediately after x .

• The five-year moving average would be each of these moving totals divided by 5.

To illustrate the idea of moving averages, let us consider the observations in example 1.8.1

13 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Example: 1.8.1
Australia’s official development assistance (ODA) from 1984-85 until 1992-93 is shown (at current
prices, $ million) in Table 1 .

Table 1

Year ODA($
million)

1984-85 1011

1985-86 1031

1986-87 976

1987-88 1020

1988-89 1195

1989-90 1174

1990-91 1261

1991-92 1330

1992-93 1384

(a) Find the three- year moving averages to obtain the trend of the data.

(b) Find the five-year moving averages for the data.

1.8.3 Regression Analysis

A trend line isolates the trend (T) component only. It shows the general direction in which the series
is moving and is therefore best represented by a straight line. The method of least squares from
regression analysis is used to determine the trend line of best fit.

Regression line is defined by: y = β 0 + β1 x . If the variable x , are not given, must be coded.

Methods for coding the time variable, x

(a) The sequential numbering method.

(b) The zero-sum method.

14 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Coding x using Zero-sum method

 n −1
• To code x when the number of time periods, n is odd, we assign a value of −   to the first
 2 
time period and for each subsequent period, add one to the previous period’s x value.

• To code x when the number of time periods, n is even, we assign a value of − (n − 1) to the
first time period and for each subsequent period, add two to the previous period’s x value.

Example 1.8.2
Table 2

Year 1977 1978 1979 1980 1981 1982 1983

Y 2 6 1 5 3 7 2

a) Calculate the regression line using sequential numbering method.

b) Calculate the regression line using Zero-sum method.

Exercise: 1.8.1
Consider the monthly earning of a small business.

Table 3

Year 1977 1978 1979 1980 1981 1982 1983

Y 12 14 18 17 13 4 17

a) Find the 3 point moving average.

b) Find the least squares trend line for small business using Zero- sum method.

c) Find the least squares trend line for small business using sequential method.

1.9 Seasonal Analysis

Seasonal analysis isolates the influence of seasonal forces on a time series. The ratio-to-moving
average method is used to measure these influences. The seasonal influence is expressed as an
index number.

15 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

1.9.1The ratio-to-moving average method

Step-1

Identify the trend/cyclical movement.

The moving average approach is used to isolate the trend/cyclical movement in a time series.

Step-2

Find seasonal ratio using the formula:

Actual y y
seasonal ratio = × 100% = t × 100%
Moving average series MA
T ×C × S × I
= = S × I × 100%
T ×C

Seasonal ratio is an index which expresses the percentage deviation of each actual y(which includes
seasonal influences) from its moving average value(which contains trend and cyclical influences
only).

Step-3

Average the seasonal ratios across corresponding periods within each year.

The averaging of seasonal ratios has the effect of smoothing out irregular component inherent in the
seasonal ratios. Generally, the median is used to find the average of seasonal ratios for correspond
periods.

Step-4

Compute adjusted seasonal indices. The adjusted factor is determined as follows:

k × 100
Adjusted factor =
∑ (Median seasonal indices )
De-seasonalising the actual time series

Seasonal influences may be removed from a time series by dividing the actual y value for each
period by its corresponding seasonal index.

Actual y
That is, Deseasonalised y = × 100
Seasonal index

16 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Example: 1. 9.1
The average daily sales (in litres) of milk at a country store are shown in Table 4 for each of the years
1983 to 1985.

Table 4

Year Quarter Average daily

sales ( Yt )

1 47.6

2 48.9

1983 3 51.5

4 55.3

1 57.9

1984 2 61.7

3 65.3

4 70.2

1 76.1

1985 2 84.7

3 93.2

4 97.2

(a) Find the four- year moving averages.

(b) Calculate the seasonal index by making use of the ratio-to-moving average method.

Exercise: 1.9.1
Table 5

Year 1993 1994

Quarter 1 2 3 4 1 2 3 4

actuals 10 20 30 16 29 45 25 50

17 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

(a) Find the three- year moving averages.

(b) Calculate an uncentred four –quarter moving average.

(c) Calculate centred moving averages for these data.

(d) Find the adjusted seasonal index for each quarter.

CHAPTER 2
THE MODEL BUILDING STRATEGY

2.1 Introduction
Perhaps the most important question we ask now is “how do we decide on the model to use?”
Finding appropriate models for time series is not an easy task. We will develop a model building
strategy which was developed by Box and Jenkins in 1976.

2.2 The Box-Jenkins Technique

There are three main steps in the Box-Jenkins procedure, each of which may be used several times:

1) Model specification

2) Model fitting.

3) Model diagnostics.

2.2.1 Model specification

In model specification (or identification) we select classes of time series that may be appropriate for a
given observed series. In this step we look at the time plot of the series, compute many different
statistics from the data, and also apply knowledge from the subject area in which the data arise, such
as economics, physics, chemistry, or biology. The model chosen at this point is tentative and may be
revised later in the analysis. In the process of model selection we shall try to adhere to the principle
of parsimony.

Definition 2.2.1 (The principle of parsimony) :The model used should require the smallest possible
number of parameters that will adequately represent the data.

a) Test for white noise

For the data set to be purely random sequence/white noise the sample autocorrelation

18 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

2
ρ̂ k < (i.e the sample ACF must be within the boundaries).
n

Example 2.2.1
200 observations on a stationary series were analyzed and gave the following sample
autocorrelation:

Table 6

k 1 2 3 4 5

ρ̂ k 0.59 0.56 0.46 0.38 0.31

a) Is the data set a white noise?

2.2.2 Model fitting

Model fitting consists of finding the best possible estimates of those unknown parameters within a
given model. After we have identified the model and estimated the unknown parameters we need to
check if the model is a good model, this is done through diagnostic checking.

2.2.3 Model Diagnostics

Here we are concerned with analyzing the quality of the model that we have specified and estimated.
We ask the following questions to guide us:

1) How well does the model fit the data?

2) Are the assumptions of the model reasonably satisfied?

If no in adequacies are found, the modeling may be assumed to be complete, and the model can be
used, for example, to forecast future values of the series. Otherwise we choose another model in light
of the inadequacies found: that is we return to model specification. In this way we cycle through the
three steps until an acceptable model is found.

2.3 Stationary time series

Definition 2.3.1: A time series is said to be stationary if there is no systematic change in mean (no
trend), if there is no systematic change in variance, and if strictly periodic variations have been
removed.

It should be noted that in real life it is not often the case that a stochastic process is stationary. This
could arise due to, for example,

1) Change of policy on the process.

19 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

2) Slump on the process.

3) Improvement on the process, e.t.c.

2.3.1 Transformations
If the process is not stationary to analyze its series we must transform the series to be stationary.
There are various transformations that we can use to make a time series stationary. Some of them
are:

1) Differencing.

2) Log transformation.

3) Square root transformation.

4) Arcsine transformation.

5) Power transformation.

The three main reasons for making a transformation are as follows:

(a) To stabilize the variance:

If the variance of the series increases with the mean in the presence of trend, then it is advisable to
transform the data. A logarithmic transformation is appropriate if the standard deviation is directly
proportional to the mean.

(b) To make the seasonal effect additive:

If size of the seasonal effect appears to increase with the mean in the presence of the trend, then it is
advisable to transform the data so as to make the seasonal effect ADDITIVE. If the size of the
seasonal effect is directly proportional to the mean, then the seasonal effect is said to be
Multiplicative and the logarithmic transformation is appropriate to make the seasonal effect additive.

(c) To make the data normally distributed:

Often the general class of transformations called the Box-Cox transformation given by




Yt = 
( λ
)
xt − 1
,λ ≠ 0
 λ
 log xt ,λ =0

For some transformation parameter λ. the logarithmic and square root transformations are special
cases of this general class.

20 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

2.3.2 Stationarity through differencing

Models that are not stationary when subjected to differencing often yield processes that are
stationary. Thus if we difference a time series we denote it by ∇ xt = xt − xt −1 . We will use the operator
∇ to denote the difference operation. In some instances, differencing ones may not yield a stationary
process.

In that situation we continue differencing the series until it is stationary. Once the process is
stationary there is no need to continue differencing it otherwise we will over difference it.

Example 2.3.1(Second order differences)

∇ 2 xt = ∇(∇xt ) = xt − 2 xt −1 + xt − 2

Exercise 2.3.1
Suppose we have a process given by xt = 5 + 2t + z t where z t is white noise with mean zero and
variance σ z2 .

1) Show that xt is not stationary.

2) Verify that the process is now stationary if we difference it once.

2.4 Analyzing Series Which Contain a Trend

2.4.1 Filtering
Definition 2.4.1: A linear filter is a linear transformation or any operator which converts one time
series, {xt }called the input or leading indicator series into another series {y t }called the output series

through the linear operation y t = ∑ a j xt + j or schematic:

j

xt → Filter → y t

Note that filter can be in a series,

z t = ∑ b j yt + j
j

= ∑ b j ∑ a r xt + j + r
j r

= ∑ c k xt + k
k

where c k = ∑ a r b(k − r )
r

21 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

, are the weights for the overall filter. The weights {c k } are obtained by a procedure called
convolution and symbolically expressed as {c k } = {a r }* {b j }

Example: 2.4.1
1 1  1 1 
(a) Consider the following two filters: A =  ,  , B =  ,  Compute A * B where *denotes the
2 2 2 2
convolution operator.

(b) Consider the following two filters: A = {− 1,1}, B = {− 1,1}Compute A * B where *denotes the
convolution operator.

Exercise: 2.4.1
Calculate the convolution of the following filters:

1 1 1 1 1 1 1 1
a)  , , ,  *  , , , 
4 4 4 4 4 4 4 4

1 1 1 1 1 1 1
b)  , , ,  *  , , 
 4 4 4 4 3 3 3

2.5 Stochastic processes

Definition 2. 5.1: A time series x0 , x1 ,... is a sequence of observation. More technically a time series
is a sample path or realization of a stochastic (random) process {xt , t ∈ T }where T is an ordered set.

Definition 2.5.2: Let {xt , t ∈ T }be a stochastic process and let ε be a set of all possible realization or
sample of path then ε is called the ensemble for the process {xt }. An individual time series is a
member of ensemble.

Remarks: In time series literature, the terms “time series” and process are used (often)
interchangeably.

2.6 Means, Variances and Covariances

The mean and autocovariance function (ACVF) are given by:

µ t = Ε[xt ] and γ t + k = Ε[(xt − µ t )(xt + k − µ t + k )] respectively.

The variance function σ t is defined by var[xt ] = σ t

2 2

22 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

CHAPTER 3
MODELS FOR STATIONARY TIME SERIES

3.1 Introduction
In order to be able to analyze or make meaningful inference about the data generating process it is
necessary to make some simplifying and yet reasonable assumption about the process. A
characteristic feature of time series data which distinguishes it from other types of data is that the
observations are, in general, correlated or dependent and one principal aim of time series is to study,
investigate, explore and model this unknown correlation structure.

3.1.1 Strictly Stationary Processes

Definition 3.1.1: A time series {xt }is said to be strictly stationary if the joint density functions
depend only on the relation location of the observations, so that f ( xt1+ h , xt 2+ h ,..., xtk + h ) = f ( xt1 , xt 2 ,..., xtk ).
meaning that ( xt1+ h , xt 2+ h ,..., xtk + h ) and (xt1 , xt 2 ,..., xtk ) have the same joint distributions for all h and for
all choices of the time points {t i }.

Example: 3.1.1
Let n=1, the distribution of xt is strictly stationary if µ t = µ and σ t = σ are both constant. And if n=2,
2

the joint distribution of xt 1 and xt 2 depend only on (t 2 − t1 ), which is called the lag. Thus the
autocovariance function γ (t1 ,t 2 ) depends only on (t 2 − t1 ) and is written as γ (k ) , where

γ (k ) = Ε{[xt − µ ][xt + k − µ ]}
And γ (k ) is called the autocovariance coefficient at lag k.
= cov( xt , xt + k )

3.1.2 Weakly Stationary Processes

Definition 3.1.2: A stochastic process {z t }is weakly stationary (or of second order stationary), if both
the mean function and the autocovariance function do not depend on time t .thus,

µ t = Ε[xt ] = (a constant) and γ t + k = γ k = (a constant).Note that γ t = k = cov( xt , xt + k )

Example: 3.1.2
Prove or disprove the following process is covariance stationary:

(a) z t = (− 1) A , where A is a random variable with zero mean and unit variance?
t

23 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Exercise: 3.1.1
Consider the following time sequence xt = z0 cos(ct ) .

Where {z0 } is a sequence of independent normal r.v with mean 0 and variance σ z2 .

Show that {x t } is not a covariance stationary.

3.2 Autocorrelation function of stationary processes

Definition 3.2.1: Suppose a stationary stochastic process x(t ) has mean µ , variance σ 2 and ACVF
γ (k ) γ (k )
γ (k ) , then the autocorrelation function (ACF) is given by: ρ (k ) = =
γ (0 ) σ 2

Note that γ (0 ) is the variance of the series given by σ 2 and ρ (0 ) = 1 .

Properties

1. The ACF is an even function, so that ρ (k ) = ρ (− k ) i.e. the correlation between x(t ) and x(t + k ) is the
same as the correlation between x(t ) and x(t − k ) .

2. ρ (k ) ≤ 1

3. The ac.f does not uniquely identify the underlying model. That is why invertibility has to be checked
in moving average processes.

Exercise: 3.2.1
Prove the following:

a) ρ (k ) = ρ (− k )

b) ρ (k ) ≤ 1

3.3 Purely random process

A very important example of a stationary process is the so-called white noise process. One simple
definition is that a white noise is a (univariate or multivariate) discrete-time stochastic process whose
terms are independent and identically distributed (IID), with zero mean. While this definition captures
the spirit of what constitutes a white noise, the IID requirement is often too restrictive for applications.
Typically the IID requirement is replaced with a requirement that terms have constant second
moments, zero autocorrelations and zero means. Let’s formalize this.

24 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Definition 3.3.1: A discrete time process is called a purely random process (white noise) if it
consists of a sequence of random variable {z t } which are mutually independent and identically
distributed. It follows that both mean and variance are constants and the acv.f. is:
γ (k ) = cov(z t , z t + k )
= 0 for k = ±1,2...

1 , for k = 0
The ac.f.is given by ρ (k ) = 
0 , for k = ±1,±2,...

Definition 3.3.2 (IID Noise Process)

A process {xt } is said to be an IID noise with mean 0 and variance σ x2 ,written {xt }~ IID(0, σ x2 ) if the
random variable xt are independent and identically distributed with Ε( xt ) = 0 and var( xt ) = σ x2 .

3.4 Random walk

Let z1 , z 2 ,... be independent identically distributed random variables, each with mean 0 and variance
σ z2 . The time series that can be observed {xt } is called a random walk if it can be expressed as
follows:

x1 = z1
x 2 = z1 + z 2
x3 = z1 + z 2 + z 3


xt = z1 + z 2 + z 3 +  + z t .................(∗)

If z' s are interpreted as the size of ‘steps’ taken forward or backward at time t , then xt is the position
of a ‘random walk’ at time t .

From (∗) we obtain the mean function: µ t = Ε( xt ) = 0 , variance of xt , var( xt ) = t σ z2 and the
covariance of xt and x s , γ t , s = cov(xt , x s ) = t σ z2 .

Backshift operator
The backshift operator is used to express and manipulate time series models. The backshift operator
denoted B on the time index of a series and shifts time back 1 time unit to form a new series i.e:
B( xt ) = xt −1 .

25 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

3.5 Moving average processes

Mathematically , we express a first order moving average model MA(1) as xt = z t − θ1 z t −1 ,where
(
z t ~ N 0, σ z
2
) based on the notation in the text book a first order MA(1) process is given by
xt = θ 0 z t + θ z ,where z ~ N (0, σ ).Clearly the two model are the same with θ = 1 and θ = −θ
2
1 t −1 t z 0 1 1

.usually in most practical situations θ 0 is standardized so that it is equal to one.

xt ⇒ moving average ⇒ z t , z t −1
Schematically, a moving average of order 1 can be expressed as
operator ( filter )

3.5.1 First- order moving average [MA (1)] process

The MA (1) for the actual data, as opposed to deviations to deviation from the mean will be written as

xt − µ = z t − θ1 z t −1 Or xt = µ + z t − θ1 z t −1 where µ is the mean of the series corresponding to the

intercept in the moving average case.

Example: 3.5.1(Backshift operator)

Consider an MA(1) model. In terms of B, we can write xt = z t − θ z t −1 = z t − θ B z t = (1 − θ B ) z t .

Where θ (B ) is the MA characteristic polynomial “evaluated at B”.

3.5.2 Second –order moving average [MA (2)] process

Invertible conditions

The second order moving average process is defined by xt = z t − θ1 z t −1 − θ 2 z t − 2 and is stationary for all
values of θ1 and θ 2 .However , it is invertible only if the roots of the characteristic equation
1 − θ1 B − θ 2 B 2 = 0 lie outside the unit circle, that is ,

(i ) θ 2 + θ 1 < 1
(ii ) θ 2 − θ 1 < 1
(iii ) − 1 < θ 2 < 1

Example: 3.5.2
Find the variance and ACF of the following process:

xt = z t − θ1 z t −1 − θ 2 z t − 2

26 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Solution:

(
(a) The variance of the process is γ (0 ) = σ z 1 + θ 21 + θ 2 .
2 2
)
− θ1 (1 − θ 2 )
(b) The ACF’s are ρ1 =
1 + θ1 + θ 2
2 2

−θ2
ρ2 =
1 + θ1 + θ 2
2 2

ρ3 = 0 for k ≥ 3

Exercise: 3.5.1
Find the ACF of the following first order MA processes:

(a) xt = z t + θ1 z t −1

1
(b) xt = z t + z t −1 ,where z t is a white noise.
θ

3.5.3 qth-order moving average [ MA (q) ] process.

Definition 3.5.1: Suppose that {z t } is a purely random process, such that E ( z t ) = 0, var( z t ) = σ z .then
2

a process {xt }is said to be a moving average process of order q (abbreviated as MA (q)) if
xt = z t + θ1 z t −1 + ... + θ q z t − q .Where {θ i }are constants.

Example: 3.5.3
Consider a MA (q) given by xt = z t + θ1 z t −1 + ... + θ q z t − q .

Find ACVF and ACF of xt .

Exercise: 3.5.2
Find the ACF of the first-order moving average given by xt = z t − θ1 z t −1 .

27 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

3.6 Partial autocorrelation function

Definition 3.6.1: The partial correlation (PACF) is defined as the correlation between xt and xt + k with
their linear dependency on the intervening variables xt + k ,..., xt + k −1 removed,
Φ kk = corr (x t , x t + k x t +1 ,..., x t + k −1 ) .

ρ 0 ρ1 ρ1
ρ1 ρ 0 ρ 2
Φ 11 = ρ1
ρ 2 ρ1 ρ 3
, Φ 33 =
ρ 0 ρ1 ρ 2
ρ0 ρ1
ρ1 ρ 0 ρ1
ρ ρ2
Φ 22 = 1 ρ 2 ρ1 ρ 0
ρ0 ρ1
ρ1 ρ0

Exercise 3.6.1
Find the Φ 11 , Φ 22 , Φ 33 of xt = z t + θ1 z t −1 .

ˆ kk = 1
Definition 3.6.2: Standard error for Φ
n

3.7 Autoregressive processes

Definition 3.7.1: Let {z t } be a purely random process with mean zero and variance σ z . Then a
2

process {xt }is said to be an autoregressive process of order p if xt = φ1 xt −1 + ... + φ p xt − p + z t .

An autoregressive process of order p will be abbreviated as AR ( p ).

3.7.1 First -order Autoregressive (Markov) process

Example: 3.7.1
Consider a model xt = φ1 xt −1 + z t where xt is a white noise.

a) Find the ACVF and ACF .

Solution:

The first-order autoregressive process is xt = φ1 xt −1 + z t where φ1 must satisfy the following condition:
− 1 < φ1 < 1 for the process to be stationary.

28 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Using the backward shift operation B, the process xt = φ1 xt −1 + z t may be written as (1 − φ1 B )xt = z t so
−1
( 2
)
that xt = (1 − φ1 B ) z t = 1 + φ1 B + φ1 B 2 + ... Z t = z t + φ1 z t −1 + φ1 z t − 2 + ...
2

Ε[xt ] = 0

2
(
var(xt ) = σ z 1 + φ1 + φ1 + ...
2 4
)
σ z2
Thus the variance is finite provided that φ1 < 1 , in which case var( xt ) = σ x =
2
.
1 − φ1
2

 a
i.e s ∞ = 1 − r , where

 φ1 4 φ1 2
= = = φ1
2
 r
 φ1 2
1
and a = 1



The ACVF is given by γ (k ) = Ε[xt xt + k ] = Ε [(∑ φ 1

i
z t −i )(∑ φ
1
j
)]
zt +k + j = σ z
2
∑φ
1
i
φ1 k +i for k ≥ 0

φ1 k σ z 2
This converges for φ1 < 1 to γ (k ) = = φ1 σ x
k 2

(
1 − φ1 )
2

For k < 0, we find γ (k ) = γ (− k ) .

The ACF is given by ρ (k ) = φ1 for k = 0,1,2,...

k

∀ int erger k , ρ (k ) = φ1 for k = 0 ± 1,±2,...

k

⇒ ρ (0 ) = 1
ρ (1) = φ1

Exercise: 3.7.1
Consider a model xt = 0.7 xt −1 + z t where z t is a white noise.

Find : ACV.F and ACF of xt .

3.7.2 Second – order autoregressive process.

Stationary condition
29 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

The second-order autoregressive process may be written as:

xt = φ1 xt −1 + φ 2 xt − 2 + z t .

For stationarity, the roots of φ (B ) = 1 − φ1 B − φ 2 B 2 = 0 must lie outside the unit circle, which implies that
the parameters φ1 and φ 2 must lie in the triangular region:

(i ) φ 2 + φ1 < 1
(ii ) φ 2 − φ1 < 1
(iii ) − 1 < φ 2 < 1

3.7.3 pth-order autoregressive process

( )
xt = φ1 xt −1 + φ 2 xt − 2 + ... + φ p xt − p + z t or 1 − φ1 B − φ 2 B 2 − ... − φ p B p xt = z t

Stationary conditions:

The roots Bi , i = 1,2..., p of the characteristic equation 1 − φ1 B − φ 2 B 2 − ... − φ P B p = 0 must lie outside the
unit circle.

3.8 The dual relation between AR & MA processes

Table 7

Process ACF PACF Stationary Invertible

condition condition

AR( p ) Damps out Cuts off after lag- Roots of Always

p characteristic invertible
equation
outside unit
circle.

MA( q ) Cuts off after lag- Damps out Always Roots of

q stationary characteristic
equation
outside unit
circle.

ARMA( Damps out Damps out Roots of Roots of

p, q ) characteristic characteristic
equation equation
outside unit outside unit
circle. circle.

30 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Example: 3.7.2
Theoretical behavior of the ACF and PACF for AR(1) and AR(2) models:
AR(1): PACF = 0 for lag > 1 ; ACF → 0 AR(2 ): PACF = 0 for lag > 2 ; ACF → 0

In this context………..
• “damps out/die out” means” tend to zero gradually”
• “cuts off” means ”disappear” or “is zero”.

Example: 3.7.3
Theoretical behavior of the ACF and PACF for MA(1) and MA(2) models:
MA(1): ACF = 0 for lag > 1 ; PACF → 0 MA(2 ): ACF = 0 for lag > 2 ; PACF → 0

31 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

3.9 Yule-Walker equation

Yule-Walker equation using the fact that ρ (k ) = ρ (− k ) for all k is given by
ρ (k ) = φ1 ρ (k − 1) + ... + φ p ρ (k − p ) , for all k > 0

The general solution is ρ (k ) = A1π 1 + ... + A p π p where {π i },are the roots of the auxiliary equation:
k k

y p − φ1 y p −1 − ... − φ p

{Ai } Are constant, ρ (0) = 1 and ∑ Ai =1

The first ( p − 1) Yule-Walker equations provide ( p − 1) further restrictions on the {Ai } using
ρ (0) = 1 and ρ (k ) = ρ (− k ) .From the general form of ρ (k ) , it is clear that ρ (k ) tends to zero as k
increases provided that π i < 1 for all i , and this is a necessary and sufficient condition for the process
to be stationary.

For stationary, the roots of the equation φ (B ) = 1 − φ1 B − ... − φ p B p = 0 must lie outside the unit circle.

Example: 3.9.1
Suppose we have AR (2) process, when π 1 , π 2 are the roots of the quadratic equation y 2 − φ1 y − φ 2 = 0

, thus π i < 1 if φ1 ±
(φ
1
2
+ 4φ 2 ) < 1.
2

When roots are real, the constants A1 , A2 are found as follows:

32 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Using Yule-Walker equations for AR (2), ρ (k ) = φ1 ρ (k − 1) + φ 2 ρ (k − 2 ) we have

ρ (1) = φ1 ρ (0) + φ 2 ρ (− 1)
φ
= 1 = A1π 1 + A2π 2
1 − φ2
∴ ρ (1) = A1π 1 + (1 − A1 )π 2

 φ1 
  −π2
 1 − φ 2 
Where A1 = and A2 = 1 − A1
π1 − π 2

Example: 3.9.2
1
Consider the AR (2) process given by x t = − x t −1 − xt −2 + z t
4

Show that x t is stationary and then calculate its ACF.

Exercise: 3.9.1
1 2
Consider the AR (2) process given by xt = xt −1 + xt − 2 + z t
3 9

Is this process stationary? If so, what is its ACF?

______________________________________________________________________

Note: for real –valued linear different, a complex root of φ (B ) = 0 must appear in pairs.

That is, if (c + di ) is a root, then its complex conjugate (c + di ) = (c − di ) is also a root. A general
*

complex number can always be written in polar form, i.e

(c ± di ) = α (cos π ± i sin π ) ⇒ (c ± di ) = α k (cos πk ± i sin πk )

k

Where α = c 2 + d 2 = (c 2 + d 2 )2 And β = tan −1   .

1
d 
c

________________________________________________________________________________

3.10 Mixed ARMA models

Definition 3.10: A mixed autoregressive/moving –average process containing p AR terms and q MA
terms is said to be an ARMA process of order (p, q) and is given by:
xt = φ1 xt −1 + ... + φ p xt − p + z t + θ1 z t −1 + ... + θ q z t − q .

33 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

The backshift operator B, is given by φ (B )xt = θ (B )z t where φ (B ), θ (B ) are polynomials of order p, q

respectively, such that φ (B ) = 1 − φ1 B − ... − φ P B p and θ (B ) = 1 + θ 1 B + ... + θ q B q .

Stationary process

The roots of φ (B ) = 0 must lie outside the unit circle.

Invertible process

The roots of θ (B ) = 0 must lie outside the unit circle.

3.10.1 ARMA (1, 1)

xt − φ1 xt −1 = z t − θ1 z t −1 , using a backshift operator we have (1 − φ1 B )xt = (1 − θ1 )z t .

Stationary and invertibility conditions

The process is stationary if − 1 < φ1 < 1 , and invertible if − 1 < θ1 < 1 .

Example: 3.10.1
1
Consider the following process: xt − xt −1 = z t + 2 z t −1 .
2

a) Is the process stationary/invertible?

b) Find the AC.F

3.10.2 Weights (ψ , π )
From the relations ψ 1 = φ1ψ 0 − θ1 = φ1 − θ1 andψ j = φ1ψ j −1 for j > 1 , we find that the ψ j weights are
given by ψ j = (φ1 − θ1 )φ1 , for j ≥ 1, and similarity it is easily seen that π j = (φ1 − θ1 )θ1
j −1 j −1
, for j ≥ 1 ,for
the stationary and ARMA (1, 1) process.

The ψ weights or π weights may be obtained directly by division or by equating powers of B in an

equation such asψ (B )φ (B ) = θ (B ) .

Example: 3.10.2
Find the ψ weights and π weights for the ARMA (1, 1) process given by xt = 0.5 xt −1 + z t − 0.3 z t −1 .

34 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Exercise: 3.10.1
1
Consider the ARMA (1,1) process given by xt = xt −1 + z t + z t −1
2

(a) Is this process stationary/invertible? (b) Calculate the ACF.

Exercise: 3.10.2
Obtain the first 3π -weights and 3ψ -weights of the following models:

a) (1 − φ1 B − φ 2 B 2 ) xt = (1 − θ B ) z t .

( )
b) 1 − 3B + B 2 xt = z t

3.11 Seasonal ARMA models

In other situations we may have data with a seasonal component. In order to fit a model we need to
take this seasonality component into consideration.

Definition 3.11.1: A process {xt } is called a seasonal ARMA process of non seasonal order p, q and
seasonal component P, Q and a seasonality order S if {xt } satisfies φ (B ) Φ (B ) xt = θ (B ) Θ(B ) z t .

Where,

Φ (B ) = 1 − Φ B S − Φ 2 B 2 S −  − Φ P B PS
φ (B ) = 1 − φ B − φ 2 B 2 −  − φ p B p
θ (B ) = 1 − θ 1 B − θ 2 B 2 −  − θ q B q
Θ(B ) = 1 − Θ1 B S − Θ 2 B 2 S −  − Θ Q B QS

CHAPTER 4
MODEL FOR NON STATIONARY SERIES

4.1 ARIMA models

A series xt is said to follow Integrated Autoregressive- Moving Average (ARIMA) model if the d th
difference wt = ∇ d xt is stationary ARIMA process. If wt is ARMA( p, q ) , we say that xt is
ARIMA( p, d , q ) . For practical purpose we usually take d to be at most 2.

Note: If we want to check whether an ARMA( p, q ) is stationary we check the AR( p ) part only.

35 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Let xt = φ1 xt −1 + φ 2 xt − 2 +  + φ p xt − p + z t + θ1 z t −1 + θ 2 z t − 2 +  + θ q z t − q be a non stationary process.

Writing wt = ∇ d xt = (1 − B ) xt the general autoregressive integrated moving average process (ARIMA)
d

is of the form: wt = φ1 wt −1 + ... + φ p wt − p + z t + θ1 z t −1 + ... + θ q z t − q

Using back shift operator B, we have φ (B )wt = θ (B )z t or φ (B )(1 − B ) xt = θ (B )z t .thus we have an

d

ARMA ( p, q ) model for wt , while the model in equation above, describing the dth differences of xt , is
said to be an ARIMA process of order ( p, d , q ) .The model for xt is clearly non-stationary, as AR
operator φ (B )(1 − B ) had d roots on the unit circle. If the value of d is taken to be one, the random
d

walk can be regarded as an ARIMA (0,1,0 ) process.

Exercise: 4.1.1
Identify the following models

a) (1 − 0.8 B )(1 − B ) xt = z t .

b) (1 − B ) xt = (1 − 0.75)z t .

c) (1 − 0.9 B )(1 − B ) xt = (1 − 0.5B ) z t

4.2 Non stationary seasonal processes

4.2.1. SARIMA model
Definition 4.2.1: A process {xt } is called a seasonal ARIMA process of non-seasonal component
p, d , q and seasonal components P, D, Q if xt satisfies φ (B ) Φ (B )∇ d ∇ sD xt = θ (B ) Θ(B ) z t .

Φ (B ) = 1 − Φ B S − Φ 2 B 2 S −  − Φ P B PS
φ (B ) = 1 − φ B − φ 2 B 2 −  − φ p B p
Where:
θ (B ) = 1 − θ 1 B − θ 2 B 2 −  − θ q B q
Θ(B ) = 1 − Θ1 B S − Θ 2 B 2 S −  − Θ Q B QS

Example: 4.2.1
Let consider a time series where a period consists of S seasons (for monthly data S = 12 , for
quarterly data S = 4 ,etc.)
Suppose a non-seasonal ARIMA model is fitted to the series i.e φ p (B )(1 − B ) xt = θ q (B )at ....... ∗
d

where at is a white noise. This series can also be represented as ARIMA model

( )(
ΦP BS 1− BS )D
( )
at = Θ Q B S bt .............. ∗ ∗

36 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

( ) ( )
where Φ P B S = 1 − Φ 1 B S − Φ 2 B 2 S − ... − Φ P B PS and Θ Q B S = 1 − Θ1 B S − Θ 2 B 2 S − ... − Θ Q B QS

( ) ( )
It is assumed that the polynomials Φ P B S and Θ Q B S have no common roots and that their roots
lie outside the unit circle.

combining ∗ and ∗ ∗ gives the multiplicative seasonal ARIMA model.

( ) (
Φ P B S φ P (B )(1 − B ) 1 − B S
d
)
D
( )
xt = θ q (B )Θ Q B S at .This model is denoted as ARIMA( p, d , q ) × (P, D, Q )S

Example: 4.2.2
Let us consider the ARIMA(0,1,1) × (0,1,1)12 model. Where Wt = (1 − B )(1 − B 12 ) xt = (1 − θB )(1 − ΘB 12 ) z t .

Find the autocovariance and the autocorrelations of Wt .

CHAPTER 5
PARAMETER ESTIMATION

5.1 Introduction
Having tentatively specified ARIMA( p, d , q ) the next step is to estimate the parameters of this model.
This chapter focuses on estimation of parameters of an AR and MA models. We shall deal with the
most commonly used method of estimating parameters, these are:

1) Method of moments.

2) Least square method

3) Maximum-likelihood method.

5.2 Method of moments

This method consist of equating sample moments such as the sample mean x ,sample variance γ 0
and sample autocorrelation function to the theoretical counterparts and solving the resultant
equation(s).

5.2.1 Mean
With only a single realization (of length n ) of the process, a natural estimator of the mean, µ is the
1 n
sample mean x = ∑ xt .where x is the average time average of n observation.
n t =1

37 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

1 n
Since E ( x ) = ∑ E (xt ) = µ , x is an unbiased estimator of µ
n t =1

If {xt } is a stationary process with autocorrelation function ρ (k ) then

γ (0)  n −1
 k 
var( x ) = 1 + 2∑ 1 −  ρ (k )
n  k =1  n 

ESTIMATION OF γk and ρk

Suppose that we have n observations x1 , x2 ,, xn then the corresponding sample autocovariance
and autocorrelation functions (or estimates) at lag k are:
n−k

1 n−k ∑ (x t − x )( xt + k − x )
γˆk = ∑ (xt − x )(xt + k − x ) and ρ̂ k = t =1
n

∑ (x − x)
n t =1 2
t
t =1

n −1

1 n −1 ∑ (x t − x )( xt +1 − x )
As an example, γˆ1 = ∑ ( xt − x )( xt +1 − x ) and ρ̂1 = t =1
n
are used to estimate γ 1 and ρ1 .
∑ (x − x)
n t =1 2
t
t =1

Example: 5.2.1a
Table 8

t 1 2 3 4

xt 4 5 2 5

Estimate ρ1 , ρ 2 ρ3 ,and ρ 4 for the time series given in table 8.

Example: 5.2.1b
Suppose that {xt } is a MA(1) process defined by xt = z t + θ z t −1 where θ is a fixed parameter and
( )
z t ~ IID 0, σ Z . Where IID stand for Independent Identically Distributed noise.
2

Calculate var( x ) in terms of θ .

Exercise: 5.2.1
38 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Suppose that {xt } is moving average process given by xt = (z t + z t −1 + z t −2 ) where {z t } is zero mean
1
3
white noise with var( z t ) = σ z2 . Calculate var( x ) .

5.2.2 Autoregressive model

Example: 5.2.2
Consider an AR(1) model: xt = φ1 xt −1 + z t Find the estimate of φ1 using the method of moments.

Exercise: 5.2.2
Consider an AR(2 ) model: xt = φ1 xt −1 + φ 2 xt − 2 + z t Find the estimates φ1 and φ 2 using the method of
moments.

5.2.3 Moving average models

Method of moments is not convenient when applied to moving average models. However, for
purposes of illustration, we shall consider MA(1) process given in the following example:

Example: 5.2.3
Consider an MA(1) model: xt = z t − θ1 z t −1 Find the estimate of θ1 using the method of moments.

5.3 The Least Square Estimation (LSE)

The method of Least Square Estimation is an estimation procedure developed for standard
regression models. In this section we discuss LSE procedure and its associated problems in time
series analysis.

Recall: For sample linear regression model given by: y t = φ xt − z t , t = 1,2,..., n

n∑ xt y t − ∑ xt ∑ y t
The Least Square Estimate is given by: φˆ =
n∑ xt2 − (∑ xt )
2

The estimate φ is a consistant and best linear unbiased estimator of φ . This holds under the
following basic assumptions on the error term z t :

1) Zero mean: Ε( z t ) = 0 .

2) Constant variance: Ε( z t ) = σ z2 .
2

39 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

3) Non –autocorrelation: Ε( z t z k ) = 0 for t ≠ k .

4) Uncorrelated with explanatory variable: Ε( xt z t ) = 0

In the next subsection we shall apply the LSE method to time series model.

5.3.1 Autoregressive models

Example: 5.3.1
Consider an AR(1) model: xt = φ1 xt −1 + z t ….*

Model *can be viewed as a regression model with predictor variable xt −1 and responded variable xt .
On LSE method we minimize the sum of squares of the difference xt − xt −1 that is
S (φ ) = ∑ z t2 = ∑ ( xt − φ xt −1 )
2

d S (φ )
Consider the minimization of S (φ ) with respect to φ we have = −2∑ [xt − φ xt −1 ] xt −1 ............... * *
d (φ )

[ ]
Setting this equal to zero and solving for φ yields − 2∑ xt − φˆ xt −1 xt −1 = 0 ⇒ φˆ =
∑x t −1 xt
.
2
x t −1

Example: 5.3.2
Consider an AR(1) model: xt − µ = φ ( xt −1 − µ ) + z t .

Find the estimates µ and φ using the LSE method.

Exercise: 5.3.1
Consider an AR(2 ) model: xt − x = φ1 ( xt −1 − x ) + φ 2 ( xt − 2 − x ) + z t Find the estimates φ1 and φ 2 using
the LSE method .

5.4 Confidence interval for mean

Suppose that xt = µ + z t where µ is constant, z t ~ N (0, γ 0 ) and z t is a stationary process. Under
 γ  n −1
 k  
these assumptions, xt ~ N (µ , γ 0 ) and {xt }is stationary , therefore X ~ N µ , 0 1 + 2 ∑ 1 −  ρ k  
 n  k =1  n  

40 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

If γ 0 and the ρ k ' s are known then a 100(1 − α ) percent confidence interval for µ is

γ0   k  α
X ± zα 1 + 2∑ 1 −  ρ k  . Where z α is upper quantile from the standard normal distribution.
2
n   n  2
2

γ0
Note that if ρ k = 0 for all k ,then this confidence interval formula reduces to X ± z α .
2
n

Example 5.4.1
Suppose that in a sample of size 100 from AR(1) : x t = φ1 x t −1 + z t process with mean µ ; φ = 0.6 and
σ 2 = 2 we obtain X 100 = 0.271

(a) Construct an approximate 95% confidence interval for µ .

(b) Are the data computable with the hypothesis that µ = 0 ?

Exercise 5.4.1
Suppose that in a sample of size 100 from MA(1) process with mean µ ; φ = −0.6 and σ 2 = 1 we obtain
X 100 = 0.157 .

a) Construct an approximate 95% confidence interval for µ .

b) Are the data computable with the hypothesis that µ = 0 ?

CHAPTER 6
MODEL DIAGNOSTICS

6.1 Introduction
Model diagnostics is primarily concerned with testing the goodness of fit of a tentative model. Two
complementary approaches are: Analysis of residuals from fitted models and analysis of over
parameterized model will be considered in this chapter.

6.1.1 Residual Analysis

Before a model can be used for inference the assumptions of the model should be assessed using
residuals. Recall from regression analysis, residuals are given by :

Residual=Actual Value-Predicted Value.

41 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

Residual can be used to assess if the ARMA is adequate and if the parameter estimates are close to
the true values. Model adequacy is checked by assessing whether the model assumptions are
satisfied.

The basic assumption is that , {z t } are white noise .That is they possess the properties of
independence, identically and normally distributed random variables with zero mean and constant
variance σ z2 .

A good model is one with residuals that satisfy these properties, that is, it should have residuals
which are:

1) Independent (uncorrelated errors),

2) Normally distributed and

3) Constant variance.

6.1.2 Test of independence

A test of independence can be performed by:

-Examining ACF: Compute the sample ACF of the residual. Residuals are independent if they do not
form any pattern and are statistically insignificant, that is, they are within Z α standard deviation.
2

6.1.3 Test for normality

Test of normality can be performed by:

-Constructing Histogram: Gross normality can be assessed by plotting histogram of residuals.

Histogram of normally distributed residuals should approximately be symmetric and bell shaped.

6.1.4 Test of constant variance

Test of constant variance can be inspected by plotting the residuals over time. If the model is
adequate we expect the plot to suggest a rectangular scatter around zero horizontal level with no
trends whatsoever.

6.2 Autocorrelation of residuals

The basic idea behind the ARIMA modeling is to account for any autocorrelation pattern in the series
{xt } with a parsimonious combination of AR and MA terms, leaving random terms {z t }as a white
noise. If the residuals are white noise this implies that they are correlated, that is, they are serial
independent. To determine if the residuals ACF are significantly different from zero we use the
following Portmanteau test.

42 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

6.2 1 Test for combined residual ACF: Portmanteau test

This test uses the magnitude of residual autocorrelations as a group to check for model adequacy.

The test is as follows:

Hypothesis

H 0 : ρ1 = ρ 2 = ... = ρ k = 0 [mod el is correct ]

H 1 : ρ k ≠ 0 for atleast one k = 1,2,..., K [mod el is incorrect ]

Here we use the modified Box-Pierce Statistic ( Q).

K
Test statistic: Q = N ∑ ρ̂ k2
k =1

where N is the number of terms in the differenced series.

K is the number of lags we wish to use in the test.

k denote the autocorrelation coefficient at lag k.

Decision rule: Reject H 0 if Q > χ k2,1−α and conclude that the random term {z t } from the estimated
model are correlated and that the estimated model may be inadequate.

Remark: If a correct ARMA( p, q ) model is fitted, then Q should be approximately distributed as χ 2

with k − p − q degree of freedom. Where p, q are numbers of AR and MA terms, respectively in the
model.

Note: The maximum lag K is taken large enough so that the weights are negligible for j > K .

Alternative test: Ljung-Box test, Q * = N ( N + 2 )∑ (ρˆ k2 / ( N − k ))

K

k =1

Example: 6.2.1
From example 5.2.1a, calculate the value of Q if k = 1 .

Example: 6.2.2
The AR(2 ) model xt = C + α 1 xt −1 + α 2 xt − 2 + z t was fitted to a data set of length 121.

24
a) The Box-Pierce statistic value was Q = 121∑ ρˆ k2 = 31.5
k =1

At 95% level of significance, test the hypothesis that AR(2 ) model fit the data.

43 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

b) The parameter estimates are αˆ 1 = 1.5630 , αˆ 2 = −0.6583 and Cˆ = 0.4843

The last four values in the series were 707, 6.90, 6.63, 6.20 .

Use AR(2 ) model to predict the next observation value.

6.3 Over fitting

Another basic diagnostic tool is that of over fitting .In this diagnostic check we add another coefficient
to see if the model is better. Recall that any ARMA( p, q ) model can be considered as a special case
of a more general ARMA model with the additional parameters equal to zero. Thus, after specifying
and fitting our tentative model, we fit a more general model that contains the original model as a
special case.

There is no unique way to over fit a model, but one should be careful not to add coefficients to both
sides of the model. Over fitting both AR and MA terms at the same time leads to estimation problems
because of the parameter redundancy as well as violating the principle of parsimony.

If our tentative model is AR(2 ) we might over fit with AR(3) . The original AR(2 ) will be confirmed if :

1) The estimates of the additional parameter φ3 is not significantly different from zero.

2) The estimates for the parameters φ1 and φ 2 do not change significantly from the original
states.

CHAPTER 7
FORECASTING

7.1 Minimum Mean Square Error

The minimum mean square error (MMSE) forecasts xˆ n (l ) of z n +l at the forecast origin n is given by
the conditional expectation: xˆ n (l ) = E (x n +l z n , z n −1 ,...) .

Example: 7.1.1
(
Let ψ (B ) = φ (B )Wt = φ (B )(1 − B ) = 1 − ψ 1 B − ... − ψ p + q B p + d
d
)
7.1.1 ARMA model forecast equation in infinite MA form

The ARMA model for xt is xt = ψ (B )z t ⇒ xt = ψ 1 z t −1 + ψ 2 z t − 2 + ... + z t

Assume that we have x1 , x 2 ,..., x n and we want to forecast x n +l (i.e l step-ahead forecast from origin,
the actual value is: x n +l = ψ 1 z n +l −1 + ψ 2 z n +l − 2 + ... + z n +l .

44 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

The “minimum mean square error” forecast for x n +l is xˆ n (l ) = ψ 1 ( z n +l −1 ) + ψ 2 ( z n +l − 2 ) + ...

This form is not very useful for computing forecasts, but is useful in finding the forecast error.

7.2 Computation of Forecasts

7.2.1 Forecast error and forecast error variances
(a) One step –ahead ( l = 1 )

en (1) = x n +1 − xˆ n (1) = z n +1

var[en (1)] = var[z n +1 ] = σ z

2

(b) Two steps –ahead ( l = 2 )

en (2 ) = x n + 2 − xˆ n (2 ) = z n + 2 + ψ 1 z n +1

var[en (2 )] = var[z n + 2 ] + ψ 1 var[z n +1 ] = σ z + ψ 1 z z = σ z 1 + ψ 1

2 2 2 2 2
( 2
)
(c) Three steps-ahead ( l = 3 )

en (3) = x n +3 − xˆ n (3) = z n +3 + ψ 1 z n + 2 + ψ 2 z n +1

var[en (3)] = var[z n +3 ] + ψ 1 var[z n + 2 ] + ψ 2 var[z n +1 ] = σ z + ψ 1 σ z + ψ 2 σ z = σ z 1 + ψ 1 + ψ 2

2 2 2 2 2 2 2 2
( 2 2
)
(d) In general, l steps-ahead

en (l ) = x n +l − xˆ n (l ) = z n +l + ψ 1 z n +l −1 + ... + ψ l −1 z n +1

( )
l −1
var[en (l )] = σ z 1 + ψ 1 + ψ 2 + ... + ψ l −1 = σ z ∑ψ where ψ 0 ≡ 1 .
2 2 2 2 2 2
i
i =0

To forecast l steps-ahead from origin n :

The actual value is x n +l = ψ 1 z n +l −1 + ψ 2 z n +l − 2 + ... + z n +l

The minimum mean square error forecast for x n +l is xˆ n (l ) = ψ 1 ( z n +l −1 ) + ψ 2 ( z n +l − 2 ) + ...

7.3 Prediction Interval

A 95% prediction interval for :

a) l step ahead forecast is xˆ n (l ) ± Z 0.025 × var(en (l )) .

b) One step ahead is xˆ n (1) ± Z 0.025 × var(en (1)) .

45 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

c) Two steps ahead is xˆ n (2 ) ± Z 0.025 × var(en (2 )) .

d) Three steps ahead is xˆ n (3) ± Z 0.025 × var(en (3)) .

7.4 Forecasting AR( p ) and MA(q ) models.

Example: 7.4.1
For each of the following models,

AR(1) process: xt − µ + φ (xt −1 − µ ) + z t .

MA(1) process: xt = z t − θ z t −1 .

Find: a) xˆ n (1) b) xˆ n (2 ) c) xˆ n (1)

Exercise: 7.4.1
For each of the following models,

AR(2 ) process: xt = φ1 xt −1 + φ 2 xt − 2 + z t .

MA(1) process: xt = z t − θ1 z t −1 − θ 2 z t − 2 .

Find: a) xˆ n (1) b) xˆ n (2 ) c) xˆ n (1)

7.5 Forecasting ARMA( p, q ) models

Example: 7.5.1
For (1 − φ B )( xt − µ ) = (1 − θ B ) z t

a) find the first, 2nd and l -step ahead forecast.

b) Calculate: i) ψ − weight ii) forecast error variance

iii) 95% forecast limits for xˆ n (l ).

Exercise: 7.5.1
Consider the model (IMA (1,1)) : (1 − B ) x t = (1 − θ B ) z t .

Calculate:

46 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

a) the l − step ahead forecast xˆ n (l ) of x n +l .

b) ψ − weight of xt .

c) 95% forecast limits for xˆ n (l ), xˆ n (1) and xˆ n (2 ) .

7.6 Forecasting ARIMA( p, d , q ) models

Consider the ARIMA( p, d , q ) model at time t = n + l , φ p (B )(1 − B ) xt = θ q (B ) z t

d

47 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

APPENDIX A

Table 7A: The chi-squared distribution (χ²), right tail probability.

( )
The table gives the values of χ α2 ,v where P χ 2 > χ α2 ,v = α with v degree of freedom.
0.10 0.05 0.025 0.01 0.005

1 2.706 3.841 5.024 6.635 7.879

2 4.605 5.991 7.378 9.210 10.597

3 6.251 7.815 9.348 11.345 12.838

4 7.779 9.488 11.143 13.277 14.860

5 9.236 11.070 12.833 15.086 16.750

6 10.645 12.592 14.449 16.812 18.548

7 12.017 14.067 16.013 18.475 20.278

8 13.362 15.507 17.535 20.090 21.955

9 14.684 16.919 19.023 21.666 23.589

10 15.987 18.307 20.483 23.209 25.188

11 17.275 19.675 21.920 24.725 26.757

12 18.549 21.026 23.337 26.217 28.300

13 19.812 22.362 24.736 27.688 29.819

14 21.064 23.685 26.119 29.141 31.319

15 22.307 24.996 27.488 30.578 32.801

16 23.542 26.296 28.845 32.000 34.267

17 24.769 27.587 30.191 33.409 35.718

18 25.989 28.869 31.526 34.805 37.156

19 27.204 30.144 32.852 36.191 38.582

20 28.412 31.410 34.170 37.566 39.997

21 29.615 32.671 35.479 38.932 41.401

22 30.813 33.924 36.781 40.289 42.796

48 | P a g e
 TIME SERIES ANALYSIS LECTURE NOTES 2023

23 32.007 35.172 38.076 41.638 44.181

24 33.196 36.415 39.364 42.980 45.559

25 34.382 37.652 40.646 44.314 46.928

26 35.563 38.885 41.923 45.642 48.290

27 36.741 40.113 43.195 46.963 49.645

28 37.916 41.337 44.461 48.278 50.993

29 39.087 42.557 45.722 49.588 52.336

30 40.256 43.773 46.979 50.892 53.672

40 51.805 55.758 59.342 63.691 66.766

50 63.167 67.505 71.420 76.154 79.490

60 74.397 79.082 83.298 88.379 91.952

70 85.527 90.531 95.023 100.425 104.215

80 96.578 101.879 106.629 112.329 116.321

90 107.565 113.145 118.136 124.116 128.299

100 118.498 124.342 129.561 135.807 140.169

110 129.385 135.480 140.917 147.414 151.948

120 140.233 146.567 152.211 158.950 163.648

________________________________________________________________________________________

49 | P a g e