Logistics

Introduction to Time Series

Statistical Models of Time Series

Stat 153: Introduction to Time Series

Joan Bruna

Department of Statistics
UC, Berkeley

September 3, 2015

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Instructor: Joan Bruna Estrach, [email protected]

Office hours and location: Tuesdays 2pm-4pm, Evans 419.
bCourses link:
https://bcourses.berkeley.edu/courses/1364346
GSIs: To be announced...

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Grading

Midterm Exams: October 13th, November 12th, 15 % each.

Final Exam: December 18th, 35 %.
Lab/Homework assignments1 . (every one or two weeks), 25
%
Lab Final Project, 10 %.

1
lowest grade will be dropped
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

S&P 500 index

Statistical Model necessary.

Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

S&P 500 index, zoomed

Interaction of random and non-random behavior.

Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

trend
seasonality (periodicity)
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

Periodic, stationary phenomena is studied with spectrograms.

Joan Bruna Stat 153: Introduction to Time Series

 !N !Ti on the same location (see Figure 1), initially used for the
i i
t=1 P (qt = j|O , λm )Ot training of traffic conflict observers in the 1980s. Their
µcj = !
i=1
N !Ti
(1)
i=1
i
t=1 P (qt = j|O , λm )
length ranges from 10 seconds to 60 seconds. Despite the
videotape aging, the approximate alignment of the field of
!N !Ti i i c i c T
Logistics
view between sequences and occasional camera jitter, it
P (qt = j|O , λm )(Ot − µj )(Ot − µj )
Σcj = i=1 t=1!N !Ti
i, λ )
Introduction to used
could be digitized and Time to test Series
our method. The training
i=1 t=1 P (qt = j|O m program highlights nine traffic conflicts in nine sequences
Statistical
(2) Models of Time Series
(see Figure 2). However, only five of them are used as
The parameters of the HMM λm can then be adapted using
training instances. Another one (fourth sequence) is not used
the following equations:
because the vehicles are not detected anymore at the time
′ they are really in conflict, and the rest of the traffic conflicts
µj = (1 − α)µj + αµcj (3)
involve pedestrians and cyclists that are very difficult to

What is a Time Series?

′
Σj =
′

′
′
(1 − α)(Σj + (µj − µj )(µj − µj )T )
′
distinguish and track on account of the videotape quality.
There is no traffic conflict highlighted in the last sequence. It
is sometimes difficult to judge the performance of the method
c c c T
+α(Σj + (µj − µj )(µj − µj ) ) (4) since there is no ground truth, apart from the traffic conflicts
′ ′ used in the program.
where µj and Σj are the mean and covariance of the
Gaussian of the state Sj of the adapted HMM. α is a weight-
ing factor that controls the balance between the original
model and the new estimates on the trajectories involved
in the training traffic conflicts. Both the original model and
the adapted model are kept in the set of HMMs used for
detection.
A traffic conflict is an interaction, defined as an obser-
vational situation in which two or more vehicles are close
enough in space and time, and are nearing each other. So
far, the method has built a set of HMMs, among which
some are models of conflicting trajectories. But a trajectory
is conflicting only with respect to another one. It is their
conjunction in time and space that creates a danger of
Fig. 1. An image of the traffic sequences.
collision. Therefore, the models of the conflicting trajectories
(to which the trajectories involved in the training traffic
conflict instances are assigned) are memorized by pairs (e.g.
pairs of models 4 and 7, 11 and 2, 3 and 7). The traffic
conflict detection proceeds as follows:
1) Vehicles are tracked.
2) If two vehicles are close enough (threshold on their
distance) and nearing each other (their distance de-
creases), an interaction is detected.
3) Each interacting vehicle trajectory is assigned to a
HMM.
4) If the HMMs of both interacting trajectories were both
memorized as conflicting (e.g. any of the pairs of
models 4 and 7, 11 and 2, 3 and 7), a traffic conflict
between these two vehicles is detected.
Fig. 2. An example of trajectories involved in a traffic conflict.
V. E XPERIMENTAL E VALUATION from Saunier and Sayed
A simple vehicle detection and tracking algorithm is used, All K HMMs of a mixture have the same structure param-
based on the implementation of the KLT feature tracking eter values (number of states, simple Gaussian observation
algorithm of [8], used in [9]. The advantages of feature- 2 http://www.cs.ubc.ca/∼murphyk/Software/HMM/hmm.

Robotics
based algorithms include the abilities to work well under html

Control Theory (Kalman)

Self-driving cars ...
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

State space models (language).

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

What is a Time Series?

State space models (language).

Prediction, compression, ...
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

This is not an IID world

Fundamental characteristic of time-series: in general, samples

are correlated (thus statistically dependent).

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

This is not an IID world

Fundamental characteristic of time-series: in general, samples

are correlated (thus statistically dependent).

Estimating, modeling and analyzing this dependency/correlation is

the main objective of this course.

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Objectives of Time Series Analysis

Compact description of the data: statistical modeling.

Interpretation
Forecasting/Prediction
Control
Simulation
Hypothesis testing

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Example: Monthly Sales of a Souvenir Shop

(from P.Bartlett’s slides, Makridakis, Wheelwright and Hyndman, 1998)

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Transformed
Example: Transformation of the data
data
12

11.5

11

10.5

10

9.5

8.5

7.5

7
0 10 20 30 40 50 60 70 80 90
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

Example: Trend Trend

12

11.5

11

10.5

10

9.5

8.5

7.5

7
0 10 20 30 40 50 60 70 80 90
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

Residuals
Example: Look at the residuals
1.5

0.5

−0.5

−1
0 10 20 30 40 50 60 70 80 90
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

Trend Seasonality
Example: Modeling and seasonaland
variation
Trend
12

11.5

11

10.5

10

9.5

8.5

7.5

7
0 10 20 30 40 50 60 70 80 90
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

Example

Compact Description: Decomposition model

Xt = Tt + St + Rt .

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Example

Compact Description: Decomposition model

Xt = Tt + St + Rt .

This model is interpretable. (eg seasonal adjustment due to

holidays).

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Example

Compact Description: Decomposition model

Xt = Tt + St + Rt .

This model is interpretable. (eg seasonal adjustment due to

holidays).
Forecasting/Prediction: Expected Sales next month?

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Overview of the Course

1 Time series basics.

2 Time domain Models.
(Midterm 1)
3 Spectral and Wavelet Analysis.
(Midterm 2)
4 State Space and Discrete Models.

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Time Series Basics

Stationarity.
Autocorrelation Function.
Chasing Stationarity.

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Time domain Models

Auto-Regressive models (AR, MA, ARMA, ...)

Local Volatility models (GARCH, ..)
ARIMA models
Forecasting
Parameter Estimation

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Spectral Methods
Correlation is a form of (global) smoothness. How to study
smoothness properties?

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Spectral Methods
Correlation is a form of (global) smoothness. How to study
smoothness properties?

Joseph Fourier (1768-1830).

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

Spectral Methods
Correlation is a form of (global) smoothness. How to study
smoothness properties?

Joseph Fourier (1768-1830). Smoothness is studied

mathematically using Spectral (or Fourier, or Harmonic) Analysis.
Several Spectral Representations possible (Fourier, Windowed
Fourier, Wavelet, etc.)
Joan Bruna Stat 153: Introduction to Time Series
 Logistics
Introduction to Time Series
Statistical Models of Time Series

Spectral and Wavelet Analysis

Spectral Density.
Time-Frequency Representations, Periodogram.
Introduction to Wavelet Analysis
Spectral Estimation.
Applications to speech recognition.

Joan Bruna Stat 153: Introduction to Time Series

 Logistics
Introduction to Time Series
Statistical Models of Time Series

State Space and Discrete Models

ARMAX models
Hidden Markov models (HMM).
Kalman Filter.
Recurrent Neural Networks (RNN). (time allowing)

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Time Series Modeling/Notation

How do we define a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Time Series Modeling/Notation

How do we define a Time Series?

A (stochastic) Time Series is a collection {Xt } of random variables
indexed by a temporal index t.

In this course, we will mostly consider discrete time series:

t = 1, 2, 3, . . . is a discrete index.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Time Series Modeling/Notation

How do we define a Time Series?

A (stochastic) Time Series is a collection {Xt } of random variables
indexed by a temporal index t.

In this course, we will mostly consider discrete time series:

t = 1, 2, 3, . . . is a discrete index.
{Xt } will always denote a stochastic process.
{xt } will always denote a particular realization.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Time Series Modeling

How do we specify a Time Series Model?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Time Series Modeling

How do we specify a Time Series Model?

A Time Series model is (fully) specified by giving the joint

distribution of {Xt }:

P(X1 ≤ x1 , X2 ≤ x2 , . . . , Xt ≤ xt ) for all t, x1 , . . . , xt .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

The Curse of Dimensionality

As t increases, the complexity of the previous model grows

exponentially.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

The Curse of Dimensionality

As t increases, the complexity of the previous model grows

exponentially.
Intractable in general.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

The Curse of Dimensionality

As t increases, the complexity of the previous model grows

exponentially.
Intractable in general.
We will resort to low-order statistics only (mostly first and
second order).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

White Noise
{Xt } is a white noise if for all t,
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
3 Xt and Xu are uncorrelated for t 6= u.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

White Noise
{Xt } is a white noise if for all t,
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
3 Xt and Xu are uncorrelated for t 6= u.
In particular, if {Xt } are i.i.d with zero mean, {Xt } is a white
noise. Also,
Y
P(X1 ≤ x1 , . . . , Xt ≤ xt ) = P(Xi ≤ xi ) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

White Noise
{Xt } is a white noise if for all t,
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
3 Xt and Xu are uncorrelated for t 6= u.
In particular, if {Xt } are i.i.d with zero mean, {Xt } is a white
noise. Also,
Y
P(X1 ≤ x1 , . . . , Xt ≤ xt ) = P(Xi ≤ xi ) .

Forecasting is not possible under iid noise:

P(Xt ≤ xt | X1 , . . . , Xt−1 ) = P(Xt ≤ xt ) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Most Famous Time Series

Gaussian White Noise: Xt ∼ N (0, σ 2 ) iid.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Most Famous Time Series

Gaussian White Noise: Xt ∼ N (0, σ 2 ) iid.

It cannot model any time-dependencies.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Bernouilli White Noise

Xt ∼ Bern(p), with p ∈ [0, 1].

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Bernouilli White Noise

Xt ∼ Bern(p), with p ∈ [0, 1].

Eg: models the success at a casino roulette.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Moving Averages

A simple way to model dependencies across samples is to average

across time.
If {Wt } is white noise, the series
∆
X
Xt = λk Wt+k
k=−∆

is called a moving average (MA(2∆)).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Moving Averages
P∆
Xt = k=−∆ λk Wt+k .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Random Walks

P
Consider {Wt } white noise, and Xt = i≤t Wi .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Random Walks

P
Consider {Wt } white noise, and Xt = i≤t Wi .

Differences ∇Xt = Xt − Xt−1 = Wt .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Random Walks

E (Xt ) = , var (Xt ) =

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Random Walks

E (Xt ) = 0 , var (Xt ) = tσ 2 .

Variance increases with time!

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Random Walks
Recall S&P data.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Random Walks
Recall S&P data.

Random walks and their generalizations (Brownian Motions) are

good financial models.
Joan Bruna Stat 153: Introduction to Time Series
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Signal with Noise

In many problems, good models combine deterministic with
stochastic aspects:
Xt = F (t) + Wt , with F deterministic .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Signal with Noise

In many problems, good models combine deterministic with
stochastic aspects:
Xt = F (t) + Wt , with F deterministic .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Lecture Review

A (stochastic) Time Series is a collection {Xt } of random variables

indexed by a temporal index t.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Lecture Review

A (stochastic) Time Series is a collection {Xt } of random variables

indexed by a temporal index t.

A Time Series model is (fully) specified by giving the joint

distribution of {Xt }:

P(X1 ≤ x1 , X2 ≤ x2 , . . . , Xt ≤ xt ) for all t, x1 , . . . , xt .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Lecture Review

A (stochastic) Time Series is a collection {Xt } of random variables

indexed by a temporal index t.

A Time Series model is (fully) specified by giving the joint

distribution of {Xt }:

P(X1 ≤ x1 , X2 ≤ x2 , . . . , Xt ≤ xt ) for all t, x1 , . . . , xt .

Intractable in general.
We will resort to low-order statistics only (mostly first and second
order).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: White Noise

{Xt } is a white noise if for all t,
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
3 Xt and Xu are uncorrelated for t 6= u.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: White Noise

{Xt } is a white noise if for all t,
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
3 Xt and Xu are uncorrelated for t 6= u.
In particular, if {Xt } are i.i.d with zero mean, {Xt } is a white
noise. Also,
Y
P(X1 ≤ x1 , . . . , Xt ≤ xt ) = P(Xi ≤ xi ) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: White Noise

{Xt } is a white noise if for all t,
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
3 Xt and Xu are uncorrelated for t 6= u.
In particular, if {Xt } are i.i.d with zero mean, {Xt } is a white
noise. Also,
Y
P(X1 ≤ x1 , . . . , Xt ≤ xt ) = P(Xi ≤ xi ) .

Forecasting is not possible under iid noise:

P(Xt ≤ xt | X1 , . . . , Xt−1 ) = P(Xt ≤ xt ) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: Random Walks

P
Consider {Wt } white noise, and Xt = i≤t Wi .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: Random Walks

P
Consider {Wt } white noise, and Xt = i≤t Wi .

Differences ∇Xt = Xt − Xt−1 = Wt .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: Mean and Covariance of a Time Series

Definition
The mean function of a time series {Xt } is

µX (t) = E (Xt ) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: Mean and Covariance of a Time Series

Definition
The mean function of a time series {Xt } is

µX (t) = E (Xt ) .

Definition
The autocovariance function of a time series {Xt } is

RX (t, s) = cov(Xt , Xs ) = E ((Xt − E (Xt ))(Xs − E (Xs ))) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Lecture 2

Autocovariance examples
Autocorrelation and Cross-Correlation
Stationarity
Linear Processes

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples

P
{Xt } Random Walk: Xt = i≤t Wi , with {Wt } iid white noise.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples

P
{Xt } Random Walk: Xt = i≤t Wi , with {Wt } iid white noise.
 
X X
µX (t) = E (Xt ) = E  Wi  = E (Wi ) = 0 .
i≤t i≤t

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples

P
{Xt } Random Walk: Xt = i≤t Wi , with {Wt } iid white noise.
 
X X
µX (t) = E (Xt ) = E  Wi  = E (Wi ) = 0 .
i≤t i≤t

X X X
RX (s, t) = cov( Wi , Wi 0 ) = cov(Wi , Wi 0 ) = min(s, t)σ 2 .
i≤s i 0 ≤t i≤s,i 0 ≤t

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples

{Xt } = F (t) + {Wt }: signal with white noise.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples

{Xt } = F (t) + {Wt }: signal with white noise.

µX (t) = E (Xt ) = E (F (t) + Wt ) = F (t) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples

{Xt } = F (t) + {Wt }: signal with white noise.

µX (t) = E (Xt ) = E (F (t) + Wt ) = F (t) .


σ2 if s=t ,
RX (s, t) = cov(F (s)+Ws , F (t)+Wt ) = cov(Ws , Wt ) =
0 otherwise .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example

P∆
Moving average process: Xt = k=−∆ Wt+k , with {Wt } white
noise.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example

P
Moving average process: Xt = ∆ k=−∆ Wt+k , with {Wt } white
noise.
RX (s, t) =? .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example

If |t − s| > 2∆:

∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k =
k=−∆ k=−∆

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example

If |t − s| > 2∆:

∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k =0.
k=−∆ k=−∆

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example

If |t − s| ≤ 2∆:

∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k =
k=−∆ k=−∆

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example

If |t − s| ≤ 2∆:

∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k = (2∆ − |t − s|)σ 2 .
k=−∆ k=−∆

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example

If |t − s| ≤ 2∆:

∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k = (2∆ − |t − s|)σ 2 .
k=−∆ k=−∆

In particular, RX only depends upon the difference t − s.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Autocorrelation function (ACF)

In some situations, it is better to normalize the autocovariance.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Autocorrelation function (ACF)

In some situations, it is better to normalize the autocovariance.

Definition
The Autocorrelation function (ACF) of a time series {Xt } is
defined as
RX (s, t)
ρX (s, t) = p .
RX (s, s)RX (t, t)

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Autocorrelation function (ACF)

In some situations, it is better to normalize the autocovariance.

Definition
The Autocorrelation function (ACF) of a time series {Xt } is
defined as
RX (s, t)
ρX (s, t) = p .
RX (s, s)RX (t, t)

Measures linear predicability of Xt from Xs .

It satisfies
−1 ≤ ρX (s, t) ≤ 1 .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Cross-Covariance and Cross-Correlation

We might also want to measure predictability from one series {Xt }
to another {X̃t }.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Cross-Covariance and Cross-Correlation

We might also want to measure predictability from one series {Xt }
to another {X̃t }.
Exxon vs Shell:

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Cross-Covariance and Cross-Correlation

We might also want to measure predictability from one series {Xt }

to another {X̃t }.
Exxon vs Google:

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Cross-Covariance and Cross-Correlation

We might also want to measure predictability from one series {Xt }

to another {X̃t }.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Cross-Covariance and Cross-Correlation

We might also want to measure predictability from one series {Xt }

to another {X̃t }.
Definition
The Cross-Covariance between {Xt } and {X̃t } is

RX ,X̃ (s, t) = cov(Xs , X̃t ) ,

and the cross-correlation is

RX ,X̃ (s, t)
ρX ,X̃ (s, t) = .
RX (s, s)RX̃ (t, t)

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Stationarity: Motivation

Q: Main difference between figures?

Joan Bruna Stat 153: Introduction to Time Series
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Strict Stationarity

Definition
A Time Series {Xt } is strictly Stationary if

P(Xt1 ≤ c1 , Xt2 ≤ c2 , . . . , Xtk ≤ ck ) =

P(Xt1 +h ≤ c1 , Xt2 +h ≤ c2 , . . . , Xtk +h ≤ ck ) , ∀ k, ti , ci , h .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Strict Stationarity

Definition
A Time Series {Xt } is strictly Stationary if

P(Xt1 ≤ c1 , Xt2 ≤ c2 , . . . , Xtk ≤ ck ) =

P(Xt1 +h ≤ c1 , Xt2 +h ≤ c2 , . . . , Xtk +h ≤ ck ) , ∀ k, ti , ci , h .

Time is relative: statistical properties do not depend upon

time reference.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Strict Stationarity

Definition
A Time Series {Xt } is strictly Stationary if

P(Xt1 ≤ c1 , Xt2 ≤ c2 , . . . , Xtk ≤ ck ) =

P(Xt1 +h ≤ c1 , Xt2 +h ≤ c2 , . . . , Xtk +h ≤ ck ) , ∀ k, ti , ci , h .

Time is relative: statistical properties do not depend upon

time reference.
In particular, using k = 1, P(Xt ≤ c) is independent of t.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Strict Stationarity

Definition
A Time Series {Xt } is strictly Stationary if

P(Xt1 ≤ c1 , Xt2 ≤ c2 , . . . , Xtk ≤ ck ) =

P(Xt1 +h ≤ c1 , Xt2 +h ≤ c2 , . . . , Xtk +h ≤ ck ) , ∀ k, ti , ci , h .

Time is relative: statistical properties do not depend upon

time reference.
In particular, using k = 1, P(Xt ≤ c) is independent of t.
Far LESS parameters to describe the process.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Strict Stationarity: Consequences

Using previous property for k = 1, P(Xt ≤ c) = P(Xs ≤ c)
for all t, s implies

µX (t) = cte .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Strict Stationarity: Consequences

Using previous property for k = 1, P(Xt ≤ c) = P(Xs ≤ c)
for all t, s implies

µX (t) = cte .

Using previous property for k = 2, joint law (Xt , Xs ) = joint

law (Xt+h , Xs+h ), hence

∀ h , RX (t, s) = RX (t + h, s + h) =⇒ RX depends only on |t − s| .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Strict Stationarity: Consequences

Using previous property for k = 1, P(Xt ≤ c) = P(Xs ≤ c)
for all t, s implies

µX (t) = cte .

Using previous property for k = 2, joint law (Xt , Xs ) = joint

law (Xt+h , Xs+h ), hence

∀ h , RX (t, s) = RX (t + h, s + h) =⇒ RX depends only on |t − s| .

(since R(0, t − s) = R(s, t) = R(t, s) = R(0, s − t)) .

Joan Bruna Stat 153: Introduction to Time Series
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Weak Stationarity

In practice, strict Stationarity is hard to impose/estimate.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Weak Stationarity

In practice, strict Stationarity is hard to impose/estimate.

Definition
A Time series {Xt } is weakly stationary if it has finite variance,
µX (t) = cte and RX (t, s) = f (|t − s|).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Weak Stationarity

In practice, strict Stationarity is hard to impose/estimate.

Definition
A Time series {Xt } is weakly stationary if it has finite variance,
µX (t) = cte and RX (t, s) = f (|t − s|).

If {Xt } is strictly stationary (and has finite variance), then it

is weakly stationary.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Weak Stationarity

In practice, strict Stationarity is hard to impose/estimate.

Definition
A Time series {Xt } is weakly stationary if it has finite variance,
µX (t) = cte and RX (t, s) = f (|t − s|).

If {Xt } is strictly stationary (and has finite variance), then it

is weakly stationary.
Converse not true, but ...

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Weak Stationarity

In practice, strict Stationarity is hard to impose/estimate.

Definition
A Time series {Xt } is weakly stationary if it has finite variance,
µX (t) = cte and RX (t, s) = f (|t − s|).

If {Xt } is strictly stationary (and has finite variance), then it

is weakly stationary.
Converse not true, but ...
...If {Xt } is Gaussian and weakly stationary, then it is strictly
stationary.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Autocorrelation Function (ACF)

The Autocorrelation of a stationary process {Xt } is

RX (h) cov(Xt , Xt+h )

ρX (h) = = .
RX (0) var (Xt )

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples: White Noise

{Wt } white noise.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples: White Noise

{Wt } white noise. It is weakly stationary since

µw (t) = 0 ∀ t , and Rw (s, t) = σ 2 1(|s − t| = 0) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples: Random Walk

{Xt } Random Walk. We saw that

µX (t) = 0 , RX (s, t) = σ 2 min(s, t) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples: Random Walk

{Xt } Random Walk. We saw that

µX (t) = 0 , RX (s, t) = σ 2 min(s, t) .

Thus it is NOT stationary.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples: MA(1) process

Moving average process MA(1):

Xt = Wt + λWt−1 , {Wt } white noise .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples: MA(1) process

Moving average process MA(1):

Xt = Wt + λWt−1 , {Wt } white noise .

We have µX (t) = 0, and

RX (t, t + h) = E (Xt Xt+h )

= E ((Wt + λWt−1 )(Wt+h + λWt+h−1 ))
 2
 σ (1 + λ2 ) if h = 0
= σ 2 λ if h = ±1

0 otherwise .
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples: MA(1) process

Moving average process MA(1):

Xt = Wt + λWt−1 , {Wt } white noise .

We have µX (t) = 0, and

RX (t, t + h) = E (Xt Xt+h )

= E ((Wt + λWt−1 )(Wt+h + λWt+h−1 ))
 2
 σ (1 + λ2 ) if h = 0
= σ 2 λ if h = ±1

0 otherwise .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples: MA(1) Process.

ACF of {Xt }:

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example: Autoregressive Process

Autoregressive AR(1) process:

Xt = λXt−1 + Wt , with {Wt } white noise and |λ| < 1 .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example: Autoregressive Process

Autoregressive AR(1) process:

Xt = λXt−1 + Wt , with {Wt } white noise and |λ| < 1 .

By substituting the recursion we obtain

Xt = Wt + λWt−1 + λ2 Wt−2 + λ3 Wt−3 + . . .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Last Important Example: Autoregressive Process

Autoregressive AR(1) process:

Xt = λXt−1 + Wt , with {Wt } white noise and |λ| < 1 .

By substituting the recursion we obtain

Xt = Wt + λWt−1 + λ2 Wt−2 + λ3 Wt−3 + . . .

∞
!
X X
µX (t) = E λk Wt−k = λk E (Wt−k ) = 0 , and
k=0 k
∞

X σ2
E Xt2 = λ2k σ 2 = .
1 − λ2
k=0

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Autoregressive Process

Suppose h > 0 first. Then

RX (t, t + h) = cov(Xt , Xt+h ) = cov(Xt , λXt+h−1 + Wt+h )

= λcov(Xt , Xt+h−1 )
= λh cov(Xt , Xt )
λ|h| σ 2
=
1 − λ2
(check for h < 0 at home).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Autoregressive Process

Suppose h > 0 first. Then

RX (t, t + h) = cov(Xt , Xt+h ) = cov(Xt , λXt+h−1 + Wt+h )

= λcov(Xt , Xt+h−1 )
= λh cov(Xt , Xt )
λ|h| σ 2
=
1 − λ2
(check for h < 0 at home).
So AR(1) is (weakly) stationary.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: AR(1) Process.

ACF of {Xt }:

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: Stationarity and Correlation

A time series {Xt } has mean function µt = E (Xt ) and

autocovariance

RX (t, t + h) = E ((Xt − µt )(Xt+h − µt+h )) .

It is (weakly) stationary if both functions are independent of t. We

then write µ and RX (h).
The Autocorrelation Function (ACF) of {Xt } is

RX (h)
ρX (h) = .
RX (0)

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: Autoregressive Process

Autoregressive AR(1) process:

Xt = λXt−1 + Wt , with {Wt } white noise and |λ| < 1 .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: Autoregressive Process

Autoregressive AR(1) process:

Xt = λXt−1 + Wt , with {Wt } white noise and |λ| < 1 .

By substituting the recursion we obtain

Xt = Wt + λWt−1 + λ2 Wt−2 + λ3 Wt−3 + . . .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: Autoregressive Process

Autoregressive AR(1) process:

Xt = λXt−1 + Wt , with {Wt } white noise and |λ| < 1 .

By substituting the recursion we obtain

Xt = Wt + λWt−1 + λ2 Wt−2 + λ3 Wt−3 + . . .

∞
!
X X
µX (t) = E λk Wt−k = λk E (Wt−k ) = 0 , and
k=0 k
∞

X σ2
E Xt2 = λ2k σ 2 = .
1 − λ2
k=0

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Autoregressive Process

Suppose h > 0 first. Then

RX (t, t + h) = cov(Xt , Xt+h ) = cov(Xt , λXt+h−1 + Wt+h )

= λcov(Xt , Xt+h−1 )
= λh cov(Xt , Xt )
λ|h| σ 2
=
1 − λ2
(check for h < 0 at home).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Autoregressive Process

Suppose h > 0 first. Then

RX (t, t + h) = cov(Xt , Xt+h ) = cov(Xt , λXt+h−1 + Wt+h )

= λcov(Xt , Xt+h−1 )
= λh cov(Xt , Xt )
λ|h| σ 2
=
1 − λ2
(check for h < 0 at home).
So AR(1) is (weakly) stationary.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: AR(1) Process.

ACF of {Xt }:

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Today’s menu

Linear Processes
Estimation of Autocovariance and ACF.
Prediction with ACF.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Linear Processes

So far, all stationary processes we have seen are of the form

∞
X
Xt = µ + ψk Wt−k ,
k=−∞

with
{Wt } white noise.
P 2
k |ψk | < ∞.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Linear Processes

So far, all stationary processes we have seen are of the form

∞
X
Xt = µ + ψk Wt−k ,
k=−∞

with
{Wt } white noise.
P 2
k |ψk | < ∞.
These are called linear processes.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Linear Processes

Proposition
Any linear Process {Xt } is weakly stationary, with

µX (t) = µ ,
∞
X
RX (h) = σ 2 ψk ψk+h .
k=−∞

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Linear Processes

Proposition
Any linear Process {Xt } is weakly stationary, with

µX (t) = µ ,
∞
X
RX (h) = σ 2 ψk ψk+h .
k=−∞

Also, stationary processes are essentially linear (outside the scope

of stat153).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples of Linear Processes

For white noise, choose µ = 0 and


1 if k = 0 ,
ψk =
0 otherwise.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples of Linear Processes

For a MA(1) process, choose µ = 0 and


 1 if k = 0 ,
ψk = λ if k = 1 ,

0 otherwise.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples of Linear Processes

For a AR(1) process, choose µ = 0 and

 k
λ if k ≥ 0 ,
ψk =
0 otherwise.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples of Linear Processes

Q: What about a random walk?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Examples of Linear Processes

Q: What about a random walk?

X X
Xt = Wt−k 6= ψk Wt−k .
0≤k≤t k

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

A slide for EE
We can view a linear process {Xt } as

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

A slide for EE
We can view a linear process {Xt } as

{Xt } is thus obtained by filtering a white noise with the filter

ψ.
The filter operation is known also as convolution:
X
(W ∗ ψ)t := ψk Wt−k .
k

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Key quantities to estimate

Suppose {Xt } is a stationary process. Given a finite number of

observations x1 , . . . , xn , we need to estimate
Its mean µ.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Key quantities to estimate

Suppose {Xt } is a stationary process. Given a finite number of

observations x1 , . . . , xn , we need to estimate
Its mean µ.
Its autocovariance function

RX (h) = cov(Xt , Xt+h ) ,

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Key quantities to estimate

Suppose {Xt } is a stationary process. Given a finite number of

observations x1 , . . . , xn , we need to estimate
Its mean µ.
Its autocovariance function

RX (h) = cov(Xt , Xt+h ) ,

and its ACF:

RX (h)
ρX (h) = .
RX (0)

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Mean and Autocorrelation

The sample mean is

n
1X
b=
µ xi .
n
i=1

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Mean and Autocorrelation

The sample mean is

n
1X
b=
µ xi .
n
i=1

The sample autocorrelation is

n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Mean and Autocorrelation

The sample mean is

n
1X
b=
µ xi .
n
i=1

The sample autocorrelation is

n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1

The sample ACF is

Rc
X (h)
ρc
X (h) = .
RcX (0)
Joan Bruna Stat 153: Introduction to Time Series
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Mean

Q: What is the variance of the sample mean estimator?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Mean

Q: What is the variance of the sample mean estimator?

1 X X
µ) =
var (b 2
cov( xi , xi 0 )
n 0
i i
1
= 2 (nRX (0) + (n − 1)(RX (1) + RX (−1)) + . . .
n
+(n − 2)(RX (2) + RX (−2)) + . . . (RX (n − 1) + RX (1 − n)))
n  
1 X |h|
= 1− RX (h) .
n n
h=−n

Consequence: If RX (h) is smooth, variance of the sample mean

increases.
Joan Bruna Stat 153: Introduction to Time Series
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Autocovariance

n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Autocovariance

n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1

Similar to the sample covariance of (x1 , x1+h ), . . . , (xn−h , xn ),

except
divide by n instead of n − h,
b using all n samples.
sample mean µ
Why?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Autocovariance
Define a linear combination using samples of a stationary process
{Xt }:
Y = a1 X1 + a2 X2 + . . . an Xn .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Autocovariance
Define a linear combination using samples of a stationary process
{Xt }:
Y = a1 X1 + a2 X2 + . . . an Xn .
Then

var (Y ) = cov(Y , Y )
Xn
= ai ai 0 RX (i − i 0 )
i,i 0 =1

≥ 0 , ( ∀a) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Autocovariance
Define a linear combination using samples of a stationary process
{Xt }:
Y = a1 X1 + a2 X2 + . . . an Xn .
Then

var (Y ) = cov(Y , Y )
Xn
= ai ai 0 RX (i − i 0 )
i,i 0 =1

≥ 0 , ( ∀a) .
P
A function (or matrix) R such that i,i 0 ai ai 0 Ri,i 0 ≥ 0 for all a is
called positive semi-definite.
Joan Bruna Stat 153: Introduction to Time Series
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Autocovariance

Proposition
The Autocovariance is positive semi-definite.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Autocovariance

Proposition
The Autocovariance is positive semi-definite.

Proposition
The Sample Autocovariance is positive semi-definite.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample Autocovariance

Proposition
The Autocovariance is positive semi-definite.

Proposition
The Sample Autocovariance is positive semi-definite.

If we divide by n − h instead it is no longer true.

The same holds for the ACF ρX and the sample ACF ρˆX .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example

Consider {Wt } white gaussian noise, and

Xt = 2 + Wt − Wt−1 .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example

Consider {Wt } white gaussian noise, and

Xt = 2 + Wt − Wt−1 .

We can derive that


 1 if h = 0 ,
ρX (h) = − 21 if |h| = 1 ,

0 otherwise.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example
Samples from {Xt }:

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example

Sample ACF ρc
X (h):

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example
Sample ACF ρc
X (h) with true ACF (in red):

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example
Sample ACF ρc
X (h) with true ACF (in red):

Q: How to assess presence/absence of correlation?

Joan Bruna Stat 153: Introduction to Time Series
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Large Sample Distribution of ACF

Proposition
Let {Wt } be a white noise and fix H > 0. Then, under mild
assumptions on the law of Wt , for large n, the sample ACF ρc X (h),
h = 1, . . . , H, is approximately Normally distributed, with zero
mean and variance 1/n.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Large Sample Distribution of ACF

Proposition
Let {Wt } be a white noise and fix H > 0. Then, under mild
assumptions on the law of Wt , for large n, the sample ACF ρc X (h),
h = 1, . . . , H, is approximately Normally distributed, with zero
mean and variance 1/n.

Consequence: Under white noise assumption,

√
ρX (h)| > 1.96/ n) ≈ 0.05.
P(|c

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example

Sample ACF ρc
X (h) with true ACF (in red) and significance region
(in green):

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Review: Sample Autocovariance

The sample mean is
n
1X
b=
µ xi .
n
i=1

The sample autocorrelation is

n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1

The sample ACF is

Rc
X (h)
ρc
X (h) = .
RcX (0)

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample ACF

Many properties of non-white processes (even non-stationary) are

revealed through the sample ACF:

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample ACF

Many properties of non-white processes (even non-stationary) are

revealed through the sample ACF:

{Xt } Sample ACF

white 0
Trend Slow Decay
Periodic Periodic
MA(q) zero for |h| > q
AR(p) exponential decay

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

White noise ACF

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Trend

Let {Xt } be a stationary process. If

Yt = Xt + βt , with β 6= 0 ,

then {Yt } has trend (or also, drift).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Trend

Let {Xt } be a stationary process. If

Yt = Xt + βt , with β 6= 0 ,

then {Yt } has trend (or also, drift).

We can also replace the linear function βt with more complicated
trends (eg quadratic).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Trend

Let {Xt } be a stationary process. If

Yt = Xt + βt , with β 6= 0 ,

then {Yt } has trend (or also, drift).

We can also replace the linear function βt with more complicated
trends (eg quadratic).
Q: Can we detect trend via the estimate of the ACF?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Trend and ACF

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Periodicity and ACF

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample ACF for MA(1)

Xt = 0.4Wt−1 + Wt

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample ACF for MA(1)

Xt = 0.4Wt−1 + Wt

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample ACF for AR(1)

Xt = 0.7Xt−1 + Wt

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Sample ACF for AR(1)

Xt = 0.7Xt−1 + Wt

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Linear Prediction in Time Series

Q: For a given time series model, how well can Xt be predicted

from the past?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Linear Prediction in Time Series

Q: For a given time series model, how well can Xt be predicted

from the past?
Q2: How is the prediction related to the Autocorrelation function?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: AR(1) process

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: AR(1) process

Scatterplot between Xt and Xt−1 :

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: AR(1) process

Scatterplot between Xt and Xt−2 :

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: AR(1) process

Scatterplot between Xt and Xt−3 :

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: AR(1) process

Scatterplot between Xt and Xt−40 :

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Example: AR(1) process

Recall the ACF function:

ACF controls quality of linear prediction.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

Least squares estimate of a random variable Y :

min E |Y − f |2 =⇒ f =
f

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

Least squares estimate of a random variable Y :

min E |Y − f |2 =⇒ f = E (Y ) ,
f

with MSE var (Y ).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

Least squares estimate of a random variable Y :

min E |Y − f |2 =⇒ f = E (Y ) ,
f

with MSE var (Y ).

Least squares estimate of Y , given X :


min E |Y − f (X )|2 | X =⇒ f (X ) =
f

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

Least squares estimate of a random variable Y :

min E |Y − f |2 =⇒ f = E (Y ) ,
f

with MSE var (Y ).

Least squares estimate of Y , given X :


min E |Y − f (X )|2 | X =⇒ f (X ) = E (Y |X ) ,
f

with MSE var (Y |X ).

Thus, least squares estimate of Xt+h given Xt : E (Xt+h |Xt ).

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

Conditional expectations are easy under Gaussian distributions!

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

Conditional expectations are easy under Gaussian distributions!
Suppose X1 , . . . , Xn is jointly Gaussian, with density
 
1 1 T −1
fX (x) = n/2 1/2 exp − (x − µ) Σ (x − µ) .
2π |Σ| 2

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

Conditional expectations are easy under Gaussian distributions!
Suppose X1 , . . . , Xn is jointly Gaussian, with density
 
1 1 T −1
fX (x) = n/2 1/2 exp − (x − µ) Σ (x − µ) .
2π |Σ| 2
In particular, joint law of (Xt , Xt+h ) is also Gaussian, with mean
(µt , µt+h ) and covariance
 
σt2 ρ(t, t + h)σt σt+h
2 .
ρ(t, t + h)σt σt+h σt+h
Conditional distribution Xt+h given Xt = xt is therefore
 
σt+h ρ(t, t + h)(xt − µt ) 2 2
N µt+h + , σ (1 − ρ(t, t + h) ) .
σt
Joan Bruna Stat 153: Introduction to Time Series
 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

{Xt } Gaussian and stationary. What is the optimal prediction of

Xt+h given Xt = xt ?

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

{Xt } Gaussian and stationary. What is the optimal prediction of

Xt+h given Xt = xt ?

f (xt ) = E (Xt+h |Xt = xt ) = µ + ρX (h)(xt − µ) .

The resulting MSE is

E |Xt+h − f (xt )|2 , |Xt = xt = σ 2 (1 − ρX (h)2 ) .

Prediction gets better as |ρ| increases.

f (xt ) is linear: f (xt ) = α + βt.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

For general stationary processes (not necessarily Gaussian), best

predictor (in terms of MSE) has no closed form.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

For general stationary processes (not necessarily Gaussian), best

predictor (in terms of MSE) has no closed form. However, we can
consider optimal linear predictors.


E |Xt+h − α − βXt |2 = E (α, β) .
Setting ∂α E (α, β) = 0 and ∂β E (α, β) = 0, we obtain

α = µ(1 − ρX (h)) , β = ρX (h) ,

with
MSE = σ 2 (1 − ρX (h)2 ) .

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

f (xt ) = µ + ρX (h)(xt − µ)

Optimal linear predictor for any stationary {Xt }.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

f (xt ) = µ + ρX (h)(xt − µ)

Optimal linear predictor for any stationary {Xt }.

Optimal predictor for stationary and gaussian {Xt }.

Joan Bruna Stat 153: Introduction to Time Series

 Important Examples
Statistical Measurements
Logistics
Stationarity
Introduction to Time Series
Estimation of Correlation
Statistical Models of Time Series
Sample ACF
Prediction with ACF

Least Squares and ACF

f (xt ) = µ + ρX (h)(xt − µ)

Optimal linear predictor for any stationary {Xt }.

Optimal predictor for stationary and gaussian {Xt }.
Corollary: For gaussian processes, linear prediction is optimal
(in MSE).
Extension to multiple time indices.

Joan Bruna Stat 153: Introduction to Time Series