MATH 7339 - Machine Learning and Statistical Learning Theory 2

Section. Times Series Introduction

1. Overview
2. Times Series Models
3. Examples
Overview of Times Series

1. Time series models

(a) Stationarity.
(b) Autocorrelation function.
(c) Transforming to stationarity.
2. Time domain methods
(a) AR/MA/ARMA models.
(b) ACF and partial autocorrelation function.
(c) Forecasting
(d) Parameter estimation
(e) ARIMA models/seasonal ARIMA models
3. Spectral analysis
(a) Spectral density
(b) Periodogram
(c) Spectral estimation
v "Time Series Analysis and Its Applications", 4th ed. 2017, by Shumway and Stoffer.

• Introduction to time series and exploratory techniques. Time plots, calculation of

the sample autocorrelation. (Class Notes. Shumway and Stoffer Ch. 1 and 2)

• Time Series Regression (Class Notes. Shumway and Stoffer Ch. 2)

• ARMA modeling. Estimation of autoregressive moving averages processes via

frequentist and Bayesian approaches. Model diagnostics, forecasting and
applications. (Class Notes. Shumway and Stoffer Ch. 3)

• Spectral estimation using Fourier analysis and Filtering. Bayesian Approach (Class
Notes. Shumway and Stoffer Ch. 4)
 Notation and Terminology :

• 𝑋! , 𝑋" , . . . , 𝑋# is a stochastic process, i.e., a collection of random

variables indexed by a set 𝑇.
• 𝑥! , 𝑥" , . . . , 𝑥# is a single realization.

• If 𝑇 consists of real numbers (or a subset), the process is called a

continuous time stochastic process.

• If T is restricted to integers (or a subset), the process is called a

discrete time stochastic process.

• These processes may take on values which are real or restricted to

integers and are called continuous state space or discrete state
space respectively.

• Time series analysis is generally restricted to discrete time, continuous state

space stochastic processes.

• Continuous time, continuous state space stochastic processes are generally

covered in stochastic processes.
Ø Time Series Models

Time Series are data collected in a sequence. They are usually evenly
spaced and because of the sequential nature are statistically dependent
observations.

A time series model specifies the joint distribution of the sequence of

random variables: {𝑋# } .

For example:

𝑃(𝑋! ≤ 𝑥! , … , 𝑋# ≤ 𝑥# ) for all 𝑡 and 𝑥! , 𝑥" , . . . , 𝑥#

We’ll mostly restrict our attention to second-order moments properties only:

𝐸[𝑋# ], 𝐶𝑜𝑣(𝑋$ , 𝑋# ).
Closing price of S&P 500 stocks

At the end of each month At the end of each week

𝑋# is a discrete time, continuous state space stochastic process.

Example: White noise

Typically, we are thinking of a sequence of random variables that

may be dependent on one another, 𝑥! , … , 𝑥% . There may be times
when we want to think of this as an infinite list … , 𝑥&! , 𝑥' , 𝑥! , …

One model 𝑋# with which we are already familiar consists of a sequence

of uncorrelated random variables. When the mean is zero and the
sequence is indexed by time (𝑡), this is usually called white noise.

𝐸 𝑋# = 0, 𝑉𝑎𝑟 𝑋# = 𝜎 " 𝑎𝑛𝑑 𝐶𝑜𝑣 𝑋$ , 𝑋# = 0 for all 𝑠 ≠ 𝑡

Denote 𝑋# ~ 𝑊𝑁(0, 𝜎 " )

Note: Uncorrelated RVs does not imply they are independent.

Independent RVs implies they are uncorrelated.
Example: iid white noise.

𝑋# independent and identically distributed, i.e.,

𝑃 𝑋! ≤ 𝑥! , … , 𝑋# ≤ 𝑥# = 𝑃 𝑋! ≤ 𝑥! ⋯ 𝑃(𝑋# ≤ 𝑥# )

It is NOT interesting for forecasting:

𝑃 𝑋# ≤ 𝑥# | 𝑋! , … , 𝑋#&! = 𝑃(𝑋# ≤ 𝑥# )
Example: Gaussian white noise

𝑋# ~ 𝑁𝑜𝑟𝑚𝑎𝑙(0, 𝜎 " )

For example, when 𝜎 = 1,

)!
1 " /"
𝑃 𝑋# ≤ 𝑥# = H 𝑒 &) 𝑑𝑥
2𝜋 &(
Example: Binary i.i.d. Process

𝑃 𝑋# = 1 = 𝑝; 𝑃 𝑋# = −1 = 1 − 𝑝

Example 𝑃 𝑋# = 1 = 𝑃 𝑋# = −1 = 1/2
Example: Random Walk

A model for analyzing trend is the random walk model. Your current position is
determined by where you were at the last step plus the random step that you
just took. So, the equation would be

𝑆# = O 𝑋+
+,!
where 𝑋+ is the iid noise.

For example, if {𝑋+ } is the binary

process as last example,
Random walks:

Differences: ∇𝑆# = 𝑆# − 𝑆#&! = 𝑋#

Five Random Walks

Mean: 𝐸 𝑆# =

10
8
6
4
values

Variance: 𝐕𝐚𝐫 𝑆# =
2
0
−2
−4

5 10 15 20

Time
S&P500 data

Differences: ∇𝑆# = 𝑆# − 𝑆#&! = 𝑋#

Trend and Seasonal Models

𝑋# = 𝑇# + 𝑆# + 𝐸#

= 𝛽' + 𝛽! 𝑡 + O 𝛼+ cos(𝜆+ 𝑡) + 𝛾+ sin(𝜆+ 𝑡) + 𝐸#

+
𝑋# = 𝑇# + 𝐸#
= 𝛽' + 𝛽! 𝑡 + 𝐸#
𝑋# = 𝑇# + 𝑆# + 𝐸#

= 𝛽' + 𝛽! 𝑡 + O 𝛼+ cos(𝜆+ 𝑡) + 𝛾+ sin(𝜆+ 𝑡) + 𝐸#

+
Trend and Seasonal Models: Residuals
Time Series Modelling (Chasing stationarity)

Step 1. Some of the features of time series data we look out for are:

• Trend.
• Periodicity / Seasonality.
• Is the mean changing over time?
• Is the variation changing over time?
• Are there abrupt/step changes?
• Are there outliers?

Time series plots will be an important tool.

Step 2. Transform data so that residuals are stationary

(1) Estimate and subtract 𝑇# 𝑎𝑛𝑑 𝑆# .

(2) Differencing.
(3) Nonlinear transformations (log, root function).

Step 3. Fit model to residuals.

 S&P 500 data. Differencing and Trend

Define the lag-1 difference operator, (think ‘first derivative)

∇ 𝑋# = 𝑋# − 𝑋#&! = 1 − 𝐵 𝑋#

where 𝐵 is the backshift operator, 𝐵𝑋# ∶= 𝑋#&!

• If 𝑋# = 𝛽' + 𝛽! 𝑡 + 𝑌# , then

∇ 𝑋# = 𝛽! + ∇𝑌#

• If 𝑋# = 𝛽' + 𝛽! 𝑡 + 𝛽" 𝑡 " + ⋯ + 𝛽- 𝑡 - + 𝑌# , then

∇ 𝑋# = 𝛽! + 𝛽" 𝑡 " − 𝑡 − 1 " + ⋯ +𝛽- 𝑡 - − 𝑡 − 1 - + ∇𝑌#

and
∇- 𝑋# = 𝑘! 𝛽- + ∇- 𝑌#
Define the lag-𝑠 difference operator,

∇$ 𝑋# ≔ 𝑋# − 𝑋#&$ = 1 − 𝐵 $ 𝑋#

where 𝐵 $ is the backshift operator applied 𝑠 times.

If 𝑋# = 𝑇# + 𝑆# + 𝑌# and 𝑆# has period 𝑠, (i.e., 𝑆# = 𝑆#&$ ), then

∇$ 𝑋# = 𝑇# − 𝑇#&$ + ∇$ 𝑌#
Objectives of Time Series Analysis

1. Compact description of data.

2. Interpretation.
3. Forecasting.
4. Control.
5. Hypothesis testing.
6. Simulation.
Classical decomposition: An example

Monthly sales for a souvenir shop at a beach resort town in Queensland.

(Makridakis, Wheelwright and Hyndman, 1998)

Transformed data Trend
Residuals Trend and seasonal variation
Objectives of Time Series Analysis

1. Compact description of data.

Example: Classical decomposition: 𝑋# = 𝑇# + 𝑆# + 𝑓(𝑌# ) + 𝑊# .
2. Interpretation. (e.g., Seasonal adjustment.)
3. Forecasting. (e.g., Predict sales, unemployment. )
4. Control. (e.g., Example: Impact of monetary policy on unemployment)
5. Hypothesis testing. (e.g., Global warming.)
6. Simulation. (e.g.,Estimate probability of catastrophic events)
Unemployment data

Monthly number of unemployed people in Australia. (Hipel and McLeod, 1994)

Trend
Trend plus seasonal variation Residuals
Predictions based on a (simulated) variable
Two Approaches to Time Series

There are two primary approaches to time series.

1. Time domain approach. This approach focuses on the rules for a

time series to move forward.

For example, how do yesterday's and today's observations affect

tomorrow's observation?

2. Frequency domain approach. This approach tries to understand how

differing oscillations can contribute to current observations.

For example, taking hourly temperatures in Boston. There will be a

very clear 24 hour oscillation. There will be another clear 8,760 hour
oscillation. The current temperature is a sum of these two sinusoids
(plus a lot of noise and fluctuation).
References:

• "Time Series Analysis and Its Applications", 4th ed. 2017, by Shumway and Stoffer.

• "Introduction to Time Series and Forecasting", 3rd ed. 2016, by Brockwell and Davis.

• Time Series Analysis 1st Edition, by James Douglas Hamilton

• Time Series: Theory and Methods (Springer Series in Statistics) by Peter

Brockwell, Richard Davis

• Facebook: Forecasting at scale: https://facebook.github.io/prophet/

Interesting and useful sources of economic and finance data:

• General data assembled by the St. Louis Fed: research.stlouisfed.org/fred2/.

• Interest rate data from the Fed Board of Governors:
www.federalreserve.gov/econresdata/statisticsdata.htm.
• Other data from the Fed Board of Governors:
www.federalreserve.gov/releases/h15/update/.
• Data from the World Bank: data.worldbank.org/.
• Stock price data: finance.yahoo.com/.

Reading and manipulating stock prices from Yahoo Finance.

• There is a useful R program written by John Nolan.
• Python: https://pypi.org/project/yfinance/
• MATLAB: https://www.mathworks.com/matlabcentral/fileexchange/68361-
yahoo-finance-and-quandl-data-downloader