Time Series Analysis

• A time series is a random sequence {Xt} recorded in a time-ordered fashion.

Application –

• Economics: daily stock market quotations or monthly unemployment rates.

• Social sciences: population series, such as birthrates or school enrollments.
• Epidemiology: the number of influenza cases observed over some time period.
• Medicine: blood pressure measurements traced over time for evaluating drugs.
• Global warming?

Time Series Statistical Models

The Stationary Time Series Model
• Distribution of the error term, as well as the correlation between error terms
a few periods apart are constant over time.
• For example - AR, MA, ARMA, ARIMA, EWMA, ARCH, GARCH, EGARCH, GJR-
GARCH, etc.

STATIONARITY CONDITION FOR AUTO-REGRESSIVE (AR) MODELS

• Series is stable over time and historical data is useful for future
predictions
• (A) Strict Stationery: All four moments defining the distribution of
observations are time-invariant. That is, the joint probability of observing
n values is the same irrespective of time.
• (B) Weakly Stationary: Assumes that only the first two moments are
time-invariant.

• An autoregressive model, or an autoregressive process, is one where a

variable y depends on past values of itself.
• The general representation with p lagged values (yt-1, yt-2 , … , yt-p ) is called
an autoregressive model (process) of order p, abbreviated as AR(p), and is
given by:

• AR models can be used to describe the time paths of variables and capture
their correlations between current and past values; they are generally used
for forecasting. Past values are used to forecast future values.

Method for detecting stationarity

• Correlograms
i) ACF Plots
ii) PACF Plots
• Statistical Test
i) Dickey-Fuller (DF test)
ii) Augmented Dickey-Fuller (ADF test)

ACF and PACF Plots

• Autocorrelation function shows relation between autocorrelation and lags.
Autocorrelation between Yt and its lagged values Yt-k (That is, at lag K) is
determined as follows:
• Autocorrelation is the correlation between a time series with a lagged version
of itself. The ACF starts at a lag of 0, which is the correlation of the time series
with itself and therefore results in a correlation of 1
• We use the plot_acf function from the statsmodels.graphics.tsaplots library
 PACF Plot
• In time series data, typically, a large proportion of correlation between
Yt & Yt-k happens due to their correlation with intermediate variables.
• PACF measures correlation between Yt and Yt-k after removing the
correlation that Yt could have with the intermediate lags (Yt-1 , Yt-2 , …,
Yt-k+1 ).
• We’ll use the plot_pacf function from the statsmodels.graphics.tsaplots
library with the parameter method = "ols" (regression of time series on
lags of it and on constant)

• Below you can see an example of an ACF and PACF plot. These plots are
called “lollipop plots”

• Both the ACF and PACF start with a lag of 0, which is the correlation of
the time series with itself and therefore results in a correlation of 1.
• The difference between ACF and PACF is the inclusion or exclusion of
indirect correlations in the calculation.
• Additionally, you can see a blue area in the ACF and PACF plots. This blue
area depicts the 95% confidence interval and is an indicator of the
significance threshold. That means, anything within the blue area is
statistically close to zero and anything outside the blue area is statistically
non-zero.
• To determine the order of the model, you check: “How [many] lollipops
are above or below the confidence interval before the next lollipop
enters the blue area?
Auto Regressive Process (AR)
Assume that a current value of the series is linearly dependent upon its previous
value, with some error. Then we could have the linear relationship

where εt is a white noise time series. [That is, the εt are a sequence of
uncorrelated random variables (possibly normally distributed, but not
necessarily normal) with mean 0 and variance σ 2 .] This model is called an
autoregressive (AR) model since X is regressed on itself. Here the lag of the
autoregression is 1. More generally, we could have an autoregressive model
of order p, an AR(p) model, defined by

AR(1) Process
The following time series is an AR(1) process with 128 timesteps and alpha_1 =
0.5. It meets the precondition of stationarity.
 We can make the following observations:
• There are several autocorrelations that are significantly non-zero. Therefore,
the time series is non-random.
• High degree of autocorrelation between adjacent (lag = 1) in PACF plot
• Geometric decay in ACF plot

Based on the above table, we can use an AR(1) model to model this process.

With AR(p=1), the formula

As you can see, the AR(1) model fits an alpha_1 = 0.4710, which is quite close to
the alpha_1 = 0.5 that we have set.

AR(2) Process
The following time series is an AR(2) process with 128 timesteps, alpha_1 = 0.5,
and alpha_2 = -0.5. It meets the precondition of stationarity.
We can make the following observations:
• There are several autocorrelations that are significantly non-zero. Therefore,
the time series is non-random.
• High degree of autocorrelation between adjacent (lag = 1) and near-adjacent
(lag = 2) observations in PACF plot
• Geometric decay in ACF plot
As you can see, the AR(2) model fits an alpha_1 = 0.5191 and alpha_2 = -0.5855,
which is quite close to the alpha_1 = 0.5 and alpha_2 = -0.5 that we have set.

Moving Average Models

The moving average process of finite order is considered an approximation to the
Wold representation that happens to be a moving average process of infinite order.
The time period at t is impacted by the unexpected external factors at various slots t-
1, t-2, t-3, ….., t-k. These unexpected impacts are known as Errors or Residuals. The
impact of previous time spots is decided by the coefficient factor α at that particular
period of time. The price of a share of any particular company X may depend on some
company merger that happened overnight or maybe the company resulted in
shutdown due to bankruptcy. This kind of model calculates the residuals or errors of
past time series and calculates the present or future values in the series in know as
Moving Average (MA) model.

The 1st-order moving average model, denoted by MA(1) is:

The 2nd-order moving average model, denoted by MA(2) is:

Examples

ACF plot
• The sample ACF for the simulated data follows. The pattern is typical for situations where an
MA(2) model may be useful. There are two statistically significant “spikes” at lags 1 and 2
followed by non-significant values for other lags. Note that due to sampling error, the
sample ACF did not match the theoretical pattern exactly.
Autoregressive Moving Average Model: ARMA(p,q)
• Autoregressive moving average models are simply a
combination of an AR model and an MA model
• In this model, the impact of previous lags along with the residuals is
considered for forecasting the future values of the time series. Here β
represents the coefficients of the AR model and α represents the
coefficients of the MA model.

• Hence, this model can explain the relationship of a time series with both
random noise (moving average part) and itself at a previous step
(autoregressive part).

• Consider the above graphs, where the MA and AR values are plotted with their
respective significant values. Let's assume that we consider only 1 significant
value from the AR model and, likewise 1 significant value from the MA model.
So the ARMA model will be obtained from the combined values of the other
two models will be of the order of ARMA(1,1).

• ARMA models are capable of capturing very complex patters of temporal

correlation.
• Thus, they are a useful and interesting class of models.
• In fact, they can capture any valid autocorrelation
 How to estimate an ARMA model with R ?
In practice, estimating an ARMA model is nothing else than estimating a
multilinear regression model where the related coefficients c0, a1,...,ap, b1,
... , bq, and the response variable is Y(t). Indeed, the common used function
with R software is the function called "arma()". Here is the standard code.

where
1. X : is the univariate time series or data to use for estimation. This correspond
to our variable Yt in our case.
2. ordrer = c(1,1) : is a vector of 2 elements defining the lag order of the AR(p)
and MA(q). In order words, p is the first element of the vector and q the
second. To transform ARMA(p,q) into an AR(p) ( an MA(q)) model, we just
have to set q=0 (p=0).
3. lag = c(p,q) : this option can be used as an alternative to the option "order" in
order to define le values of p and q. So, it is a list defining the values p and q
for the AR(p) and MA(q) respectively.
4. coef : it a vector of the coefficients of an ARMA obtained from an initial
estimation.
5. include.intercept : A boolean value saying if we should include the intercept
in our estimation or not.
6. series : is the name of the series
7. qr.tol : is the stopping value of the iteration from the estimation procedure of
the standard error of the coefficients.

Example- An estimation with the Bitcoin data

• We run a simple estimation of an ARMA(p,q) model using the bitcoin

monthly data from January 2015 to November 2020 and Data is from
Yahoo Finance.
• The code to follow-up for importing the data frame from your
computer and transform it into a monthly time series
• After plotting the results of the time series of the monthly adjusted
close price of BTCUSD, we obtain the following graph. As you can see
the BTCUSD displays a deterministic and a stochastic trend. The
deterministic trend is directly observed via the red trend line, while the
stochastic trend is observed by the variability of the time series
changing with time.

• Let us assume now that the monthly bitcoin data can be fitted by an
ARIMA(2,1,2) model, meaning that it has an autoregressive pattern with
2 lags and a moving average with 2 lags. Let then try to estimate the
coefficients of this model specification and see what we obtain. Here is
the code.

• Therefore the results of the estimations are presented on the following

image. We obtain the table of coefficients of the AR(2) part of BTCUSD
and MA(2) considered in the model.
• The first line of the table represent the estimated values of theirs
related coefficients and the second row represents their corresponding
asymptotic standard errors. In addition we have the log-likelihood of
the estimation and the AIC. These values are nothing else than the
criteria used to select the optimal number of the parameters p, q and r
in order to get the best fit.

• As the estimation is done, we can plot the original data and the
predicted one all together on the same figure in order to check how
well we have done so far. Here is the code to follow-up

• Therefore, we obtain the following figure. As you can see, we are doing
pretty well with a certain lag. The original series is the solid black line
while the predicted one is the red line.
Forecasting using ARMA model
• As we have already estimated the model, we will forecast it. For such
purpose, we can use the function Predict(). It is worth noting that the
forecasted value of Bitcoin for the next month is evaluated as the
conditional expectation of the true value, given its past value. Here is the
code for the prediction.

• We run first a one-month ahead forecast and then a 5-months ahead

forecast. Hence, the results of the code are presented as follows. Also we
have the related standard error. According to this model, the BTCUSD
has the potential to reach the value of $20301.96 in December 2020, and
may reach the value of $20879.90 in April 2021

• In the previous section we have presented how to forecast a single point.

But in terms of forecasting, one could be interested in the band in which
the price of BTCUSD could variate. That's why the interval forecast are
useful. Here is the related code
• And we obtain the following plot of the 5 months ahead forecast. The
figure of the forecasts is not that clear because the standard error is big.
This induce that the confidence intervals become more large.
 ARIMA Model
• Arima, in short term as Auto-Regressive Integrated Moving Average, is a
group of models used in R programming language to describe a given
time series based on the previously predicted values and focus on the
future values. The Time series analysis is used to find the behavior of
data over a time period. This model is the most widely used approach
to forecast the time series.Arima() function is used to process the
model.

• ARIMA being an easier model in predicting a future value in series, takes

time series data which are equally spaced points in a time(a pattern of
value, rate of change of growth, outliers, or noise between the time
points). Maximum Likehood Estimation (MLE) is used to estimate the
ARIMA model. The model takes up three important parameters: p,d,q,
respectively.MLE helps to maximize the likelihood for these parameters
when calculating parameter estimates.

ARIMA(p,d,q)

p- is the order of Auto-regressive or linear model

q – is the order of Moving Average/ number of lagged values
d- difference value to make the time series stationary from non-stationary. So
we perform ARMA here, not ARIMA(means no Integration). The
improvement over ARIMA is Seasonal ARIMA.

Now let us see how these three parameters bound each other and, lastly, the
plots of ACF and PACF. As AR uses its lag errors as predictors. And works
best when the used predictor is not independent of each other. The Model
works on two important key concepts:
1. The Data series as input should be stationary.
2. As ARIMA takes past values to predict the future output, the input data must
be invariant.
Implementation Steps:

1. Load the data set after installing the package forecast.

2. The Steps of Pre-processing are done, which creates a separate time-series or
timestamp.
3. Making Time-series stationary and check the required transformations.
4. The difference value ‘d’ will be performed.
5. The core important step in ARIMA is plotting ACF and PACF.
6. Determine the two parameters p and q from the plots.
7. The previously created value fits the Aroma model and predicts the future
values. The Fitting Process is also named as Box-Jenkins Method.
8. Doing Validation.

• auto. Arima() function is used for automatic prediction and ARIMA

Models. This function uses unit root tests, minimization of the AIC and
MLE to obtain an ARIMA model. To make the series stationary, we need
to differentiate a previous value from the current value.
• d= pval-cval, if the value is already stationary the d=0.
• predict() – Used to predict the model based on the results of the
various fitting model used.

Example: CO2 Emission Forecasting of Indonesia

• Because this is not a stationary time series, we have to find the
differenced data with the previous value.

• If the differenced data has stationary properties, then it can be

modelled using the ARMA model. Therefore, we can find which perfect
lag for the model from it.

• To do that, we have to make the value as a time series object and then
we look at the result of the time-series that is applied on diff function.
Here is the code look like,

• And here is the visualization result from it,

• Based on this visualization, we can see that the differenced data has
slightly more stationary properties to it and there’s no trend on it.
Therefore, we can use this differenced data to find which AR and MA
parameters are good for our ARIMA model.

• To find which AR’s and MA’s parameter values that suit our ARIMA
model, we can use ACF and PACF function to find which are the best
value,

• Here is the function that we can visualize the ACF and PACF plot,
Here is the result,

• Based on this plot, we know that the ACF and the PACF function have
slightly tail off on it and it’s not reaching 0’s quickly. Because of that, we
can assume that it will use parameter p with 1 and parameter q also
with 1. Therefore, the ARMA model that suits is ARMA(1,1).

• To make the model becomes an ARIMA model, we just add another

parameter, which is d that stands for the differences that are given to
the data. With that, our ARIMA model will be ARIMA(1, 1, 1) where
each parameter represents the p, d, and q. The p is AR model
parameter, d is the amount of the differences that gives to the data,
and q is the MA model parameter.

Model Diagnostic
• The ARIMA(1, 1, 1) will be our model if there are no significant changes
between the other model. In this case, we will try ARIMA(2, 1, 1) and
ARIMA(1, 1, 2) to see if the parameter has a significant p-value on it.

• We can diagnose this model simply by using this function below,

 Here are the p-values for each parameters,

• As we can see here, the ARIMA(1,1,1) has a significant p-value on its

parameters. Then, both models, ARIMA(1,1,2) and ARIMA(2,1,1), have a
non-significant p-values on it. By changing its parameters doesn’t have
any effect on the model. Therefore, we can use ARIMA(1,1,1) model.

• After we chose the best model for the data, we have to diagnose one
more thing, which is the residuals. Below are the residual analysis result
from the sarima function,
• If we want to use this model, the residual must have a normal
distribution to it and there is no correlation between each lag of its
values. Based on this visualization, we can see that the residual don’t
have any significant autocorrelation to it as we can see from the ACF
plot, most of the points also get aligned on the line as we can see from
the Q-Q plot, and the Ljung-Box statistic p values doesn’t cross the blue
line. Therefore, we can use this model to do forecasting.

Forecasting
• If we want to do some forecasting, we can use sarima.for function to do
it. Each parameter for this function consists of the data, what time
ahead we want to forecast, and then set the p, d, and q parameter that
represents our ARIMA model.

• In this example, we want to know what is the CO2 emission amount

that will be produced for the next 35 years, and what is the amount of
CO2 emission that is produced in 2050.

• Here is the function that is used to make a forecast,

• As we can see here, the black dot represents our data, and the red one
represents the forecast result. As we can see here, the forecast result
also shows the grey boundary between the red line. It basically
indicates the standard error from the forecast result. The boundary
doesn’t have a large boundary on it, therefore the model has a good
result to it.
 ARCH and GARCH MODEL
ARCH and GARCH. These model(s) are also called volatility model(s). These
models are exclusively used in the finance industry as many asset prices are
conditional heteroskedastic.

ARCH — Autoregressive Conditional Heteroskedasticity

GARCH — Generalized Autoregressive Conditional Heteroskedasticity

• These models relate to economic forecasting and measuring volatility.

• Some of the techniques adopted in the finance sector — ARCH, ARCH-
M, GARCH, GARCH-M, TGARCH, and EGARCH.
• ARCH model is concerned about modeling volatility of the variance of
the series.
• These model(s) deals with stationary (time-invariant mean) and
nonstationary (time-varying mean) variable(s).
• Some of the real-time examples where ARCH model(s) applied: Stock
prices, oil prices, bond prices, inflation rates, GDP, unemployment
rates, etc.,

What is the ARCH model?

ARCH is an Autoregressive model with Conditional Heteroskedasticity .

• “The ARCH process introduced by Engle (1982) explicitly recognizes the

difference between the unconditional and the conditional variance
allowing the latter to change over time as a function of past errors.”

• Autoregressive: The current value can be expressed as a function of the

previous values i.e. they are correlated.

• Conditional: This informs that the variance is based on past errors.

• Heteroskedasticity: This implies the series displays unusual variance

(varying variance).

Why an ARCH model?

• Autoregressive models can be developed for univariate time-series data

that is stationary (AR), has a trend (ARIMA), and has a seasonal
component (SARIMA). But, these Autoregressive models do not model
is a change in the variance over time.
• The error terms in the stochastic processes generating the time series
were homoscedastic, i.e. with constant variance.
• There are some time series where the variance changes consistently
over time. In the context of a time series in the financial domain, this
would be called increasing and decreasing volatility.
 • Volatility in Finance: Degree of variation price series over time as
measured by the standard deviation of the series. Suppose that Si is the
value of a variable on a day ‘i’. The volatility per day is the standard
deviation of ln(Si /Si-1).
• In time series where the variance is increasing in a systematic way, such
as an increasing trend, this property of the series is called
heteroskedasticity. This means changing or unequal variance across the
series.
• If this change in the variance can be correlated over time, then it can be
modeled using an autoregressive process, such as ARCH.

When to apply an ARCH Model?

• In practice, this can be used to model the expected variance on the

residuals after another autoregressive model has been used, such as an
ARMA or similar.

• Since we can only tell whether the ARCH model is appropriate or not by
squaring the residuals and examining the correlogram, we also need to
ensure that the mean of the residuals is zero.

• Crucially, ARCH should only ever be applied to series that do not have
any trends or seasonal effects, i.e. that have no (evident) serially
correlation. ARIMA is often applied to such a series (or even Seasonal
ARIMA), at which point ARCH may be a good fit.

ARCH Model of Order Unity:

ARCH(p) model is simply an AR(p) model applied to the variance of a time
series.

ARCH(1):

• A time-series {ϵ(t)} is given at each instance by ϵ(t) = w(t)*σ(t)

• where w(t) is the white noise with zero mean and unit variance.

• Var(x(t)) = σ²(t) = ⍺0+⍺1 * σ²(t-1)

• where ⍺0, ⍺1 are parameters of the model and ⍺0 > 0, ⍺1 ≥ 0 to ensure

that the conditional variance is positive. σ²(t-1) is lagged square error.

• We say that ϵ(t) is an autoregressive conditional heteroskedastic model

of order unity, denoted by ARCH(1).

• ϵ(t) = w(t)* σ(t) = w(t)* ⎷(⍺0 + ⍺1 *ϵ²(t-1))

• similarly ARCH(2):

• ϵ(t) = w(t)* σ(t) = w(t)* ⎷(⍺0 + ⍺1 * ϵ²(t-1) + ⍺2 * ϵ²(t-2))

 • similarly ARCH(p):

• ϵ(t) = w(t) * ⎷(⍺0 + ⍺(p) * ∑ ϵ²(t-i)

• where: p is the number of lag squared residual errors to include in the

ARCH model.i = (1,2,3,-,-,-, -, p) tells us the number of logged periods of
the square error.

Interpretation:

• If the error is high during the period (t-1), it is more likely that the value
of error at the period (t) is also higher.
• vice versa — If the error is low during the period (t-1) then the value
inside sqrt will be low which results in a decreased error in (t).
• Remember, ⍺1 ≥ 0 for the positive variance.
• For the stability condition to hold, ⍺1 < 1, otherwise ϵ(t) will be
explosive (continue to increase over time).

What is a GARCH model?

Generalized Autoregressive Conditional Heteroskedasticity, or GARCH, is an

extension of the ARCH model that incorporates a moving average
component together with the autoregressive component.

Introduction of moving average component allows the model:

• To model the conditional change in variance over time.

• Changes in the time-dependent variance.
• Examples include conditional increases and decreases in the variance.

• Thus GARCH is the “ARMA equivalent” of ARCH, which only has an

autoregressive component. GARCH models permit a wider range of
behavior more persistent volatility.

GACH Model of Order p, q — GARH(p,q):

GARCH(1,1):

• Here we are going to consider a single autoregressive lag and a single

“moving average” lag. The model is given by the following:

• ϵ(t) = w(t) * σ(t)

 • ϵ(t) =w(t) * ⎷(⍺0 + ⍺1 *ϵ²(t-1)) + β1 * σ²(t−1)

• Similarly GARH(p,q):

• A time-series {ϵ(t)} is given at each instance by ϵ(t) = w(t)*σ(t)

• and σ²(t) is given by:

• where α(i) and β(j) are parameters of the model.

• ⍺0 > 0, ⍺i ≥ 0, i =1, … q, β≥ 0, j = 1, … p imposed to ensure that the

conditional variances are positive.

• Here we are adding moving average term, that is the value of σ² at t,

σ²(t), is dependent upon previous σ²(t-j) values.

Interpretation:

• The large value of β1 causes σ(t) to be highly correlated with σ²(t−1)

and gives the conditional standard deviation process a relatively long-
term persistence, at least compared to its behavior under an ARCH
model.
• For p = 0 the process reduces to the ARCH(q) process.
• For p = q = 0, ϵ(t) is simply white noise.

How to configure the ARCH and GARCH Model(s):

• The configuration for an ARCH model is best understood in the context

of ACF and PACF pl
•
• ots of the variance of the time series.
• This can be achieved by subtracting the mean from each observation in
the series and squaring the result, or just squaring the observation if
you’re already working with white noise residuals from another model.
• The ACF and PACF plots can then be interpreted to estimate values for
p and q, in a similar way as is done for the ARMA model

ARCH and GARCH models Python code:

We look at the generalized python code using the above formula:

Import required libraries

2. ARCH(1) model:

output graph:
3. ARCH(1) Squared model:

output graph:

• Observation: ACF and PACF seem to show significance at lag 1

indicating an AR(1) model for the variance may be appropriate.
4. GARCH(1,1) model:

output graph:

• Observation: We have noticed that overall this process closely

resembles white noise, however, take a look when we view the squared
eps series.
5. GARCH(1,1) Squared model:

output graph:

• Observation: we can observe clearly autocorrelation present and the

significance of the lags in both the ACF and PACF indicates we need
both AR and MA components for our model.
6. Fitting GARCH(1,1) model to our simulated EPS Series:

Now, let's run the above model through an example using “SPY returns”,

• Iterate through combinations of ARIMA(p, d, q) models to best fit our

time series.
• Pick the GARCH model orders according to the ARIMA model with the
lowest AIC.
• Fit the GARCH(p, q) model to our time series.
• Examine the model residuals and squared residuals for autocorrelation.
output: aic — -5255.56734 | order: (3, 0, 2)

We can say ARIMA(3,0,2) best fits our time series.