Modeling and Forecasting

Forecasting Methods
Measuring the Accuracy of Forecast

FORECASTING AND MODEL SELECTION

Anurag Prasad

Department of Mathematics and Statistics

Indian Institute of Technology Kanpur, India

REACH Symposium, March 15-18, 2008

1 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Outline

1 Modeling and Forecasting

2 Forecasting Methods

3 Measuring the Accuracy of Forecast

2 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Assumptions of Forecasting

1 Element of Uncertainty
2 Blind Spots
3 Change in Forecast Accuracy

3 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Framework of a Forecast System

Model−Building Phase Forecasting Phase

Theory and/or
Historical Data

No No
Model Model Diagnostic Forecast Stability Forecast
Specification Estimation Checking Generation Checking Updation
Yes Yes

New
Data
Observations

4 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Choice of a Particular Forecast Model

1 Degree of Accuracy Required

2 Cost of Producing Forecasts
3 Forecast Horizon
4 Degree of Complexity Required
5 Available Data

5 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Classification of Estimation Methods

1 Time Series Methods

2 Causal Methods
3 Judgemental Methods

6 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Time Series Methods

• Use historical data as a basis

• Underlying patterns are fairly stable

1 Autoregressive Moving Average (ARMA)

2 Exponential Smoothing
3 Extrapolation
4 Linear Prediction
5 Trend Estimation
6 Growth Curve
7 Box-Jenkins Approach

7 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Causal Methods

• Belief that some other time series can be useful

• Assumption that it is possible to identify the underlying factors

1 Regression Analysis
* Linear Regression
* Non-Linear Regression
2 Econometrics

8 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Judgemental Methods

• Incorporate intuitive judgements, opinions and probability

estimates

1 Composite Forecasts
2 Surveys
3 Delphi Method
4 Scenario Building
5 Technology Forecasting
6 Forecast by Analogy

9 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Forecast Error

For t = 1, . . . , N,
y (t) : Actual value at period t,
yb(t) : Forecast value at period t;
e(t) : Forecast error at period t; e(t) = y (t) − yb(t)

^y(t )
y(t) i

y(t )
i

t
t i

10 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Graphical Measures of Forecast Accuracy

Plot of y (t) versus yb(t)

• Keep the same scale for both the axes.

• Departure of points from the 450 line through origin indicates
imperfect forecasts.

11 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

y(t) y(t)

^
y(t) ^
y(t)
Correct Model Form Incorrect Model Form

12 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Plot of e(t) versus t

• Reveals patterns of variability which the model has failed to
explain.
• For a good model, the forecast errors should vary in a
horizontal band around zero.

13 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

e(t) e(t)
0 0

t t
Correct Model Form Incorrect Model Form

e(t) e(t)
0 0

t t
Incorrect Model Form Incorrect Model Form

14 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Descriptive Measures of Forecast Accuracy

Descriptive Measures of Forecast Accuracy are used to...

1 Provide a single, easily interpreted measure of model’s
reliability
2 Compare the accuracy of two different models
3 Search for an optimal model
4 Monitor a model’s performance

15 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Descriptive Measures of Forecast Accuracy

Mean Absolute Deviation (MAD)

N
|forecast error| |e(t)|
MAD = number of forecasts = t=1
N

Mean Square Error (MSE)

N
(forecast error)2 e(t)2
MSE = number of forecasts = t=1
N

Root Mean Square Error (RMSE)

√
RMSE = MSE

16 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Descriptive Measures of Forecast Accuracy

Mean Absolute Percent Error (MAPE)

|forecast error/actual value|
MAPE = number of forecasts .100%
N
|e(t)/y (t)|
= t=1
N .100%

Pearson’s Correlation Coefficient (r )

between y (t) and yb(t)
N ¯
t=1 (y (t)−ȳ)(y (t)−y)
r=

 

N 2 N ¯ 2
t=1 (y (t)−ȳ) t=1 (y (t)−y)

17 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Descriptive Measures of Forecast Accuracy

"No Change" model is : yb(t + 1) = y (t)

Theil’s Inequality Coefficient (U)

RMSE("new" model)
U= RMSE("no change" model)

U>1 ⇒ worse than "no change" model

U=1 ⇒ as good as "no change" model
U<1 ⇒ better than "no change" model

18 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Further Readings

Quantitative Forecasting Methods, N.R. Farnum and L.W.

Stanton, 1989, PWS-KENT Publishing Co.
Statistical Methods for Forecasting, B. Abraham and J.
Ledolter, 1983, John Wiley & Sons
Introduction to Time Series and Forecasting, P.J. Brockwell
and R.A. Davis, 2002 (Second Edition) , Springer-Verlag
Time Series Analysis and Forecasting, R. Yaffee, 2000,
Academic Press

19 Forecasting and Model Selection

 Modeling and Forecasting
Forecasting Methods
Measuring the Accuracy of Forecast

Thank You

20 Forecasting and Model Selection