See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/281718517

A survey of forecast error measures

Article in World Applied Sciences Journal · January 2013

DOI: 10.5829/idosi.wasj.2013.24.itmies.80032

CITATIONS READS
264 9,062

6 authors, including:

Maxim Shcherbakov Adriaan Brebels

Volgograd State Technical University KU Leuven
77 PUBLICATIONS 578 CITATIONS 13 PUBLICATIONS 601 CITATIONS

SEE PROFILE SEE PROFILE

Anton Tyukov
Volgograd State Technical University
15 PUBLICATIONS 322 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Building energy management View project

Proactive Decision Support Systems Design View project

All content following this page was uploaded by Adriaan Brebels on 24 March 2016.

The user has requested enhancement of the downloaded file.

World Applied Sciences Journal 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013
ISSN 1818-4952
© IDOSI Publications, 2013
DOI: 10.5829/idosi.wasj.2013.24.itmies.80032

A Survey of Forecast Error Measures

Maxim Vladimirovich Shcherbakov, Adriaan Brebels,

Nataliya Lvovna Shcherbakova, Anton Pavlovich Tyukov,
Timur Alexandrovich Janovsky and Valeriy Anatol’evich Kamaev

Volgograd State Technical University, Volgograd, Russia

Submitted: Aug 7, 2013; Accepted: Sep 18, 2013; Published: Sep 25, 2013
Abstract: This article reviews the common used forecast error measurements. All error measurements have been
joined in the seven groups: absolute forecasting errors, measures based on percentage errors, symmetric errors,
measures based on relative errors, scaled errors, relative measures and other error measures. The formulas are
presented and drawbacks are discussed for every accuracy measurements. To reduce the impact of outliers, an
Integral Normalized Mean Square Error have been proposed. Due to the fact that each error measure has the
disadvantages that can lead to inaccurate evaluation of the forecasting results, it is impossible to choose only
one measure, the recommendations for selecting the appropriate error measurements are given.

Key words: Forecasting Forecast accuracy Forecast error measurements

INTRODUCTION et
= ( yt − ft(m) ) (1)
Different criteria such as forecast error measurements,
the speed of calculation, interpretability and others have where - yt is the measured value at time t, ft
( m) - predicted
been used to assess the quality of forecasting [1-6].
value at time t, obtained from the use of the forecast
Forecast error measures or forecast accuracy are the most
model m. Hereinafter referred to as the index of the model
important in solving practical problems [6]. Typically, the
(m) will be omitted.
common used forecast error measurements are applied for
Mean Absolute Error, MAE is given by:
estimating the quality of forecasting methods and for
choosing the best forecasting mechanism in case of n
1
multiple objects. A set of "traditional" error measurements
in every domain is applied despite on their drawbacks.
MAE
= = ∑
n i =1
ei mean ei ,
i =1, n
(2)

These error measurements are used as presets in domains

where n –forecast horizon, mean(•) – a mean operation.
despite on drawbacks.
Median Absolute Error, MdAE is obtained using the
This paper provides an analysis of existing and quite
following formula
common forecast error measures that are used in
forecasting [4, 7-10]. Measures are divided into groups
MdAE = median ei ,
according to the calculating method an value of error for i =1, n
(3)
certain time t. The calculating formula, the description of
the drawbacks, the names of assessments are considered
where mean(•) – operation for calculation of a median.
for each error measure.
Mean Square Error, MSE is calculated by the formula

A Review n
∑( ) ( )
Absolute Forecasting Error: The first group is based on = 1
MSE = ei2 mean ei2 , (4)
the absolute error calculation. It includes estimates based n i =1 i =1, n
on the calculation of the value ei

Corresponding Author: Shcherbakov, Volgograd State Technical University, Lenin avenue, 28, 400005, Volgograd, Russia.
171
 World Appl. Sci. J., 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

hence, Root Mean Square Error, RMSE is calculated as:

RMdSPE
= median (100 ⋅ pi
i =1, n
)2 , (10)
n
∑( ) ( )
1
RMSE
= = ei2 mean ei2 (5) We note the following shortcomings.
n i =1
i =1, n
Appearance division by zero when the actual value
These error measures are the most popular in various is equal to zero.
domains [8, 9]. However, absolute error measures have the Non-symmetrical issue - the error values differ
following shortcomings. whether the predicted value is bigger or smaller than
the actual [12-14].
The main drawback is the scale dependency [9]. Outliers have significant impact on the result,
Therefore if the forecast task includes objects with particularly if outlier has a value much bigger then
different scales or magnitudes then absolute error the maximal value of the "normal" cases [4].
measures could not be applied. The error measures are biased. This can lead to an
The next drawback is the high influence of outliers in incorrect evaluation of the forecasting models
data on the forecast performance evaluation [11]. performance [15].
So, if data contain an outliers with maximal value
(this is common case in real world tasks), then Symmetric Errors: The criteria which have been included
absolute error measures provide conservative values. in this group are calculated based on the value:
RMSE, MSE have a low reliability: the results could
be different depending on different fraction of data et
[4]. st = . (11)
( yt + ft )
Measures Based on Percentage Errors: Percentage errors The group includes next measures. Symmetric
are calculated based on the value Pt Mean Absolute Percentage Error, sMAPE is calculated
according to
et
pt = (6) n
yt 1
sMAPE
=
n ∑ 200=
⋅ si mean ( 200 ⋅ si ) ,
i =1, n
(12)
Also these errors are the most common in forecasting i =1

domain. The group of percentage based errors includes and the median mean absolute percentage error
the following errors.
Mean Absolute Percentage Error, MAPE sMdAPE median ( 200 ⋅ si ) .
=
i =1, n (13)
n
1
MAPE
=
n ∑100 ⋅=
pi mean (100 ⋅ pi
i =1, n
) (7) To avoid the problems associated with the division
i =1 by zero, a modified sMAPE - Modified sMAPE, msMAPE
Median Absolute Percentage Error, MdAPE is more has been proposed. Their denominators have an
resistant to outliers and calculated according to the additional member:
formula
n
1 y −f
MdAPE median (100 ⋅ pi
= ) msMAPE =
n ∑ ( yi + ifi ) /i2 + Si , (14)
i =1, n (8) i =1

Root Mean Square Percentage Error, RMSPE is i −1 i −1

calculated according to: Si =
1
where
i −1 ∑
yk − yi −1 , yi −1 =
1
i −1
yk . . ∑
= k 1= k 1

RMSPE
= mean (100 ⋅ pi
i =1, n
)2 , (9) Developing the idea for the inclusion of an additional
terms, more sophisticated measures was presented [16]:
and the median percentage error of the quadratic KL-N, KL-N1, KL-N2, KL-DE1, KL-DE2, IQR

172
 World Appl. Sci. J., 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

The following disadvantages should be noted. If naive model has been chosen then division by zero
error occurs in case of continuous sequence of
Despite its name, this error is also non-symmetric identical values of the time series.
[13].
Furthermore, if the actual value is equal to forecasted Scaled Error: As a basis for calculating the value of the
value, but with opposite sign, or both of these values scaled errors qi is given by
are zero, then a divide by zero error occurs.
These criteria are affected by outliers in analogous et
qt = .
with the percentage errors. 1
n

If more complex estimations have been used, the ∑

n − 1 i=2
yi − yi −1 (19)
problem of interpretability of results occurs and this
fact slows their spread in practice [4]. This group contains Mean Absolute Scaled Error,
MASE proposed in [9]. It is calculated according to:
Measures Based on Relative Errors: The basis for
calculation of errors in this group is the value determined MASE = mean qi , (20)
i =1, n
as follows:
Another evaluation of this group is Root Mean
Square Scaled Error, RMSSE is calculated by the formula
et
rt = , [10]:
( yt − ft* ) (15)

RMSSE = mean qi2 .

i =1, n
( ) (22)
where ft*
- the predictive value obtained using a reference
model prediction (benchmark model). The main practice is These measures is symmetrical and resistant to
to use a naive model as a reference model outliers. However, we can point to two drawbacks.

ft* = yt −l , If the forecast horizon real values are equal to each

(16) other, then division by zero occurs.
where l - the value of the lag and l = 1. Besides it is possible to observe a weak bias
The group includes the next measures. Mean Relative estimates if you do the experiments by analogy with
Absolute Error, MRAE is given by the formula [15].

MRAE = mean ri , Relative Measures: This group contains of measures

i =1, n
(17) calculated as a ratio of mentioned above error measures
obtained by estimated forecasting models and reference
Median Relative Absolute Error, MRAE is calculated models. Relative Mean Absolute Error, RelMAE is
according to calculated by the formula.

MdRAE = median ri , MAE

RMAE = ,
i =1, n
(18) MAE* (23)

and Geometric Mean Relative Absolute Error, GMRAE), where MAE and MAE* the mean absolute error for the
which is calculated similarly to (17), but instead of mean(•) analyzed forecasting model and the reference model
the geometric mean is obtained gmean(•). respectively, calculated using the formula (2).
Relative Root Mean Square Error, RelRMSE is
It should be noted the following shortcomings. calculated similarly to (23), except that the right side is
calculated by (5)
Based the formulas (15-18), division by zero error
occurs, if the predictive value obtained by reference RMSE
RRMSE = .
model is equal to the actual value. RMSE* (24)

173
 World Appl. Sci. J., 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

In some situations it is reasonable to calculate the over the entire interval or time horizon or defined
logarithm of the ratio (23). In this case, the measure is short interval of observation [18]. However, this
called the Log Mean Squared Error Ratio, (LMR) estimate is affected by influence of outliers, if outlier has
a value much bigger the maximal "normal" value. To
 RMSE  reduce the impact of outliers, Integral Normalized Mean
LMR = log  .
 RMSE *  (25) Square Error [19] have been proposed, calculated by the
formula:
Syntetos et al. proposed a more complex assessment
of the relative geometric standard deviation Relative
Geometric Root Mean Square Error, RGRMSE [17].
inRSE = n
1
( )
mean ei2 .

The next group of measures counts the number of ∑ yi i =1, n

(29)
i =1
cases where the error of the model prediction error is
greater than the reference model. For instance, PB (MAE) Some research contains the the ways of NRMSE
- Percentage Better (MAE), calculated by the formula: calculation as [16]:

({
100% ⋅ mean I MAE < MAE * .
PB ( MAE ) = }) (26) n
∑ ( ei )
2

where I{•} - the operator that yields the value of zero or inRSE = i =1 , (30)
n
∑ ( yi − y )
2
one, in accordance with the expression:
i =1
0, if MAE < MAE *;
I ( MAE ) =  (27) n
 1. where y=
1
n ∑ yk .
k =1
By analogy with PB (MAE), Percentage Better (MSE)
Other measures are called normalized std_APE and
can be defined.
std_MAPE [20, 21] and calculated by the formula
The disadvantages of these measures are the
n
following.
∑ ( ei − MAE )2
Std _ AE = i =1 (31)
Division by zero error occurs if the reference forecast n −1
error is equal to zero.
These criteria determine the number of cases when
and
the analyzed forecasting model superior to the base
but do not evaluate the value of difference.
n

Other Error Measures: This group includes measures

∑ ( pi − MAPE )2
Std _ APE = i =1 (32)
proposed in various studies to avoid the shortcomings of n −1
existing and common measures.
To avoid the scale dependency, Normalized Root
respectively.
Mean Square Error (nRMSE) has been proposed,
As a drawback, you can specify a division by zero
calculated by the formula:
error if normalization factor is equal to zero.

nRMSE =
1
y
( )
mean ei2 ,
(28)
Recommendations How to Choose Error Measures:
i =1, n One of the most difficult issues is the question of
choosing the most appropriate measures out of the
where y - the normalization factor, which is usually equal groups. Due to the fact that each error measure has the
to either the maximum measured value on the forecast disadvantages that can lead to inaccurate evaluation of
horizon, or the difference between the maximum and the forecasting results, it is impossible to choose only one
minimum values. Normalization factor can be calculated measure [5].

174
 World Appl. Sci. J., 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

We provide the following guidelines for choosing the ACKNOWLEDGMENTS

error measures.
Authors would like to thank RFBR for support of the
If forecast performance is evaluated for time series research (Grants #12-07-31017, 12-01-00684).
with the same scale and the data preprocessing
procedures were performed (data cleaning, anomaly REFERENCES
detection), it is reasonable to choose MAE, MdAE,
RMSE. In case of different scales, these error 1. Tyukov, A., A. Brebels, M. Shcherbakov and
measures are not applicable. The following V. Kamaev, 2012. A concept of web-based energy
recommendations are provided for mutli-scales cases. data quality assurance and control system. In the
In spite of the fact that percentage errors are Proceedings of the 14th International Conference on
commonly used in real world forecast tasks, but due Information Integration and Web-based Applications
to the non-symmetry, they are not recommended. & Services, pp: 267-271.
If the range of the values lies in the positive 2. Kamaev, V.A., M.V. Shcherbakov, D.P. Panchenko,
half-plane and there are no outliers in the data, it is N.L. Shcherbakova and A. Brebels, 2012. Using
advisable to use symmetric error measures. Connectionist Systems for Electric Energy
If the data are "dirty", i.e. contain outliers, it is Consumption Forecasting in Shopping Centers.
advisable to apply the scaled measures such as Automation and Remote Control, 73(6): 1075-1084.
MASE, inRSE. In this case (i) the horizon should be 3. Owoeye, D., M. Shcherbakov and V. Kamaev, 2013.
large enough, (ii) no identical values should be, (iii) A photovoltaic output backcast and forecast method
the normalized factor should be not equal to zero. based on cloud cover and historical data. In the
If predicted data have seasonal or cyclical patterns, Proceedings of the The Sixth IASTED Asian
it is advisable to use the normalized error measures, Conference on Power and Energy Systems (AsiaPES
wherein the normalization factors could be calculated 2013), pp: 28-31.
within the interval equal to the cycle or season. 4. Armstrong, J.S. and F. Collopy, 1992. Error measures
If there is no results of prior analysis and a-prior for generalizing about forecasting methods: Empirical
information about the quality of the data, it comparisons. International Journal of Forecasting,
reasonable to use the defined set of error measures. 8(1): 69-80.
After calculating, the results are analyzed with 5. Mahmoud, E., 1984. Accuracy in forecasting: A
respect to division by zero errors and contradiction survey. Journal of Forecasting, 3(2): 139-159.
cases: 6. Yokuma, J.T. and J.S. Armstrong, 1995. Beyond
For the same time series the results for model m1 accuracy: Comparison of criteria used to select
is better than m2, based on the one error forecasting methods. International Journal of
measure, but opposite for another one; Forecasting, 11(4): 591-597.
For different time series the results for model m1 7. Armstrong, J.S., 2001. Evaluating forecasting
is better in most cases, but worst for a few of methods. In Principles of forecasting: a handbook for
cases. researchers and practitioners. Norwell, MA: Kluwer
Academic Publishers, pp: 443-512.
CONCLUSION 8. Gooijer, J.G.D. and R.J. Hyndman, 2006. 25 years of
time series forecasting. International Journal of
The review contains the error measures for time series Forecasting, 22(3): 443-473.
forecasting models. All these measures are grouped into 9. Hyndman, R.J. and A.B. Koehler, 2006. Another look
seven groups: absolute forecasting error, percentage at measures of forecast accuracy. International
forecasting error, symmetrical forecasting error, measures Journal of Forecasting, 22(4): 679-688.
based on relative errors, scaled errors, relative errors and 10. Theodosiou, M., 2011. Forecasting monthly
other (modified). For each error measure the way of and quarterly time series using STL
calculation is presented. Also shortcomings are defined decomposition. International Journal of Forecasting,
for each of group. 27(4): 1178-1195.

175
 World Appl. Sci. J., 24 (Information Technologies in Modern Industry, Education & Society): 171-176, 2013

11. Shcherbakov, M.V. and A. Brebels, 2011. Outliers and 18. Tyukov, A., M. Shcherbakov and A. Brebels, 2011.
anomalies detection based on neural networks Automatic two way synchronization between server
forecast procedure. In the Proceedings of the 31st and multiple clients for HVAC system. In the
Annual International Symposium on Forecasting, Proceedings of The 13th International Conference on
ISF-2011, pp: 21-22. Information Integration and Web-based Applications
12. Goodwin, P. and R. Lawton, 1999. On the asymmetry & Services, pp: 467-470.
of the symmetric MAPE. International Journal of 19. Brebels, A., M.V. Shcherbakov and V.A. Kamaev,
Forecasting, 15(4): 405-408. 2010. Mathematical and statistical framework for
13. Koehler, A.B., 2001. The asymmetry of the sAPE comparison of neural network models with other
measure and other comments on the M3-competition. algorithms for prediction of Energy consumption in
International Journal of Forecasting, 17: 570-574. shopping centres. In the Proceedings of the 37 Int.
14. Makridakis, S., 1993. Accuracy measures: Theoretical Conf. Information Technology in Science Education
and practical concerns. International Journal of Telecommunication and Business, suppl. to Journal
Forecasting, 9: 527-529. Open Education, pp: 96-97.
15. Kolassa, S. and R. Martin, 2011. Percentage errors 20. Casella, G. and R. Berger, 1990. Statistical inference.
can ruin your day (and rolling the dice shows how). 2nd ed. Duxbury Press, pp: 686.
Foresight, (Fall): 21-27. 21. Kusiak, A., M. Li and Z. Zhang, 2010. A data-driven
16. Assessing Forecast Accuracy Measures. Date View approach for steam load prediction in buildings.
01.08.2013 http:// www.stat.iastate.edu/ preprint/ Applied Energy, 87(3): 925-933.
articles/2004-10.pdf
17. Syntetos, A.A. and J.E. Boylan, 2005. The accuracy
of intermittent demand estimates. International
Journal of Forecasting, 21(2): 303-314.

176

View publication stats