Comput Stat

DOI 10.1007/s00180-016-0646-6

ORIGINAL PAPER

Comparison of Value-at-Risk models using the MCS

approach

Mauro Bernardi1 · Leopoldo Catania2

Received: 10 March 2015 / Accepted: 28 January 2016

© Springer-Verlag Berlin Heidelberg 2016

Abstract This paper compares the Value-at-Risk (VaR) forecasts delivered by alterna-
tive model specifications using the Model Confidence Set (MCS) procedure recently
developed by Hansen et al. (Econometrica 79(2):453–497, 2011). The direct VaR
estimate provided by the Conditional Autoregressive Value-at-Risk (CAViaR) mod-
els of Engle and Manganelli (J Bus Econ Stat 22(4):367–381, 2004) are compared to
those obtained by the popular Autoregressive Conditional Heteroskedasticity (ARCH)
models of Engle (Econometrica 50(4):987–1007, 1982) and to the Generalised Autore-
gressive Score (GAS) models recently introduced by Creal et al. (J Appl Econom
28(5):777–795, 2013) and Harvey (Dynamic models for volatility and heavy tails:
with applications to financial and economic time series. Cambridge University Press,
Cambridge, 2013). The MCS procedure consists in a sequence of tests which permits
to construct a set of “superior” models, where the null hypothesis of Equal Predictive
Ability (EPA) is not rejected at a certain confidence level. Our empirical results, sug-
gest that, during the European Sovereign Debt crisis of 2009–2010, highly non-linear
volatility models deliver better VaR forecasts for the European countries as opposed to
other regional indexes. Model comparisons have been performed using the R package
MCS developed by the authors and freely available at the CRAN website.

B Mauro Bernardi
[email protected]
Leopoldo Catania
[email protected]

1 Department of Statistical Sciences, University of Padua, Via Cesare Battisti 241–243,

35121 Padua, Italy
2 Department of Economics and Finance Faculty of Economics, University of Rome Tor Vergata,
Via Columbia, 2, 00133 Rome, Italy

123
 M. Bernardi, L. Catania

Keywords Hypothesis testing · Model Confidence Set · Value-at-Risk ·

VaR combination · ARCH · GAS · CAViaR models

1 Introduction

During last decades hundreds of models have been developed, estimated and validated
from both an empirical and theoretical perspective. As a result, several alternative
model specifications are usually available to the econometricians to address the same
empirical problem. Just to confine our considerations within a given family, and with-
out claiming to be complete, the Autoregressive Conditional Heteroskedastic (ARCH)
models of Engle (1982) and Bollerslev (1986), for example, have seen an exponen-
tially increasing number of different specifications in the last few decades. Despite
their popularity, they do not exhaust the set of models introduced for dynamic condi-
tional volatility modelling which includes also the stochastic volatility models initially
proposed by Taylor (1994) and extensively studied by Harvey and Shephard (1996)
and Gallant et al. (1997) within the context of non-linear state space models. The fam-
ily of dynamic conditional volatility models has been recently enlarged by the GAS
model of Harvey (2013) and Creal et al. (2013) also known as Dynamic Conditional
Score (DCS). The availability of such an enormous number of models raises the ques-
tion of providing a statistical method or procedure that delivers the “best” model with
respect to a given criterion. Furthermore, a model selection issue appears to be nec-
essary to reduce the uncertainty when the usual comparing procedures do not deliver
an unique result. This could happen for example, when models are compared in terms
of their predictive ability, so that models that produce better forecasts are preferred.
Unfortunately, when evaluating the performances of different forecasting models it is
not always trivial to establish which one clearly outperforms the remaining available
alternatives. This problem is particularly relevant even from an empirical perspective
especially when the set of competing alternatives is large. As observed by Hansen and
Lunde (2005) and Hansen et al. (2011), it is unrealistic to expect that a single model
dominates all the competitors either because the different specifications are statisti-
cally equivalent or because there is not enough information coming from the data to
univocally discriminate the models.
Recently, several alternative procedures have been developed to deliver the “best
fitting” model, see e.g., the Reality Check (RC) of White (2000), the Stepwise Multiple
Testing procedure of Romano and Wolf (2005), the Superior Predictive Ability (SPA)
test of Hansen (2005) and the Conditional Predictive Ability (CPA) test of Giacomini
and White (2006). Among those multiple-testing procedures, the Model Confidence
Set procedure (MCS) of Hansen et al. (2003, 2011) consists in a sequence of statistic
tests which permits to construct, the “Superior Set of Models” (SSM), where the null
hypothesis of equal predictive ability (EPA) is not rejected at certain confidence level
α. The EPA statistic test is evaluated for an arbitrary loss function, which essentially
means that it is possible to test models on various aspects depending on the chosen
loss function. The possibility to specify user supplied loss functions enhances the
flexibility of the procedure that can be used to test several different aspects. The
MCS procedure starts from an initial set of m competing models, denoted by M0 , and

123
Comparison of Value-at-Risk models using the MCS approach

∗
results in a smaller set of superior models, the SSM, denoted by M̂1−α . Of course, the
most discriminating scenario is when the final set consists of a single model. At each
iteration, the MCS procedure tests the null hypothesis of EPA among the competing
models and ends with the creation of the SSM only if the null hypothesis is accepted,
otherwise the MCS is iterated again and the EPA is tested on a smaller set of models
obtained by eliminating the worst one at the previous step.
This paper compares the Value-at-Risk (VaR) forecasts delivered by alternative
model specifications recently introduced in the financial econometric literature, using
the MCS procedure, similarly to Caporin and McAleer (2014) and Chen and Ger-
lach (2013). More specifically, the direct quantile estimates obtained by the dynamic
CAViaR models of Engle and Manganelli (2004) are compared with the VaR fore-
casts delivered by several ARCH-type models of Engle (1982) and Bollerslev (1986)
and with those obtained by two different specifications of the GAS models of Creal
et al. (2013) and Harvey (2013). The CAViaR model of Engle and Manganelli (2004)
has been proven to provide reliable quantile-based VaR estimates in several empir-
ical experiments, see for example Chen et al. (2012). During the last few decades,
the ARCH-type models of Bollerslev (1986) have became a standard approach to
model the conditional volatility dynamics. Those approaches are compared to the
new class of score driven models, which are promising in modelling highly nonlinear
volatility dynamics, as documented by Creal et al. (2013) and Harvey (2013). Up to
our knowledge, VaR forecasting comparisons using the MCS procedure has not been
considered using nonlinear GAS dynamic models. Our empirical results suggest that,
during periods of financial instability, such as the recent Global Financial Crisis (GFC)
of 2007–2008 or the recent European Sovereign debt crisis, highly non-linear volatil-
ity models deliver better VaR forecasts for the European countries. On the contrary,
for the North America and Asia Pacific regions we find quite homogeneous results
with respect to the models’ complexity. As discussed in Kowalski and Shachmurove
(2014), this empirical finding is consistent with the greater impact that the GFC had
in the European financial markets as compared to the non-Euro areas.
The models belonging to the superior set delivered by the Hansen’s procedure can
then be used for different purposes. For example, they can be used to forecast future
volatility levels, to predict the future observations, conditional to the past information,
see, e.g., Gneiting (2011), or to deliver future Value-at-Risk estimates, as argued by
Bernardi et al. (2015). Alternatively, the models can be combined together to obtain
better forecast measures. Since the original work of Bates and Granger (1969), a
lot of papers have argued that combining predictions from alternative models often
improves upon forecasts based on a single “best” model. In an environment where
observations are subject to structural breaks and models are subject to different levels
of misspecification, a strategy that pools information coming from different models
typically performs better than methods that try to select the best forecasting model. Our
analysis also considers the Dynamic Quantile Averaging (DQA) technique proposed
by Bernardi et al. (2015) in order to aggregate the VaR forecasts delivered by the SSM,
conditional on model’s past out-of-sample performances as in Samuels and Sekkel
(2011) and Samuels and Sekkel (2013). For further informations about the application
of the model averaging methodology the reader is referred to Bernardi et al. (2015).

123
 M. Bernardi, L. Catania

Our results confirm that, under an optimal combination of models, individual VaR
forecasts can be substantially improved with respect to standard backtest measures.
Model comparisons are performed using the R (R Development Core Team 2013)
package MCS developed by the authors. The MCS package provides an integrated
environment for the comparison of alternative models or model’s specifications within
the same family using the MCS procedure of Hansen et al. (2011) and it is freely
available on the CRAN repository, http://cran.r-project.org/web/packages/MCS/index.
html. The MCS package is very flexible since it allows for the specification of the
model’s types and loss functions that can be supplied by the user. This freedom allows
for the user to concentrate on substantive issues, such as the construction of the initial
set of model’s specifications, M0 , without being limited by the constraints imposed
by the software.
The layout of the paper is as follows. In Sect. 2 we present the MCS procedure of
Hansen et al. (2011) highlighting the alternative specifications of the test statistics.
Section 3 details about the alternative model specifications used to compare VaR
forecasts. Section 4 covers the empirical application which also aims at illustrating
the implementation of the procedure using the provided package. Section 5 concludes
the paper.

2 The MCS procedure

The availability of several alternative model specifications being able to adequately

describe the unobserved data generating process (DGP) opens the question of selecting
the “best fitting model” according to a given optimality criterion. This paper imple-
ments the MCS procedure recently developed by Hansen et al. (2011) with the purpose
of comparing the VaR forecasts obtained by several alternative model specifications.
The Hansen’s procedure consists of a sequence of statistic tests which permits to
construct a set of “superior” models, the “Superior Set Models” (SSM), where the
null hypothesis of equal predictive ability (EPA) is not rejected at a certain confi-
dence level α. The EPA statistic tests is calculated for an arbitrary loss function that
satisfies general weak stationarity conditions, meaning that we could test models on
various aspects, as, for example, punctual forecasts, as in Hansen and Lunde (2005),
or in-sample goodness of fit, as in Hansen et al. (2011). Formally, let yt denote the
observation at time t and let ŷi,t be the output of model i at time t, the loss function
i,t associated to the ith model is defined as
 
i,t =  yt , ŷi,t , (1)

and measures the difference between the output ŷi,t and the “a posteriori” realisation
yt . As an example of loss function, Bernardi et al. (2015) consider the asymmetric
VaR loss function of González-Rivera et al. (2004) to compare the ability of different
GARCH specifications to predict extreme loss within the framework of high-frequency
financial data setting. The asymmetric VaR loss function of González-Rivera et al.
(2004) is defined as
    
 yt , VaRτt = τ − dtτ yt − VaRτt , (2)

123
Comparison of Value-at-Risk models using the MCS approach

where VaRτt denotes the τ -level  predicted VaR at time t, given information up to
time t − 1, Ft−1 , and dtτ = 1 yt < VaRτt is the τ -level quantile loss function. The
asymmetric VaR loss function represents the natural candidate to backtest quantile-
based risk measures since it penalises more heavily observations below the τ th quantile
level, i.e. yt < VaRτt . Details about the loss function specifications can be found in
Hansen and Lunde (2005).
We now briefly describe how the MCS procedure is implemented. The procedure
starts from an initial set of models M0 of dimension m, encompassing all the alternative
model specifications, and delivers, for a given confidence level α, a smaller set, the
superior set of models (SSM), M̂1−α∗ , of dimension m ∗ ≤ m. The SSM, M̂∗ , contains
1−α
all the models having superior predictive ability according to the selected loss function.
Of course, the most discriminating scenario is when the final set consists of a single
model, i.e., m ∗ = 1. Formally, let di j,t denote the loss differential between models i
and j at time t:

di j,t = i,t −  j,t , i, j = 1, . . . , m, t = 1, . . . , n, (3)

and let 
di·,t = (m − 1)−1 di j,t , i = 1, . . . , m, (4)
j∈M\{i}

be the average loss of model i relative to any other model j at time t. The EPA
hypothesis for a given set of models M can be formulated in two alternative ways:

H0,M : ci j = 0, for all i, j = 1, 2, . . . , m

HA,M : ci j = 0, for some i, j = 1, . . . , m, (5)

or

H0,M : ci· = 0, for all i = 1, 2, . . . , m

HA,M : ci· = 0, for some i = 1, 2, . . . , m, (6)
 
where ci j = E di j and ci· = E (di· ) are assumed to be finite and time independent.
According to Hansen et al. (2011), the two hypothesis defined in equations (5)–(6)
can be tested by constructing the following two statistics

d̄i j
ti j =   , (7)
 d̄i j
var
d̄i,·
ti· =   , (8)
 d̄i,·
var


for i, j ∈ M, where d̄i,· = (m − 1)−1 j∈M\{i} d̄i j is the average loss of the ith
model relative
 to the average losses across the models belonging to the set M, and
d̄i j = n −1 nt=1 di j,t measures the relative average loss between models i and j. The

123
 M. Bernardi, L. Catania

       
 d̄i,· and var
variances var  d̄i j are bootstrapped estimates of var d̄i,· and var d̄i j ,
   
 d̄i,· and var
respectively. The bootstrapped variances var  d̄i j,t are calculated by
performing a block-bootstrap procedure where the block length p is set as the maxi-
mum number of significants parameters obtained by fitting an AR( p) process on the
di j terms. Details about the implemented bootstrap procedure can be found in White
(2000), Kilian (1999), Clark and McCracken (2001), Hansen et al. (2003, 2011),
Hansen and Lunde (2005), and Bernardi et al. (2015). The statistic ti j is used in the
well know test for comparing two forecasts; see e.g., Diebold and Mariano (2002) and
West (1996), while the second one is used in Hansen et al. (2003) and Hansen et al.
(2011). As discussed in Hansen et al. (2011), the two EPA null hypothesis presented
in Eqs. (5)–(6) map naturally into the two test statistics

TR,M = max | ti j | and Tmax,M = max ti· , (9)

i, j∈M i∈M

where ti j and ti are defined in Eqs. (7)–(8). Since the asymptotic distributions of the
two test statistics is nonstandard, the relevant distributions under the null hypothesis
 
is estimated using a bootstrap procedure similar to that used to estimate var d̄i,· and
 
var d̄i j . For further details about the bootstrap procedure, see e.g., White (2000),
Hansen et al. (2003), Hansen (2005), Kilian (1999) and Clark and McCracken (2001).
As said in the Introduction, the MCS procedure consists on a sequential testing
procedure, which eliminates at each step the worst model, until the hypothesis of
equal predictive ability (EPA) is accepted for all the models belonging to the SSM.
At each step, the choice of the worst model to be eliminated has been made using an
elimination rule that is coherent with the statistic test defined in Eqs. (7)–(8) which
are
⎧ ⎫
⎨ d̄i j ⎬ d̄i,·
eR,M = arg max sup    , emax,M = arg max √   , (10)
i ⎩ j∈M ⎭ i∈M  d̄i,·
var
 d̄i j
var

respectively. Summarazing, the MCS procedure to obtain the SSM, consists of the
following steps:

1. set M = M0 ;
2. test for EPA-hypothesis: if EPA is accepted terminate the algorithm and set
∗
M̂1−α = M, otherwise use the elimination rules defined in Eq. (10) to determine
the worst model;
3. remove the worst model, and go to step 2.

Note that, since the 1−α coverage probability refers to the superior models’ inclusion,
∗
then, there is no guarantee that the final set M̂1−α does not contain any inferior models.
To address this possibility, in the next sections, we also describe how to implement a
procedure that optimally combines the VaRs forecasts of those models included in the
SSM.

123
Comparison of Value-at-Risk models using the MCS approach

3 Model specifications

In our empirical illustration we apply the MCS procedure detailed in Sect. 2 to com-
pare the VaR forecasts obtained by fitting a list of popular models introduced in the
econometric literature over the last few decades. The autoregressive conditional het-
eroskedastic models, introduced by Engle (1982) and Bollerslev (1986), are probably
among the most widely employed tools in quantile-based quantitative risk manage-
ment. Here, we consider the ARCH-type models not only because of their popularity,
but also because of their ability to account for the main stylised facts about financial
returns. Moreover, since the seminal paper of Engle (1982), hundreds of different
specifications have been proposed, some of them having a huge flexibility in handling
series having different characteristics. Despite their vast popularity, the ARCH mod-
els are principally focused on the variance of the modelled conditional distributions.
The GAS models, recently introduced by Creal et al. (2013) and Harvey (2013), have
been raising popularity also because they nest some of the traditional dynamic con-
ditional variance approaches enlarging the class of time-varying parameter models.
For the purposes of this paper, the ARCH family of models is discussed in Sect. 3.1,
while GAS models are considered in Sect. 3.2. ARCH-type and GAS models provide
an indirect estimation of the quantile-based risk measures because of the parametric
assumption of the conditional distribution. A way to overcome the need to specify
the conditional distribution is to model directly the quantile as in the Conditional
Autoregressive Value-at-Risk (CAViaR) approach of Engle and Manganelli (2004).
The CAViaR specifications are briefly discussed in Sect. 3.3.
Let yt be the logarithmic return at time t, the following general model formulation
encompasses all the specifications considered throughout the paper:

yt | (Ft−1 , ζt , ϑ) ∼ D (yt , ζt , ϑ) , t = 1, 2, . . . , T, (11)

 
ζt = h ζt−1 , . . . , ζt− p , yt−1 , . . . , yt− p , | Ft−1 , (12)

where Ft is the information set up to time t, ζt is a vector of time-varying parameters, ϑ

is a vector or static parameters, ζt−1 , . . . , ζt− p and yt−1 , . . . , yt− p are lagged values
of the dynamic parameters and the observations, up to order p ≥ 1, respectively.
Finally, the function h (·) refers to one of the dynamics reported below, while D (·) is a
specified density function. As regards to the employed parameter estimation technique,
model parameters are estimated by maximum likelihood, see, e.g., Francq and Zakoian
(2011).

3.1 ARCH models

ARCH-type models are flexible and powerful tools for conditional volatility mod-
elling, because they are able to consider the volatility clustering phenomena as well as
other established stylised facts. Models belonging to this class, have been principally
proposed in order to describe the time-varying nature of the conditional volatility that
characterises financial time series. They have also become one of the most used tools
for researchers and practitioners dealing with financial market exposure. The sim-

123
 M. Bernardi, L. Catania

plest conditional volatility dynamics we consider is the GARCH(p,q) specification

introduced by Bollerslev (1986)


p 
q
σt2 = ω + αi εt−i−1
2
+ β j σt−
2
j−1 , (13)
i=1 j=1

where εt−i = yt−i − μ, for i = 1, 2, . . . , p and t = 1, 2, . . . , T , μ ∈ is the mean

ω > 0, 0 ≤
of yt , αq i < 1, ∀i = 1, 2, . . . , p and 0 ≤ β j < 1, ∀ j = 1, 2, . . . , q with
p
P ≡ i=1 αi + j=1 β j < 1 to preserve weak ergodic stationarity of the conditional
variance. Despite its popularity, the GARCH specification is not able to account for
returns exhibiting an asymmetric response of the conditional volatility to positive and
negative shocks as theorised by the “leverage effect” of Black (1976). Consequently,
in the financial econometric literature several alternative specifications have been pro-
posed. The EGARCH(p,q) model of Nelson (1991), for example, assumes that the
conditional volatility dynamics follows

  
p
  
q  
log σt = ω +
2
αi t−i + γi (| t−i | − E| t−i |) + β j log σt−
2
j , (14)
i=1 j=1

where t−i = σεt−i

t−i
, for i = 0, 1, . . . , p and t = 1, 2, . . . , T . The asymmetric response
is introduced through the γi parameters: for γi < 0 negative shocks will obviously
have a larger impact on future volatility than positive shocks of the same magnitude.
For the generic EGARCH(p,q) specification any positivity constraints is imposed q on
the parameters αi , β j , γi , and the persistence parameter P is equal to P = j=1 β j .
One of the most flexible models belonging to the ARCH family is the Asymmetric-
Power-ARCH(p,q) (APARCH, henceforth) model of Ding et al. (1993) which imposes
the following dynamic to the conditional variance


p 
q
σtδ = ω + αi (|εt−i | − γi εt−i )δ + δ
β j σt− j, (15)
i=1 j=1

where the δ parameter plays the role of a Box–Cox transformation (see Box and Cox
1964). To ensure the positiveness of the conditional variance the following parameter
restrictions are imposed: ω > 0, δ ≥ 0, 0 ≤ γi ≤ 0 for i = 1, 2, . . . , p and the
usual conditions αi ≥ 0, and β j ≥ 0, for i, j = 1, 2, . . . , max { p, q}. In the APARCH
specification the persistence strongly depends upon the distributional assumption made
on yt , i.e.
p 
q
P= αi κi + βj, (16)
i=1 j=1
 δ
where κi = E | t | − γi t , for i = 1, 2, . . . , p. The APARCH specification results
in a very flexible model that nests several of the most popular univariate ARCH
parameterisations, such as

123
Comparison of Value-at-Risk models using the MCS approach

(i) the GARCH(p,q) of Bollerslev (1986), for δ = 0 and γi = 0, for i = 1, 2, . . . , p;

(ii) the Absolute-Value-GARCH (AVARCH, henceforth) specification, for δ = 1
and γi = 0 for i = 1, 2, . . . , p, proposed by Taylor (1986) and Schwert (1990)
to mitigates the influence of large, in an absolute sense, shocks with respect to
the traditional GARCH specification;
(iii) the GJR-GARCH model (GJRGARCH, henceforth) of Glosten et al. (1993), for
δ = 2 and 0 ≤ γi ≤ 1 for i = 1, 2, . . . , p;
(iv) the Threshold GARCH (TGARCH, henceforth) of Zakoian (1994), for δ = 1,
which allows different reactions of the volatility to different signs of the lagged
errors;
(v) the Nonlinear GARCH (NGARCH, henceforth) of Higgins and Bera (1992), for
γi = 0 for i = 1, 2, . . . , p and β j = 0 for j = 1, 2, . . . , q.
Another interesting specification is the Component-GARCH(p,q) of Engle and Lee
(1993) (CGARCH, henceforth) which decomposes the conditional variance into a
permanent (ξt ) and transitory component in a straightforward way


p   q  
σt2 = ξt + αi εt−i
2
− ξt−i + β j σt−
2
j − ξ t− j (17)
i=1 j=1
 
ξt = ω + ρξt−1 + η εt−1
2
− σt−1
2
, (18)

p
where in order to ensure the stationarity of the process we impose
q i=1 αi +
j=1 β j < 1 and the additional condition that ρ < 1. Further parameters restrictions
for the positiveness of the conditional variance are given in Engle and Lee (1993). This
solution is usually employed because it permits to investigate the long and short-run
movements of volatility. The considered conditional volatility models are a minimal
part of the huge number of specifications available in the financial econometric litera-
ture. We chose these models because of their heterogeneity, since each of them focuses
on a different kind of stylised fact. Moreover, even if they look quite similar to each
other, the way in which they account for the stylised fact is different. For an extensive
and up to date survey on GARCH models we refer the reader to the works of Bollerslev
(2008), Teräsvirta (2009), Bauwens et al. (2006), Silvennoinen and Teräsvirta (2009)
and the recent book of Francq and Zakoian (2011).

3.2 GAS models

The GAS framework recently introduced by Creal et al. (2013) and Harvey (2013) is
gaining lots of consideration by econometricians in many field of time series analysis.
Under the Cox et al. (1981) classification the GAS models can be considered as a class
of observation driven models, with the usual consequence of having a closed form for
the likelihood and ease of evaluation. The key feature of GAS models is that the
predictive score of the conditional density is used as forcing variable into the updating
equation of a time-varying parameter. Two main reasons for adopting this updating
procedure has been given in literature. Harvey (2013), for example, argues that the

123
 M. Bernardi, L. Catania

GAS specification can be seen as an approximation to a filter for a models driven by

an unobservable stochastic latent parameter. Creal et al. (2013) instead consider the
conditional score as a steepest ascent direction for improving the model’s local fit
given the current parameter position, as it usually happens into the Newton–Raphson
algorithm. Moreover, the flexibility of the GAS framework make this class of models
nested with an huge amount of famous econometrics models such as, for example,
some of the ARCH-type models of Engle (1982) and Bollerslev (1986) for volatility
modelling, and also the MEM, ACD and ACI models of Engle (2002), Engle and Gallo
(2006), Engle and Russell (1998) and Russell (1999), respectively. Finally, one of the
practical implications of using this framework in order to update the time-varying
parameters is that it avoids the problem of using a non-adequate forcing variable
when its specification is not so obvious. In fact, it may be argued that, the use of the
conditional score, allows for the dynamic parameter to be updated considering all the
information coming from the entire distribution as usually happen into a state space
framework. On the contrary, many others observation driven models, such as ARCH-
type models, only make use of the expected value of the conditional distribution.
GAS model applications range in several interesting areas, such as, risk measure and
dependence modelling, Bernardi and Catania (2015), Lucas and Zhang (2014) and
Salvatierra and Patton (2014), volatility modelling, Harvey and Sucarrat (2014), and
in a nonlinear autoregressive setting, Koopman et al. (2015) and Delle Monache and
Petrella (2014).
Formally, let us consider the general model specification in Eqs. (11)–(12), where,
as before, D (·) denotes a probability density, ζt is a set of time-varying parameters and
ϑ is a vector of time-independent parameters. For example, D (·) may be a Student-t
distribution with fixed degree of freedom (λ = ν) and time-varying volatility (ζt = σt ).
Then, the updating equation for the time-varying parameters according to the GAS
framework is

ζt+1 = ωζ + αζ st + βζ ζt
st = St (ζt | ϑ) ∇t (yt , ζt | ϑ) ,

where ∇t (yt , ζt | ϑ) is the conditional score of the pdf D (·), evaluated at ζt

∂ ln D (yt , ζt | ϑ)
∇t (yt , ζt | ϑ) = ,
∂ζt

and St (ζt | ϑ) is a positive definite, possible parameter-dependent scaling matrix. A

convenient choice for the scaling matrix St (ζt | ϑ) is usually given by the Fisher
information matrix
St (ζt | ϑ) = [I (ζt | ϑ)]−a , (19)
where I (ζt | ϑ) can be written as:
 2 
∂ ln D (yt , ζt | ϑ)
I (ζt | ϑ) = −Et−1
∂ζt ∂ζt
 
= Et−1 ∇t (yt , ζt | ϑ) × ∇t (yt , ζt | ϑ) , (20)

123
Comparison of Value-at-Risk models using the MCS approach

and a is usually set equal to {0, 21 , 1}. Note that for a = 0 the scaling matrix St (ζt | ϑ)
coincides with the identity matrix. Creal et al. (2013) suggest to use the inverse Fisher
information matrix (a = 1) or its (pseudo)-inverse square root (a = 21 ) in order to scale
the conditional score for a quantity that accounts for its variance. Our empirical tests
confirm that scaling the conditional score by the Fisher information matrix is more
efficient than using an identity scaling matrix. However, sometimes the Fisher infor-
mation matrix is not available in closed form, and simulation or numerical evaluation
techniques should be used.
Another interesting property of the GAS framework is that it adapts quite naturally
to various reparameterisations of the problem in hand. This aspect is particularly useful
when the natural parameters’ space is constrained into a subset of the real line , and
therefore a mapping function λ (·) between this and the real line, becomes necessary.
For example, let us define  ∈ as the natural parameters’ space and λ : → 
an absolutely continuous deterministic invertible mapping function that maps the real
line into the natural parameter space . Moreover, let us define ζ̃t = λ−1 (ζt ) as
the unmapped version of the parameter ζt , then the GAS model suited for the new
time-varying parameter ζ̃t with a = 1 is defined as

ζ̃t+1 = ωζ + αζ s̃t + βζ ζ̃t , (21)

 
∂λ ζ̃t
where s̃t = λ̇t st , and λ̇t = . For other possible choices of a we refer to Creal
∂ ζ̃t
et al. (2013).
In the empirical application, we will consider the GAS specification for the parame-
ters of the Gaussian and Student-t
  distributions. Formally, the Gaussian GAS model
for the parameters ζtN = μt , σt2 is given by:
   
yt | Ft−1 , ζtN , ϑ ∼ N μt , σt2 , t = 1, 2, . . . , T,

with
        
μt+1 ωμ α 0 β 0 μt
= + μ s̃ + μ ,
σ̃t+1
2 ωσ 2 0 ασ 2 t 0 βσ 2 σ̃t2
 
where σ̃t2 = log σt2 and borrowing the previous notation

1  ⎡ ⎤
  0
(yt −μt )
1   σt
2

, It = σt ⎣ 2 ⎦.
2
λ̇t = , ∇t =
exp σ̃t2 − 2σ 2 1 − (yt −μ t)
1 1
0 2σt4 t σt2

The Student-t
 GAS
 model with time varying location, scale and shape parameters
ζtT = μt , φt2 , νt is given by:
   
yt | Ft−1 , ζtT , ϑ ∼ T μt , φt2 , νt , t = 1, 2, . . . , T,

123
 M. Bernardi, L. Catania

with
⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞⎛ ⎞
μt+1 ωμ αμ 0 0 βμ 0 0 μt
⎝ φ̃ 2 ⎠ = ⎝ωφ ⎠ + ⎝ 0 αφ 0 ⎠ s̃t + ⎝ 0 βφ 0 ⎠ ⎝φ̃t2 ⎠,
t+1
ν̃t+1 ων 0 0 αν 0 0 βν ν̃t

 
where φ̃t2 = log φt2 , ν̃t = log (νt − 2), and
⎛ ⎞
1


⎜ ⎟
λ̇t = ⎝exp φ̃t2 ⎠ ,
exp (ν̃t )
⎛ ν +1 ⎞
t
0 0
⎜ φt (νt +3)
2
νt ⎟
It = ⎜ ⎟
1
2φ 2 (νt +3)(νt +1) ⎠ ,
⎝ 0 2φt4 (νt +3)
1 h 1 (νt )
0 2φ 2 (νt +3)(νt +1) 2
⎛ (νt +1)(rt −μt ) ⎞
ν φ 2 +(rt −μt )2
⎜ 1 t t (νt +1)(r t −μt )
⎟
∇t = ⎝− φt + 3 ⎠,
ν φ +φ
 t t t2 t  t (r −μ )
h 2 rt , μt , φt , νt

and
 
1 νt + 1 1  νt  νt + 5
h 1 (νt ) = − ψ + ψ −
2 2 2 2 νt (νt + 3) (νt + 1)
  1 ν + 1 1 ν  π
t t
h 2 rt , μt , φt2 , νt = ψ − ψ −
2 2 2 2 2νt
" #
1 (rt − μt ) 2
(νt + 1) (rt − μt )2
− log 1 + +  ,
2 νt φt
2
2νt φ 2 νt + (rt − μt )2

where ψ (x) and ψ (x) are the digamma and the trigamma functions, respectively.
Recently, an OxMetrics package providing functions to estimate various specifications
of GAS models has been developed by Andres (2014).

3.3 Dynamic quantile models

The CAViaR models of Engle and Manganelli (2004), extends the standard quantile
regression model introduced by Koenker and Bassett (1978) and belongs to the family
of dynamic quantile autoregressive
 models
 proposed by Koenker and Xiao (2006).
Formally, let f t β τ ≡ f t xt−1 , β τ denote the τ th level conditional quantile for
τ ∈ (0, 1) at time t of the observed variable yt conditional to the information available
at time t − 1, i.e., xt−1 and the vector of unknown parameters β τ , the generic CAViaR
specification can be written as:

123
Comparison of Value-at-Risk models using the MCS approach

 
yt = f t β τ + t , t = 1, 2, . . . , T, (22)
  
p
  
q
 
ft β τ = θ τ + θiτ f t−i β τ + φ τj  xt− j , (23)
i=1 j=1

 
where θ = θ0 , θ1 , . . .q, θ p ∈
p+1 collects the autoregressive parameters, while φ =

φ1 , φ2 , . . . , φq ∈ ,  (·) is the function linking the lagged exogenous information

or past returns y1:t−1 = (y1 , y2 , . . . , yt−1 ) to the current conditional quantile and the
error term t . The measurement Eq. (22) is such that its  τ th level quantile is equal to
zero, i.e., qτ (t | x1:t−1 ) = 0 in order to ensure that f t β τ is the τ th level conditional
quantile of the observed variable yt given the observations up to time t − 1. As noted
by Engle and Manganelli (2004), the smooth evolution of the quantile over time is
guaranteed by the autoregressive terms denoted by βi f t−i (β), i = 1, 2, . . . , p, while
the function  (·) can be interpreted as the News Impact Curve (NIC) introduced by
Engle and Ng (1993) for ARCH-type models.
Parameters estimation of the τ th level quantile regression in the frequentist frame-
work is based on the solution of the following minimisation problem
"  #
 yt − f t β τ
arg min ρτ , (24)
β,σ
t
σ

where
 σ
 > 0 is the scale parameter, with ρτ (u) = u (τ − 1 (u < 0)), with respect to
β τ , σ . Alternatively, the Asymmetric Laplace distribution (ALD) can be used as mis-
specified likelihood function to perform maximum likelihood inference as suggested
by Yu and Moyeed (2001) from a Bayesian perspective. They also prove that, under
improper prior for the regression parameters β τ , the Baysian Maximum a Posteriori
estimate (MaP) coincides with the solution of the minimisation problem in Eq. (24).
Examples of specifications of the CAViaR dynamic in Eq. (23) have been provided in
the seminal paper of Engle and Manganelli (2004):
(i) Symmetric Absolute Value (SAV):

f t (β) = β1 + β2 f t−1 (β) + β3 |yt−1 |, (25)

(ii) Asymmetric Slope (AS):

f t (β) = β1 +β2 f t−1 (β)+β3 yt−1 1[0,+∞) (yt−1 )+β4 yt−1 1(−∞,0) (yt−1 ) , (26)

(iii) Indirect GARCH(1,1) (IGARCH):

$ %1
2
f t (β) = β1 + β2 f t−1
2
(β) + β3 yt−1
2
, (27)

123
 M. Bernardi, L. Catania

(iv) Adaptive (AD):

β1
f t (β) = f t−1 (β) + , (28)
1 + exp {G (yt−1 − f t−1 (β))}

where G ∈ + is a positive constant.

The different specifications introduced by the original paper of Engle and Manganelli
(2004) can be estimated using the code available at the first author’s web page: http://
www.simonemanganelli.org/Simone/Research.html.

4 Application

In this empirical study, a panel of four major worldwide stock markets indexes is con-
sidered. The four daily stock price indices includes the Asia/Pacific 600 (SXP1E), the
North America 600 (SXA1E), the Europe 600 (SXXP) and the Global 1800 (SXW1E).
The data are freely available and can be downloaded from the STOXX website http://
www.stoxx.com/indices/types/benchmark.html. The data were obtained over a 23-
years time period, from December 31, 1991 to July 24, 2014, comprising a total of
5874 observations. For each market, the returns are calculated as the logarithmic dif-
ference of the daily index level multiplied by 100

yt = (log ( pt ) − log ( pt−1 )) × 100,

where pt is the closing index value on day t. To examine the performance of the
models to predict VaR levels, the complete dataset of daily returns is divided into two
samples: an in-sample period from January 1, 1992 to October 6, 2006, for a total
of 3814 observations, and a forecast or validation period, containing the remaining
2000 observations: from October 9, 2006 to July 24, 2014. A fixed rolling window
approach is then used to produce 1-day ahead forecasts of the 5 % VaR thresholds,
t+1 , for t = 1, 2, . . . , 2000 of the considered series in the forecast sample. Table 1
VaR0.05
reports some descriptive statistics for the in sample as well as the out of sample period.
As expected, we found evidence of departure from normality, mainly because all the
series appear to be leptokurtic and skewed. Moreover, the Jarque and Bera (1980)
test statistic strongly rejects the null hypothesis of normality for all the considered
series. It is interesting to note that, the departure from normality, is stronger for the
out of sample returns. As widely discussed by Shiller (2012), this empirical evidence
can be considered as an effect of the recent GFC of 2007-2008 that affected the entire
world’s financial markets. Furthermore, the unconditional distribution of each return’s
series, in the out of sample period, is negatively skewed and shows higher standard
deviation and kurtosis. The 5 % unconditional quantile, which represents the VaR at
τ = 5 % under the iid assumption of the returns’ conditional distribution, has been
moved further to the left tail, in the second part of the sample. Given the changes in the
indexes returns’ behaviour we would expect the final SSM contains more sophisticated
models being able to accurately describe the conditional distributions during turbulent
market periods.

123
Comparison of Value-at-Risk models using the MCS approach

Table 1 Summary statistics of the panel of international indexes, for both the in sample and the out of
sample period

Index Min Max Mean SD Skewness Kurtosis 5 % Str. Lev. JB

In-sample, from 02/01/1992 to 06/10/2006

SXA1E −8.05 7.79 0.03 1.28 −0.07 5.82 −2.08 1269.17
SXP1E −5.80 9.71 0.01 1.29 0.10 5.90 −1.99 1341.19
SXW1E −5.54 5.02 0.03 0.99 −0.07 5.57 −1.60 1055.46
SXXP −6.41 5.64 0.03 1.05 −0.27 6.82 −1.68 2376.35
Index Min Max Mean SD Skewness Kurtosis 1 % Str. Lev. JB

Out-of-sample, from 09/10/2006 to 24/06/2014

SXA1E −9.18 9.96 0.02 1.38 −0.28 11.32 −2.13 5812.50
SXP1E −7.73 9.42 0.00 1.28 −0.33 8.72 −2.02 2768.92
SXW1E −6.81 8.37 0.01 1.05 −0.28 10.43 −1.62 4641.35
SXXP −7.93 9.41 0.00 1.33 −0.11 9.61 −2.06 3655.56

The seventh column, denoted by “5 % Str. Lev.” is the 5 % empirical quantile of the returns distribution,
while the eight column, denoted by “JB” is the value of the Jarque-Berá test-statistics

As previously said, our empirical application focuses on the ability to forecast the
Value-at-Risk delivered by several competing models. To apply the MCS procedure,
we forecast the VaR at τ = 5 % using the models described in Sect. 3 estimated on
each of the four indexes. More precisely, we consider eight different GARCH(1,1)
specifications (GARCH, EGARCH, APARCH, AVARCH, GJRGARCH, TGARCH,
NGARCH, CGARCH) with Gaussian and Student-t innovations, the GAS-N and the
GAS-T models, and two CaViaR model specifications (SAV, AS), comprising a total
of 20 models. Estimated coefficients for each model, over the in sample period, are not
reported to save space, but they are available upon request to the second author. For
the GARCH and the GAS models, VaR estimates are performed by inverting the cor-
responding conditional cumulative density function, while the CAViaR specification
reports directly the quantile estimates. The MCS procedure of Hansen et al. (2011) is
then applied to obtain the set of models with superior predictive ability in term of the
supplied VaR forecasts. To this end, we developed the R package MCS freely available
at the CRAN repository. The implementation of the MCS procedure has been car-
ried out using the MCSprocedure() rountine, and MCS p-values were evaluated
using 8000 bootstrap replications. Model performances are assessed according to the
asymmetric quantile loss function of González-Rivera et al. (2004) defined in Eq. (21),
which heavily penalises observations far away from the selected empirical quantile.
Clearly, the higher the number of eliminated models, the higher the heterogeneity of
the competing forecasts. On the contrary, if the final SSM contains a big portion of the
starting set M0 , then the competing model are statistically equivalent in term of their
forecast ability of future VaR levels. The p-values of the TR,M and Tmax,M statistics, are
reported in the second and sixth columns, respectively. The overall p-values of the test
statistics, are equal to the minimum of the p-values reported Table 2, respectively. For
a detailed discussion about the interpretation of the MCS p-values we refer to Hansen

123
 Table 2 Comparison of the SSMs for the four considered international stock indexes over the whole out of sample period from October 9, 2006 to June 24, 2014

123
Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

SXA1E: 4 eliminations
GJRGARCH-N 1 −1.73 1.00 1 −0.30 1.00 137.81
GJRGARCH-T 2 −1.42 1.00 2 0.30 1.00 138.16
APARCH-N 3 −1.07 1.00 3 0.59 1.00 138.50
APARCH-T 4 −0.61 1.00 4 0.91 0.98 138.89
AVGARCH-N 5 −0.54 1.00 6 0.96 0.98 138.94
GARCH-N 6 −0.52 1.00 5 0.96 0.98 138.95
NGARCH-N 7 −0.40 1.00 7 1.03 0.96 139.05
GARCH-T 8 0.03 1.00 8 1.28 0.84 139.39
TGARCH-N 9 0.26 1.00 9 1.41 0.72 139.57
NGARCH-T 10 0.34 1.00 10 1.45 0.67 139.63
TGARCH-T 11 0.57 0.99 11 1.58 0.49 139.82
EGARCH-T 12 0.78 0.96 12 1.68 0.32 140.00
EGARCH-N 13 0.80 0.96 13 1.69 0.30 140.02
AVGARCH-T 14 1.16 0.73 14 1.88 0.09 140.32
GAS-N 15 1.24 0.63 15 1.91 0.07 140.42
CGARCH-N 16 1.25 0.61 16 1.92 0.06 140.43
SXP1E: 2 eliminations
GJRGARCH-N 1 −1.87 1.00 1 −0.47 1.00 134.16
APARCH-N 2 −1.43 1.00 2 0.46 1.00 134.37
GJRGARCH-T 3 −1.05 1.00 3 0.74 1.00 134.52
M. Bernardi, L. Catania
 Table 2 continued

Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

APARCH-T 4 −0.81 1.00 4 0.89 1.00 134.60

CGARCH-N 5 −0.36 1.00 5 1.18 0.96 134.75
GARCH-N 6 −0.32 1.00 7 1.25 0.94 134.77
TGARCH-N 7 −0.17 1.00 8 1.29 0.92 134.82
AVGARCH-N 8 −0.17 1.00 6 1.18 0.96 134.80
EGARCH-N 9 0.11 1.00 10 1.44 0.83 134.91
TGARCH-T 10 0.14 1.00 9 1.40 0.85 134.92
GAS-N 11 0.15 1.00 11 1.46 0.81 134.93
NGARCH-N 12 0.36 1.00 13 1.62 0.67 134.99
GARCH-T 13 0.48 1.00 12 1.60 0.70 135.04
EGARCH-T 14 0.60 1.00 14 1.64 0.64 135.08
NGARCH-T 15 0.91 0.96 16 1.79 0.45 135.19
CGARCH-T 16 0.93 0.95 15 1.78 0.49 135.21
Comparison of Value-at-Risk models using the MCS approach

AVGARCH-T 17 0.94 0.94 17 1.86 0.39 135.24

GAS-T 18 1.38 0.58 18 1.97 0.28 135.42
SXW1E: 10 eliminations
GJRGARCH-N 1 −1.27 1.00 1 −0.08 1.00 105.09
APARCH-N 2 −1.20 1.00 2 0.08 1.00 105.14
GJRGARCH-T 3 −1.07 1.00 3 0.17 1.00 105.20
APARCH-T 4 −0.75 1.00 4 0.42 1.00 105.35
AVGARCH-N 5 0.16 1.00 5 0.98 0.79 105.73
TGARCH-N 6 0.63 0.89 6 1.25 0.48 105.92

123
 Table 2 continued

Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

123
TGARCH-T 7 0.65 0.88 7 1.25 0.48 105.93
EGARCH-T 8 0.92 0.59 8 1.42 0.26 106.04
EGARCH-N 9 1.01 0.47 9 1.47 0.20 106.09
AVGARCH-T 10 1.06 0.40 10 1.49 0.18 106.11
SXXP: 13 eliminations
TGARCH-N 1 −1.68 1.00 1 −0.76 1.00 134.12
APARCH-N 2 −0.63 1.00 2 0.76 1.00 134.34
AVGARCH-N 3 −0.09 1.00 3 1.02 0.99 134.44
EGARCH-N 4 −0.01 1.00 4 1.12 0.96 134.46
TGARCH-T 5 0.21 1.00 5 1.27 0.85 134.50
EGARCH-T 6 1.02 0.94 6 1.70 0.11 134.67
APARCH-T 7 1.20 0.67 7 1.81 0.04 134.70

The p-values of the TR,M and Tmax,M statistics, are reported in the third and seventh columns, respectively. The overall p-value of the test statistics, are equal to the minimum
of the p-values. The columns RankR,M and Rankmax,M report the ranking over the models belonging to the SSMs. Finally, the last column Loss × 103 is the average loss
across the considered period
M. Bernardi, L. Catania
Comparison of Value-at-Risk models using the MCS approach

et al. (2011). The estimated SSMs differ for the number of the eliminated models
as well as for their compositions. We can observe that, for the SXP1E index, only 2
models were eliminated by the MCS procedure. This empirical finding highlights the
statistical equivalence of forecasting future VaR levels using a simple models such as
the GARCH(1,1)-N or more sophisticated ones like the GAS-T . Furthermore, this
evidence suggests that, the SXP1E index may not be affected by some stylised facts
such as the leverage effect or by complex nonlinear conditional volatility dynamics. For
the SXA1E and SXW1E indexes the MCS procedure eliminates four and ten models,
respectively. An higher level of discrimination among models is instead evident for
the European index (SXXP). In fact, in that case, 13 of the 20 considered models do
not belong to the final SSM and the seven remaining models are those characterised by
strongest nonlinear dynamics for the conditional volatility process. At a first glance,
it would seem strange to observe an homogeneous composition of the final SSM with
respect to the conditional distribution assumption. Indeed, our results suggest that all
series appear to be accurately described by both a Gaussian or a Student-t distribution.
However, as discussed by Jondeau et al. (2007), it should be noted that the Gaussian
assumption for the innovations does not implies Gaussianity for the unconditional
distribution of the returns. Concerning the empirical relevance of the distribution
assumption, it is worth noting that the considered return series can be viewed as highly
diversified portfolios. Well diversified portfolios are characterised by the fact that
positive and negative tail events affecting the conditional distribution and its kurtosis,
are mitigated by the diversification. Finally, concerning the CAViaR specifications,
we note that they are always excluded from the SSM, suggesting the those dynamics
do not adequately describe future VaR levels for the considered indexes.
During the evaluation period we have considered so far, the equity market expe-
riences tranquil as well as more turbulent phases, and in particular, the GFC of
2007–2008 and the European Debt Crises of 2010–2011. To further investigate the
VaR forecasting performance of the considered models, we also perform the MCS
analysis for each series, conditional of being either in a tranquil or in a turbulent
phase of the financial cycle. To this end, we classify the period starting on October 9,
2006 and ending on October 2, 2010 as the “Turbulent” period, and the subsequent
period starting on October 3, 2010 and ending on July 24, 2014 as the “Tranquil”
period. The periods classification is the same for all the considered series except for
the SXXP, where it is reversed in order to account for the European Debt Crises. To
keep results comparable, both subperiods were chosen of the same length, comprising
1000 observations. Tables 3 and 4 report the final SSM composition for the “Tran-
quil” and “Turbulent” periods, respectively. Comparing the two tables there is little
evidence that the final composition of the SSM is influenced by the actual conditions
of the financial markets. The only exception concerns the European index (SXXP)
where the final SSM is substantially smaller during the “Turbulent” period.
In order to test the benefits of the MCS procedure, we also apply the Dynamic
Quantile Averaging (DQA) technique proposed by Bernardi et al. (2015) on the initial
DQA
set of models M0 (VaRALL ) and on the subset of models that that belong to final
SSM, M̂1−α∗ DQA
(VaRMCS ). The DQA procedure averages VaR forecasts using a dynamic
updating scheme based on each model’s relative contribution to the total VaR loss.

123
 Table 3 Comparison of the SSMs for the four considered international stock indexes over the “Tranquil” subsample

123
Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

SXA1E: 9 eliminations
CGARCH-T 1 −1.24 1.00 1 −0.28 1.00 104.83
CGARCH-N 2 −0.92 1.00 2 0.28 1.00 104.88
GJRGARCH-T 3 −0.82 1.00 3 0.30 1.00 104.89
NGARCH-N 4 −0.72 1.00 4 0.43 1.00 104.92
GJRGARCH-N 5 −0.22 1.00 5 0.64 1.00 104.98
NGARCH-T 6 −0.19 1.00 7 0.79 1.00 104.99
APARCH-N 7 −0.16 1.00 6 0.68 1.00 104.99
GARCH-N 8 0.46 1.00 8 1.18 1.00 105.07
GAS-N 9 1.06 0.71 9 1.56 0.88 105.15
APARCH-T 10 1.32 0.38 10 1.65 0.79 105.20
GARCH-T 11 1.46 0.24 11 1.94 0.39 105.23
SXP1E: 2 eliminations
GAS-T 1 −2.25 1.00 1 −0.04 1.00 112.21
GAS-N 2 −2.05 1.00 2 0.04 1.00 112.23
EGARCH-N 3 −0.44 1.00 3 1.32 1.00 112.89
EGARCH-T 4 −0.35 1.00 4 1.37 1.00 112.92
AVGARCH-T 5 −0.22 1.00 5 1.45 0.99 112.96
CGARCH-N 6 −0.18 1.00 6 1.48 0.99 112.98
APARCH-N 7 −0.08 1.00 7 1.54 0.99 113.01
M. Bernardi, L. Catania
 Table 3 continued

Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

APARCH-T 8 −0.04 1.00 8 1.57 0.98 113.02

GJRGARCH-N 9 0.07 1.00 9 1.65 0.98 113.06
TGARCH-N 10 0.14 1.00 10 1.68 0.97 113.09
TGARCH-T 11 0.21 1.00 12 1.73 0.96 113.11
GJRGARCH-T 12 0.21 1.00 11 1.72 0.96 113.11
CGARCH-T 13 0.41 1.00 13 1.86 0.91 113.18
AVGARCH-N 14 0.58 1.00 14 1.96 0.84 113.24
GARCH-N 15 0.77 1.00 15 2.08 0.74 113.31
GARCH-T 16 0.95 1.00 16 2.18 0.60 113.37
NGARCH-N 17 1.21 0.99 17 2.34 0.35 113.46
NGARCH-T 18 1.47 0.96 18 2.49 0.15 113.55
SXW1E: 10 eliminations
GAS-N 1 −1.65 1.00 1 −0.55 1.00 84.62
Comparison of Value-at-Risk models using the MCS approach

GJRGARCH-T 2 −0.96 1.00 2 0.55 1.00 84.73

TGARCH-T 3 −0.87 1.00 3 0.62 1.00 84.75
APARCH-T 4 −0.62 1.00 4 0.80 1.00 84.78
EGARCH-T 5 0.27 1.00 5 1.33 0.95 84.90
AVGARCH-N 6 0.49 1.00 6 1.46 0.85 84.93

123
 Table 3 continued

Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

123
TGARCH-N 7 0.52 1.00 7 1.49 0.82 84.93
GJRGARCH-N 8 0.61 1.00 8 1.55 0.76 84.95
APARCH-N 9 1.03 0.92 9 1.81 0.46 85.00
EGARCH-N 10 1.35 0.61 10 1.98 0.26 85.04
SXXP: 11 eliminations
TGARCH-N 1 −1.55 1.00 1 −0.48 1.00 162.60
APARCH-N 2 −0.86 1.00 2 0.48 1.00 162.87
AVGARCH-N 3 −0.85 1.00 4 0.49 1.00 162.88
EGARCH-N 4 −0.83 1.00 3 0.49 1.00 162.88
GJRGARCH-N 5 0.45 1.00 5 1.30 0.97 163.39
TGARCH-T 6 0.64 1.00 6 1.46 0.76 163.45
EGARCH-T 7 0.74 1.00 8 1.53 0.66 163.49
APARCH-T 8 0.74 1.00 7 1.53 0.66 163.49
CGARCH-N 9 1.38 0.41 9 1.89 0.23 163.79

The p-values of the TR,M and Tmax,M statistics, are reported in the third and seventh columns, respectively. The p-value of the test statistic, is equal to the minimum of the
overall p-values. The columns RankR,M and Rankmax,M report the ranking over the models belonging to the SSMs. Finally, the last column Loss × 103 is the average loss
across the considered period
M. Bernardi, L. Catania
 Table 4 Comparison of the SSMs for the four considered international stock indexes over the “Turbulent” subsample

Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

SXA1E: 5 eliminations
GJRGARCH-N 1 −1.80 1.00 1 −0.15 1.00 170.51
APARCH-N 2 −1.60 1.00 2 0.15 1.00 170.80
GJRGARCH-T 3 −1.21 1.00 3 0.43 1.00 171.32
APARCH-T 4 −0.85 1.00 4 0.69 1.00 171.81
TGARCH-N 5 −0.32 1.00 5 1.04 1.00 172.49
NGARCH-N 6 −0.22 1.00 6 1.11 1.00 172.62
GARCH-N 7 −0.18 1.00 7 1.14 1.00 172.67
EGARCH-N 8 0.31 1.00 8 1.45 1.00 173.30
AVGARCH-N 9 0.41 1.00 9 1.52 1.00 173.42
TGARCH-T 10 0.48 1.00 10 1.56 1.00 173.52
NGARCH-T 11 0.62 1.00 11 1.65 0.98 173.70
Comparison of Value-at-Risk models using the MCS approach

GARCH-T 12 0.66 1.00 12 1.67 0.98 173.75

EGARCH-T 13 0.94 1.00 13 1.86 0.80 174.12
AVGARCH-T 14 1.20 0.98 14 2.01 0.39 174.46
GAS-N 15 1.58 0.52 15 2.25 0.01 174.98
SXP1E: 3 eliminations
GJRGARCH-N 1 −2.18 1.00 1 −0.68 1.00 155.10
GJRGARCH-T 2 −1.28 1.00 2 0.68 1.00 155.74
APARCH-N 3 −1.22 1.00 3 0.72 1.00 155.78
APARCH-T 4 −0.56 1.00 4 1.16 1.00 156.21

123
 Table 4 continued

Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

123
TGARCH-N 5 −0.48 1.00 5 1.22 1.00 156.26
AVGARCH-N 6 −0.45 1.00 6 1.22 1.00 156.27
GARCH-N 7 −0.39 1.00 7 1.29 1.00 156.32
CGARCH-N 8 −0.15 1.00 8 1.44 1.00 156.47
TGARCH-T 9 0.10 1.00 9 1.57 1.00 156.63
EGARCH-N 10 0.13 1.00 10 1.61 1.00 156.65
NGARCH-N 11 0.44 1.00 11 1.8 1.00 156.85
GARCH-T 12 0.58 1.00 12 1.88 1.00 156.95
CGARCH-T 13 0.75 1.00 13 1.98 0.98 157.06
EGARCH-T 14 0.86 1.00 14 2.06 0.95 157.13
AVGARCH-T 15 0.99 1.00 15 2.14 0.84 157.23
NGARCH-T 16 1.16 1.00 16 2.23 0.68 157.34
GAS-N 17 1.66 0.88 17 2.59 0.01 157.68
SXW1E: 8 eliminations
GJRGARCH-N 1 −1.51 1.00 1 −0.04 1.00 125.18
APARCH-N 2 −1.46 1.00 2 0.04 1.00 125.23
GJRGARCH-T 3 −0.94 1.00 3 0.41 1.00 125.70
APARCH-T 4 −0.84 1.00 4 0.47 1.00 125.79
AVGARCH-T 5 −0.08 1.00 5 0.98 1.00 126.45
TGARCH-N 6 −0.05 1.00 6 1.00 1.00 126.48
AVGARCH-N 7 0.08 1.00 7 1.08 1.00 126.59
EGARCH-N 8 0.22 1.00 8 1.17 1.00 126.71
TGARCH-T 9 0.76 1.00 9 1.52 0.99 127.19
EGARCH-T 10 1.06 0.98 10 1.71 0.55 127.46
M. Bernardi, L. Catania
 Table 4 continued

Model RankR,M ti j p-valueR,M Rankmax,M ti· p-valuemax,M Loss × 103

GARCH-N 11 1.31 0.57 11 1.90 0.03 127.70

NGARCH-N 12 1.41 0.22 12 1.96 0.00 127.79
SXXP: 17 eliminations
TGARCH-N 1 −0.90 1.00 1 −0.27 1.00 107.33
APARCH-N 2 −0.31 1.00 2 0.27 0.99 107.35
AVGARCH-N 3 1.04 0.33 3 1.17 0.41 107.40

The p-values of the TR,M and Tmax,M statistics, are reported in the third and seventh columns, respectively. The p-value of the test statistic, is equal to the minimum of the
overall p-values. The columns RankR,M and Rankmax,M report the ranking over the models belonging to the SSMs. Finally, the last column Loss × 103 is the average loss
across the considered period
Comparison of Value-at-Risk models using the MCS approach

123
 M. Bernardi, L. Catania

Table 5 VaR backtesting measures of the dynamic VaR combination considering all the available models
DQA DQA
(VaRALL ) and only the models belonging to the SSM (VaRMCS )

DQA DQA
Asset VaRALL VaRMCS

AE ADmean ADmax AE ADmean ADmax

SXA1E 1.43 0.734 4.519 1.17 0.711 4.344

SXP1E 1.28 0.692 5.103 1.06 0.656 5.043
SXW1E 1.35 0.570 3.053 1.16 0.533 2.970
SXXP 1.55 0.687 4.093 1.32 0.671 4.080

More specifically, the DQA method aggregates the information coming from the indi-
vidual VaR estimates VaRτj,t|t−1 , j = 1, 2, . . . , m ∗ with the following convex linear
combination
m∗

τ,DQA τ
VaRt|t−1 = j,t|t−1 VaR j,t|t−1 , (29)
j=1

where the set of model specific weights j,t|t−1 , for j = 1, 2, . . . , m ∗ follow the
exponential smoothing moving average process
   
t+1, j = κj t, j + 1 − κ j π̃ rt , VaRτj,t|t−1 , σ̂ j,t , (30)
 
with κ j ∈ (0, 1), and π̃ rt , VaRτj,t|t−1 , σ̂ j,t is the exponential of the τ -quantile loss
kernel defined in equation normalised over all the possible models belonging to the
∗ , i.e.
final set M̂1−α
&   '
  exp  rt , VaRτj,t|t−1 /σ̂ j,t
π̃ rt , VaRτj,t|t−1 , σ̂ j,t =  ∗ &   ', (31)
m τ
j exp  rt , VaR j,t|t−1 /σ̂ j,t

where σ̂ j,t , is the predicted conditional variance at time t of model j = 1, 2, . . . , m ∗ .

For further details about the DQA here considered, we refer to Bernardi et al. (2015),
where the optimality properties of the weighting scheme are discussed. Table 5 reports
the backtesting performances of the VaR aggregation schemes. Three VaR backtesting
measures are considered. The first is the Actual over Expected ratio AE, defined as
the ratio between the realised VaR exceedances over a given time horizon and their
“a priori” expected values; VaR forecasts series for which the AE ratio is closer to
the unity are preferred. The second and the third backtesting measures are the mean
and maximum Absolute Deviation (ADmean and ADmax) of VaR violating returns
described in McAleer and da Veiga (2008). The AD in general provides a measure
of the expected loss given a VaR violation; of course models with lower mean and/or
maximum ADs are preferred. As showed in Table 5, the DQA aggregation scheme
∗
based on the final set M̂1−α
DQA
(VaRMCS ) always outperforms that based on M0 (VaRALL )
DQA

is terms of ADmean and ADmax measures. Moreover, even the AE ratio is strongly

123
Comparison of Value-at-Risk models using the MCS approach

improved for all the considered indexes when the dynamic average scheme considers
only the final SSM constituents.

5 Conclusion

In this paper we compare alternative model specifications in term of their VaR fore-
casting performances. The model comparison is performed using the MCS procedure
recently proposed by Hansen et al. (2011). The MCS technique is particularly useful
when several different models are available and it is not obvious which one performs
better. The MCS sequence of tests delivers the Superior Set of Models having Equal
Predictive Ability in terms of an user supplied loss function discriminating models.
This flexibility helps to discriminate models with respect to desired characteristics,
such as, for example, their forecasting performances. In our empirical application,
we compare the VaR forecast ability of several models also considering different
phases of the financial cycle. More specifically, the direct quantile estimates obtained
by the dynamic CAViaR models of Engle and Manganelli (2004) are compared with
the VaR forecast delivered by several ARCH-type models of Engle (1982) and with
those obtained by two different specifications of the Generalised Autoregressive Score
(GAS) models of Creal et al. (2013) and Harvey (2013). The MCS procedure is firstly
performed to reduce the initial number of models, and then to show that aggregat-
ing VaR forecasts using the Dynamic Quantile Averaging (DQA) technique proposed
by Bernardi et al. (2015) to the final set of superior models improves the predictive
performances of the individual VaR forecasts.
The ability of the MCS procedure has been empirically investigated on an extensive
out of sample analysis of the 5 % VaR forecast based on a panel of daily returns of
four major worldwide stock markets indexes: the Asia/Pacific 600 (SXP1E), the North
America 600 (SXA1E), the Europe 600 (SXXP) and the Global 1800 (SXW1E). Model
comparisons have been performed using R package MCS developed by the authors
and available on the CRAN repository, http://cran.r-project.org/web/packages/MCS/
index.html. The MCS package is very flexible since it allows for the specification of
the model’s types and loss functions that can be supplied by the user. This freedom
allows for the user to concentrate on substantive issues, such as the construction of
the initial set of model’s specifications M0 , without being limited by the the software
constrains. Our empirical results, suggest that, during the European Sovereign Debt
crisis of 2009–2010, highly non-linear volatility models are preferred by the MCS
procedure for the European countries. On the contrary, quite homogenous results,
with respect to the models’ complexity, were found for the the North America and
Asia Pacific regions.

Acknowledgments This research is supported by the Italian Ministry of Research PRIN 2013–2015,
“Multivariate Statistical Methods for Risk Assessment” (MISURA), and by the “Carlo Giannini Research
Fellowship”, the “Centro Interuniversitario di Econometria” (CIdE) and “UniCredit Foundation”. In the
development of package MCS we have benefited from the suggestions and help of several users. Our sincere
thanks go to all the developers of R since without their continued effort and support no contributed package
would exist. The authors would like to thank the anonymous reviewers and the Associated Editor for their
helpful and constructive comments that greatly contributed to improving the final version of the paper.

123
 M. Bernardi, L. Catania

Compliance with ethical standards

Funding This study was partially funded by the Italian Ministry of Research PRIN 2013–2015, “Multi-
variate Statistical Methods for Risk Assessment” (MISURA).

Conflict of interest The authors declare that they have no conflict of interest.

Patients’ rights and animal protection statements This article does not contain any studies with human
participants or animals performed by any of the authors.

References
Andres P (2014) Maximum likelihood estimates for positive valued dynamic score models; the dysco pack-
age. Comput Stat Data Anal 76(0):34–42 CFEnetwork: The Annals of Computational and Financial
Econometrics 2nd Issue
Bates JM, Granger CW (1969) The combination of forecasts. Oper Res Q 20(4):451–468
Bauwens L, Laurent S, Rombouts JVK (2006) Multivariate garch models: a survey. J Appl Econom
21(1):79–109
Bernardi M, Catania L (2015) Switching-GAS copula models for systemic risk assessment.
arXiv:1504.03733v3
Bernardi M, Catania L, Petrella L (2015) Are News Important to Predict the Value-at-Risk? Eur J Finance.
doi:10.1080/1351847X.2015.1106959
Black F (1976) Studies of stock price volatility changes. Proceedings of the 1976 American Statistical
Association. Business and Economical Statistics Section. American Statistical Association, Alexan-
dria, VA, pp 177–181
Bollerslev T (1986) Generalized autoregressive conditional heteroskedasticity. J Econom 31(3):307–327
Bollerslev T (2008) Glossary to ARCH (GARCH). CREATES Research Papers 2008–49, School of Eco-
nomics and Management, University of Aarhus
Box GE, Cox DR (1964) An analysis of transformations. J R Stat Soc Ser B 26:211–252
Caporin M, McAleer M (2014) Robust ranking of multivariate garch models by problem dimension. Comput
Stat Data Anal 76(0):172–185 CFEnetwork: The Annals of Computational and Financial Econometrics
2nd Issue
Chen Q, Gerlach R, Lu Z (2012) Bayesian Value-at-Risk and expected shortfall forecasting via the asym-
metric Laplace distribution. Comput Stat Data Anal 56(11):3498–3516
Chen Q, Gerlach RH (2013) The two-sided weibull distribution and forecasting financial tail risk. Int J
Forecast 29(4):527–540
Clark TE, McCracken MW (2001) Tests of equal forecast accuracy and encompassing for nested models.
J Econom 105(1):85–110
Cox DR, Gudmundsson G, Lindgren G, Bondesson L, Harsaae E, Laake P, Juselius K, Lauritzen SL (1981)
Statistical analysis of time series: Some recent developments [with discussion and reply]. Scand J Stat
8:93–115
Creal D, Koopman SJ, Lucas A (2013) Generalized autoregressive score models with applications. J Appl
Econom 28(5):777–795
Delle Monache D, Petrella I (2014) Adaptive models and heavy tails. Technical report, Queen Mary,
University of London, School of Economics and Finance
Diebold FX, Mariano RS (2002) Comparing predictive accuracy. J Bus Econ Stat 20(1):134–144
Ding Z, Granger CW, Engle RF (1993) A long memory property of stock market returns and a new model.
J Empir Finance 1(1):83–106
Engle RF (1982) Autoregressive conditional heteroscedasticity with estimates of the variance of united
kingdom inflation. Econometrica 50(4):987–1007
Engle R (2002) New frontiers for arch models. J Appl Econom 17(5):425–446
Engle RF, Gallo GM (2006) A multiple indicators model for volatility using intra-daily data. J Econom
131(1):3–27
Engle RF, Lee GG (1993) A permanent and transitory component model of stock return volatility. University
of California at San Diego, Economics Working Paper Series
Engle RF, Manganelli S (2004) CAViaR: conditional autoregressive value at risk by regression quantiles. J
Bus Econ Stat 22(4):367–381

123
Comparison of Value-at-Risk models using the MCS approach

Engle RF, Ng VK (1993) Measuring and testing the impact of news on volatility. J Finance 48(5):1749–1778
Engle RF, Russell JR (1998) Autoregressive conditional duration: a new model for irregularly spaced
transaction data. Econometrica 66:1127–1162
Francq C, Zakoian J-M (2011) GARCH models: structure, statistical inference and financial applications.
Wiley, New York
Gallant A, Hsieh D, Tauchen G (1997) Estimation of stochastic volatility models with diagnostics. J Econom
81(1):159–192
Giacomini R, White H (2006) Tests of conditional predictive ability. Econometrica 74(6):1545–1578
Glosten LR, Jagannathan R, Runkle DE (1993) On the relation between the expected value and the volatility
of the nominal excess return on stocks. J Finance 48(5):1779–1801
Gneiting T (2011) Making and evaluating point forecasts. J Am Stat Assoc 106(494):746–762
González-Rivera G, Lee T-H, Mishra S (2004) Forecasting volatility: a reality check based on option pricing,
utility function, value-at-risk, and predictive likelihood. Int J Forecast 20(4):629–645
Hansen PR (2005) A test for superior predictive ability. J Bus Econ Stat 23(4):365–380
Hansen PR, Lunde A (2005) A forecast comparison of volatility models: Does anything beat a garch(1,1)?
J Appl Econom 20(7):873–889
Hansen PR, Lunde A, Nason JM (2003) Choosing the best volatility models: the model confidence set
approach. Oxf Bull Econ Stat 65(s1):839–861
Hansen PR, Lunde A, Nason JM (2011) The model confidence set. Econometrica 79(2):453–497
Harvey A, Sucarrat G (2014) EGARCH models with fat tails, skewness and leverage. Comput Stat Data
Anal 76:320–338 CFEnetwork: The Annals of Computational and Financial Econometrics
Harvey AC (2013) Dynamic models for volatility and heavy tails: with applications to financial and economic
time series. Cambridge University Press, Cambridge
Harvey AC, Shephard N (1996) Estimation of an asymmetric stochastic volatility model for asset returns.
J Bus Econ Stat 14(4):429–434
Higgins ML, Bera AK (1992) A class of nonlinear arch models. Int Econ Rev 33(1):137–158
Jarque CM, Bera AK (1980) Efficient tests for normality, homoscedasticity and serial independence of
regression residuals. Econ Lett 6(3):255–259
Jondeau E, Poon S-H, Rockinger M (2007) Financial modeling under non-Gaussian distributions. Springer,
Berlin
Kilian L (1999) Exchange rates and monetary fundamentals: What do we learn from long-horizon regres-
sions? J Appl Econom 14(5):491–510
Koenker R, Bassett G (1978) Regression quantiles. Econometrica 46(1):33–50
Koenker R, Xiao Z (2006) Quantile autoregression. J Am Stat Assoc 101(475):980–990
Koopman SJ, Lucas A, Scharth M (2015) Predicting time-varying parameters with parameter-driven and
observation-driven models. Rev Econom Stat. doi:10.1162/REST_a_00533
Kowalski T, Shachmurove Y (2014) The reaction of the u.s. and the european monetary union to recent
global financial crises. Global Finance J 25(1):27–47
Lucas A, Zhang X (2014) Score driven exponentially weighted moving averages and Value-at-Risk fore-
casting. Tinbergen Institute Discussion paper, n. 14–092
McAleer M, da Veiga B (2008) Single-index and portfolio models for forecasting Value-at-Risk thresholds.
J Forecast 27(3):217–235
Nelson DB (1991) Conditional heteroskedasticity in asset returns: a new approach. Econometrica 59(2):347–
370
R Development Core Team (2013) R: a language and environment for statistical computing. R Foundation
for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0
Romano JP, Wolf M (2005) Stepwise multiple testing as formalized data snooping. Econometrica
73(4):1237–1282
Russell JR (1999) Econometric modeling of multivariate irregularly-spaced high-frequency data. GSB,
University of Chicago, Manuscript
Salvatierra IDL, Patton AJ (2014) Dynamic copula models and high frequency data. J Empir Finance
30:120–135
Samuels JD, Sekkel RM (2011) Forecasting with large datasets: trimming predictors and forecast combi-
nation. Working paper
Samuels JD, Sekkel RM (2013) Forecasting with many models: Model confidence sets and forecast com-
bination. Technical report, Bank of Canada Working paper
Schwert G (1990) Stock volatility and the crash of ’87. Rev Financ Stud 3(1):77–102

123
 M. Bernardi, L. Catania

Shiller RJ (2012) The subprime solution: How today’s global financial crisis happened, and what to do
about it. Princeton University Press, Princeton
Silvennoinen A, Teräsvirta T (2009) Multivariate garch models. In: Handbook of financial time series.
Springer, Berlin, pp 201–229
Taylor SJ (1986) Modelling financial times series. Wiley, New York
Taylor SJ (1994) Modeling stochastic volatility: a review and comparative study. Math Finance 4(2):183–
204
Teräsvirta T (2009) An introduction to univariate garch models. Handbook of Financial Time Series.
Springer, Berlin, pp 17–42
West KD (1996) Asymptotic inference about predictive ability. Econometrica 64(5):1067–1084
White H (2000) A reality check for data snooping. Econometrica 68(5):1097–1126
Yu K, Moyeed R (2001) Bayesian quantile regression. Stat Probab Lett 54:437–447
Zakoian J-M (1994) Threshold heteroskedastic models. J Econ Dyn Control 18(5):931–955

123