Specification Testing of Discrete Choice Models: A Note On The Use of A Nonparametric Test
Specification Testing of Discrete Choice Models: A Note On The Use of A Nonparametric Test
www.jocm.org.uk
Received 21 August 2007, revised version received 4 December 2007, accepted 7 January 2008
Abstract
Model misspecification is a serious issue since misspecification generally
renders statistical inference invalid. However, specification testing of dis-
crete choice models is rarely applied. This paper describes a nonparametric
test procedure which uses a combination of smoothed residual plots and a
test statistic able to detect general misspecification. Nonparametric meth-
ods require large datasets when the number of independent variables is more
than a few. A way to circumvent this problem is indicated, increasing the
usefulness of the approach also with limited datasets.
1 Introduction
It is standard practice in regression models to perform model control using the
residuals of the estimated model. Residuals are plotted to verify whether they
are in fact white noise unrelated to the independent variables. Residuals are
less easily defined in discrete choice models and similar model control is rarely
performed for such models.
In fact, a variety of specification tests are available in the literature for some
discrete choice models, but are not widely used. Lechner (1991) presents some
specification tests for the binary logit model. Gourieroux et al. (1987a) and
Gourieroux et al. (1987b) present tests based on generalised residuals for a range
of models including the multinomial logit model. McFadden (1987) presents
regression-based specification tests for the multinomial logit model similar in
nature to the test presented here. The seminal paper by McFadden and Train
∗
T: +45 4525 6521, F: +45 4593 6533, [email protected]
Licensed under a Creative Commons Attribution-Non-Commercial 2.0 UK: England & Wales License
http://creativecommons.org/licenses/by-nc/2.0/uk/
ISSN 1755-5345
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
(2000) provides specification tests of MNL and mixed logit against alternatives
with more mixing. Finally, software exists that allows the comparison of observed
choices to predictions when data are grouped on a categorical variable. This
approach may be viewed as a kind of residual test.1
The point of this paper is to describe how the nonparametric test of func-
tional form in Zheng (1996) may be applied to discrete choice models of general
form. This means that the test applies even to complicated models such as the
mixed generalised extreme value model. The Zheng test is based on nonparamet-
ric kernel regression of the parametric model residuals against the independent
variables. With residuals defined as the difference between choice 0-1 indicators
and predicted probabilities, the Zheng test applied to a discrete choice model is
based on the comparison of predicted and observed choices. Pagan and Ullah
(1999) review a range of nonparametric tests of functional form that are more or
less similar to the Zheng test. The procedure presented in this paper of applying
a test to a function of the independent variables is not restricted to the Zheng
test but may be applied with other tests as well. On the use of nonparametrics in
a discrete choice context and in transport applications see, e.g. Fosgerau (2006),
Fosgerau (2007) and Fosgerau and Bierlaire (2007). Pagan and Ullah (1999),
Yatchew (2003) and Härdle (1990) give general introductions to nonparametric
and semiparametric methods.2
A general problem in using nonparametric techniques is that the demands
on data increase exponentially in the dimension of the space of independent
variables, this is the so-called curse of dimensionality. This issue is addressed
in this paper by showing how such tests may be applied to a subset of variables
or more generally to functions of variables such as the index representing the
indirect utility of a choice alternative. It is thus possible to apply nonparametric
tests to a model, while regressing only on a low number of variables or even just
one. This reduces the demand on data and is of practical importance when the
model to be tested has several independent variables.
Furthermore, the test may be applied to variables that are not included in
the model. Thus the test may serve as a test of omitted variables.
The paper is organised as follows. Section 2 presents an exposition of the
Zheng test. Section 3 shows how such tests may be applied to low-dimensional
functions of the independent variables. Section 4 presents an example of appli-
cation of the test to a discrete choice model while section 5 concludes.
27
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
H0 : P [f (x : θ0 ) = g(x)] = 1. (1)
The null hypothesis thus states that f (· : θ0 ) coincides with g for almost all x.
The corresponding alternative hypothesis is that, for all θ ∈ Θ, f differs from g
on a set of probability greater than zero, or formally,
The alternative hypothesis is the negation of the null hypothesis. It thus encom-
passes all possible departures from the null.
Zheng (1996) defines a statistic Tn , where n is sample size, and shows that,
under the null hypothesis, Tn converges in distribution to a standard normal
as sample size increases, whereas it converges in probability to infinity under
the alternative. Therefore the test is consistent against all departures from the
parametric model f .
The idea of the test is the following. Define residuals by ε = y − f (x :
θ0 ). Then under the null hypothesis, E(ε|x) = 0 almost surely and hence also
E[εE(ε|x)p(x)] = 0. However, under the alternative hypothesis,
E[εE(ε|x)p(x)] = E[E(ε|x)2 p(x)]
= E[(g(x) − f (x : θ0 ))2 p(x)]
> 0. (3)
where the first equality follows from the law of iterated expectations.
Consider now an i.i.d. sample (yi , xi ). The test statistic is formed from a
sample analogue of E[εE(ε|x)p(x)] = 0, constructed using kernel regression and
kernel density estimation (see for example Pagan and Ullah, 1999). In order to
apply these methods
R we need a non-negative, bounded, continuous and symmetric
kernel K with K(u)du = 1 and a bandwidth h depending on the sample size
n. The application in section 4 uses a standard normal density for K. Then we
construct first an estimate of the density of x ∈ Rm as
1 X 1 xi − xj
p̂(xi ) = K (4)
n−1 hm h
j≤n,j6=i
28
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
The weighting ensures that local discrepancies between g and f are detected even
when positive and negative discrepancies would net out globally.
The estimator in eq. (5) is a nonparametric kernel regression of the residuals
ε against x. It is useful to use this regression to produce smoothed plots of
29
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
the residuals against the independent variables. Confidence bands around the
regression can be computed using that
Z
(nh)1/2 [Ê(εi |xi ) − E(εi |xi )] ∼ N 0, σ 2 (xi )p−1 (xi ) K 2 (u)du , (10)
where σ 2 is the variance of ε conditional on x.3 For the purpose of testing the
specification in the parametric model f we may estimate this variance using the
null hypothesis that E(y|x) = f (x|θ) a.e. In the case of a uni-dimensional x, we
may estimate σ(x) by
fˆ(x)(1 − fˆ(x))
Z
2
σ̂ (x) = K 2 (u)du, (11)
p̂(x)nh
The idea that the bandwidth h should depend on sample size may be unfamil-
iar. If the bandwidth is too large then averaging over too large neighbourhoods
will smooth away relevant differences and hence introduce bias. If the bandwidth
is too small then estimates will be noisy, they will be too much influenced by
random fluctuations in data and hence variance will be high. Choosing the band-
width appropriately as a function of sample size balances bias and variance and
ensures that the bandwidth tends to zero at an appropriately slow rate, such that
the asymptotical results obtain.
These considerations suggest that using a large bandwidth will tend to reduce
the Zheng statistic as positive differences between f and g in some regions of the
data will then be averaged with negative differences in other regions.
For a given sample size n, the bandwidth may be chosen by a rule such as, e.g.
1
h = n− 2m . This choice is easy and agrees with the conditions for the Zheng test.
It is also possible to select a bandwidth using cross-validation in the regression of
the residuals against x. This is however time consuming and the gain from doing
it is not clear.4 In applied research, it may be sufficient or even preferable to
select a bandwidth by just inspecting the resulting regression and the confidence
bands visually, this is so-called eye-balling, suggested by Pagan and Ullah when
the dimension of x is low enough to make this feasible.
3 Reducing dimensionality
A concern with the application of tests like the Zheng test, just as with any
nonparametric technique, is the curse of dimensionality. The size of the dataset
3
See Pagan and Ullah (1999).
4
See the discussion in Li and Racine (2007). Generally all that is required for the test to be
consistent is that bandwidths tend to zero as sample size tends to infinity but slowly enough
that n multiplied by the product of bandwidths tends to infinity. This is, however, not very
helpful in finite samples.
30
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
31
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
4 Examples
4.1 Mode choice
This section provides an illustrative application of the test to a mode choice
model. The model is not intended as a serious model. It merely serves the pur-
pose of showing how the Zheng test may be used to detect significant differences
between model and data. I use a mode choice data set from Sweden comprising
1799 observations of trips to Stockholm using either air, bus, car or train. For
each observation the data give travel time and cost for each alternative as well
as some background variables.
I estimated an MNL with alternative specific constants, mode-specific pa-
rameters for travel time and a common cost parameter. Write this model with
utilities Ui = Vi + εi , where εi are iid. extreme value and Vi = βxi . Then I com-
puted the predicted mode choice probabilities for the observations in the sample.
Residuals were computed for each mode by subtracting the predicted probability
from a dummy indicator for the observed choice. All the tests are carried out
using uni-dimensional kernel regressions and density estimates were carried out
using a normal density kernel and a bandwidth of n−1/2 , where n is sample size
and data (t(x)) are scaled to the unit interval. The choice of kernel is generally
not important. Judging from the graphs in the following, the bandwidth seems
appropriate. For computing confidence intervals around the kernel regression I
used equation (10).5 I trimmed away 1 percent of the sample, excluding the up-
5
The MNL was estimated in Biogeme (Bierlaire, 2003, 2005), which was also used to compute
the predicted choice probabilities. The Zheng test and the nonparametric regressions were
performed in Ox (Doornik, 2001). The Ox code for the test is given in the appendix.
32
Density_realincome 09:51:26 17-Jan-2008
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
2.75 3.00 3.25 3.50 3.75 4.00 4.25 4.50 4.75 5.00 5.25 5.50 5.75 6.00 6.25 6.50
per and lower 0.5 percent after sorting on the independent variable, in order to
exclude regions with very thin data and hence uncertain estimates.
33
P_Airrealincome 09:52:01 17-Jan-2008
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
P_Busrealincome 09:52:22 17-Jan-2008
-0.3
-0.4
2.75 3.00 3.25 3.50 3.75 4.00 4.25 4.50 4.75 5.00 5.25 5.50 5.75 6.00 6.25 6.50
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
-0.3
-0.4
2.75 3.00 3.25 3.50 3.75 4.00 4.25 4.50 4.75 5.00 5.25 5.50 5.75 6.00 6.25 6.50
34
P_TraV_Bus 10:57:58 26-Nov-2007
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
-0.3
-0.4
-3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5
not possible to conclude from the test which alternatives income should enter as
an income variable in one alternative would affect the predicted probabilities of
all alternatives. I tested a model formulation with separate income parameters
for air, car and train and found those for air and car to be confidently positive,
while that for train to be insignificant.
35
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
36
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
5 Concluding remarks
The issue of potential model misspecification should be taken very seriously, since
statistical inference from a misspecified model is generally not valid. This paper
has pointed out how an existing nonparametric test of functional form, the Zheng
test, may be applied to discrete choice models in order to detect general misspec-
ification.
The curse of dimensionality is a general problem with nonparametric methods
and limits their usefulness. The paper points out how dimensionality may be
reduced by applying the test not to the raw data but to functions of the data.
This opportunity is not unique to the Zheng test but may be used with other
similar test.
Reducing the dimensionality of the data by applying the test to a function of
the data makes it a lot easier to apply the test and data requirements are reduced.
The associated cost is that the result is conditional on the chosen function, since
any given function may not be able to reveal a given misspecification. So while
significant rejection of the model using a given function of the data is conclusive
evidence against the model, it is possible that some misspecification will remain
undetected. It is therefore advisable to perform the test using several functions
of the data.
Application of the test is not difficult but requires some programming. In
the appendix I have included the Ox function that I used to compute the Zheng
statistic. The test has also been included in Biogeme along with facilities to
produce graphs, making the test very easy to use within that environment.
Acknowledgments
Mogens Fosgerau has received financial support from the Danish Social Science
Research Council.
Tnumerator = 0;
37
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
Tdenominator = 0;
References
Bierlaire, M., 2003. BIOGEME: a free package for the estimation of discrete choice
models. Proceedings of the 3rd Swiss Transport Research Conference, Monte Verità,
Ascona, Switzerland.
Bierlaire, M., 2005. An introduction to Biogeme. biogeme.epfl.ch.
Doornik, J. A., 2001. Ox: An Object-Oriented Matrix Language. Timberlake Consultants
Press, London.
Ellison, G., Ellison, S. F., 2000. A simple framework for nonparametric specification
testing. Journal of Econometrics 96 (1), 1–23.
Fosgerau, M., 2006. Investigating the distribution of the value of travel time savings.
Transportation Research Part B: Methodological 40 (8), 688–707.
Fosgerau, M., 2007. Using nonparametrics to specify a model to measure the value of
travel time. Transportation Research Part A: Policy and Practice 41 (9), 842–856.
Fosgerau, M., Bierlaire, M., 2007. A practical test for the choice of mixing distribution
in discrete choice models. Transportation Research Part B: Methodological 41 (7),
784–794.
Gourieroux, C., Monfort, A., Renault, E., Trognon, A., 1987a. Generalised residuals.
Journal of Econometrics 34 (1-2), 5–32.
Gourieroux, C., Monfort, A., Renault, E., Trognon, A., 1987b. Simulated residuals.
Journal of Econometrics 34 (1-2), 201–252.
Härdle, W., 1990. Applied Nonparametric Regression. Vol. 19 of Econometric Society
Monograph Series. Cambridge University Press.
Lechner, M., 1991. Testing logit models in practice. Empirical Economics 16, 177–198.
Li, Q., Racine, J. S., 2007. Nonparametric Econometrics: Theory and Practice. Princeton
University Press, Princeton and Oxford.
McFadden, D., 1987. Regression-based specification tests for the multinomial logit model.
Journal of Econometrics 34 (1-2), 63–82.
38
Fosgerau, Journal of Choice Modelling, 1(1), 2008, 26-39
McFadden, D., Train, K., 2000. Mixed mnl models for discrete response. Journal of
Applied Econometrics 15, 447–470.
Pagan, A., Ullah, A., 1999. Nonparametric Econometrics. Cambridge University Press.
Yatchew, A., 2003. Semiparametric Regression for the Applied Econometrician. Themes
in modern econometrics. Cambridge University Press.
Zheng, J. X., 1996. A consistent test of functional form via nonparametric estimation
techniques. Journal of Econometrics 75 (2), 263–289.
39