92 PROC. OF THE 9th PYTHON IN SCIENCE CONF.

(SCIPY 2010)

Statsmodels: Econometric and Statistical Modeling

with Python
Skipper Seabold§∗ , Josef Perktold‡

Abstract—Statsmodels is a library for statistical and econometric analysis in The State of the Union: Open Source and Statistics
Python. This paper discusses the current relationship between statistics and
Python and open source more generally, outlining how the statsmodels package
fills a gap in this relationship. An overview of statsmodels is provided, including Currently R is the open source language of choice for applied
a discussion of the overarching design and philosophy, what can be found in the statistics. In applied econometrics, proprietary software pacakges
package, and some usage examples. The paper concludes with a look at what such as Gauss, MATLAB, Stata, SAS, and NLOGIT remain the
the future holds.
most popular and are taught in most graduate programs. However,
there is a growing call for the use of open source software in eco-
Index Terms—statistics, econometrics, R
nomic research due in large part to its reliability, transparency, and
the paradigm it offers for workflow and innovation [YaltaYalta],
Introduction [YaltaLucchetti]. In particular R is increasing in popularity as
Statsmodels (http://statsmodels.sourceforge.net/) is a library for evidenced by the recent textbooks by Cryer and Chan (2008),
statistical and econometric analysis in Python1 . Its intended au- Kleiber and Zeileis (2008), and Vinod (2008). Gretl is another
dience is both theoretical and applied statisticians and econo- notable open source alternative for econometrics [Gretl].
metricians as well as Python users and developers across dis- However, there are those who would like to see Python
ciplines who use statistical models. Users of R, Stata, SAS, become the language of choice for economic research and applied
SPSS, NLOGIT, GAUSS or MATLAB for statistics, financial econometrics. Choirat and Seri’s "Econometrics with Python" is
econometrics, or econometrics who would rather work in Python the first publication of which we are aware that openly advocates
for all its benefits may find statsmodels a useful addition to the use of Python as the language of choice for econometri-
their toolbox. This paper introduces statsmodels and is aimed cians [ChoiratSeri]. Bilina and Lawford express similar views
at the researcher who has some prior experience with Python, [BilinaLawford]. Further, John Stachurski has written a Python-
NumPy/SciPy [SciPy]2 . based textbook, Economic Dynamics: Theory and Computation
On a historical note, statsmodels was started by Jonathan [Stachurski], and Alan Isaac’s "Simulating Evolutionary Games:
Taylor, a statistician now at Stanford, as part of SciPy under the a Python-Based Introduction" showcases Python’s abilities for
name models. Eventually, models was removed from SciPy and implementing agent-based economic models [Isaac].
became part of the NIPY neuroimaging project [NIPY] in order to
mature. Improving the models code was later accepted as a SciPy- In depth arguments for the choice of Python are beyond
focused project for the Google Summer of Code 2009 and again the scope of this paper; however, Python is well known for its
in 2010. It is currently distributed as a SciKit, or add-on package simple syntax, gentle learning curve, and large standard library.
for SciPy. Beyond this, it is a language for much more than statistics and
The current main developers of statsmodels are trained as can be the one toolbox for researchers across discplines. A few
economists with a background in econometrics. As such, much examples of statistics-related packages that are outside of the
of the development over the last year has focused on econometric main numpy/scipy code are packages for Markov Chain Monte
applications. However, the design of statsmodels follows a con- Carlo and Bayesian statistics [PyMC], machine learning and mul-
sistent pattern to make it user-friendly and easily extensible by tivariate pattern analysis [scikits-learn], [PyMVPA], neuroimaging
developers from any discipline. New contributions and ongoing [NIPY] and neuroscience time series [NITIME], visualization
work are making the code more useful for common statistical [Matplotlib], [Enthought], and efficient handling of large datasets
modeling needs. We hope that continued efforts will result in a [PyTables].
package useful for all types of statistical modeling. We hope that statsmodels too can become an integral a part
of the Scientific Python community and serve as a step in the
* Corresponding author: [email protected] direction of Python becoming a serious open source language
§ American University
‡ CIRANO, University of North Carolina Chapel Hill for statistics. Towards this end, others are working on an R-like
formula framework to help users specify and manipulate models
Copyright © 2010 Skipper Seabold et al. This is an open-access article [charlton], and packages like pandas [pandas] (discussed in these
distributed under the terms of the Creative Commons Attribution License,
which permits unrestricted use, distribution, and reproduction in any medium, proceedings) and larry [larry] are providing flexible data structures
provided the original author and source are credited. and routines for data analysis currently lacking in NumPy.
STATSMODELS: ECONOMETRIC AND STATISTICAL MODELING WITH PYTHON 93

Statsmodels: Development and Design Package Overview

It should not be the case that different conclusions can be had Currently, we have five modules in the main codebase that contain
from the same data depending on the choice of statistical software statistical models. These are regression (least squares regression
or its version; however, this is precisely what Altman and Mac- models), glm (generalized linear models), rlm (robust linear mod-
Donald (2003) find [AltmanMcDonald]. Given the importance of els), discretemod (discrete choice models), and contrast (contrast
numerical accuracy and replicablity of research and the multitude analysis). Regression contains generalized least squares (GLS),
of software choices for statistical analysis, the development of weighted least squares (WLS), and ordinary least squares (OLS).
statsmodels follows a process to help ensure accurate and trans- Glm contains generalized linear models with support for six
parent results. This process is known as Test-Driven Development common exponential family distributions and at least ten standard
(TDD). In its strictest form, TDD means that tests are written link functions. Rlm supports M-estimator type robust linear models
before the functionality which it is supposed to test. While we do with support for eight norms. Discretemod includes several dis-
not often take the strict approach, there are several layers in our crete choice models such as the Logit, Probit, Multinomial Logit
development process that ensure that our results are correct versus (MNLogit), and Poisson within a maximum likelihood framework.
existing software (often R, SAS, or Stata). Any deviations from Contrast contains helper functions for working with linear con-
results in other software are noted in the test suite. trasts. There are also tests for heteroskedasticity, autocorrelation,
First, we employ a distributed version control system in which and a framework for testing hypotheses about linear combinations
each developer has his own copy of the code, a branch, to make of the coefficients.
changes outside of the main codebase. Once a model is specified, In addition to the models and the related post-estimation
early changes, such as working out the best API or bug hunting, results and tests, statsmodels includes a number of convenience
take place in the main branch’s, or trunk’s, sandbox directory so classes and functions to help with tasks related to statistical anal-
that they can be tried out by users and developers who follow ysis. These include functions for conveniently viewing descriptive
the trunk code. Tests are then written with results taken from statistics, a class for creating publication quality tables, and
another statistical package or Monte Carlo simulation when it is functions for translating foreign datasets, currently only Stata’s
not possible to obtain results from elsewhere. After the tests are binary .dta format, to numpy arrays.
written, the developer asks for a code review on our mailing list The last main part of the package is the datasets. There are
(http://groups.google.ca/group/pystatsmodels). When all is settled, currently fourteen datasets that are either part of the public domain
the code becomes part of the main codebase. Periodically, we re- or used with express consent of the original authors. These datasets
lease the software in the trunk for those who just want to download follow a common pattern for ease of use, and it is trivial to add
a tarball or install from PyPI, using setuptools’ easy_install. This additional ones. The datasets are used in our test suite and in
workflow, while not foolproof, helps make sure our results are examples as illustrated below.
and remain correct. If they are not, we are able to know why and
document discrepancies resulting in the utmost transparency for
Examples
end users. And if all else fails, looking at the source code is trivial
to do (and encouraged!). All of the following examples use the datasets included in
The design of the package itself is straightforward. The main statsmodels. The first example is a basic use case of the OLS
idea behind the design is that a model is itself an object to be used model class to get a feel for the rest of the package, using Long-
for data reduction. Data can be both endogenous and exogenous ley’s 1967 dataset [Longley] on the US macro economy. Note that
to a model, and these constituent parts are related to each other the Longley data is known to be highly collinear (it has a condition
through statistical theory. This statistical relationship is usually number of 456,037), and as such it is used to test accuracy of
justified by an appeal to discipline-specific theory. Note that in least squares routines than to examine any economic theory. First
place of endogenous and exogenous, one could substitute the we need to import the package. The suggested convention for
terms dependent and independent variables, regressand and re- importing statsmodels is
gressors, response and explanatory variables, etc., respectively, as >>> import scikits.statsmodels as sm
you prefer. We maintain the endogenous-exogenous terminology
throughout the package, however. Numpy is assumed to be imported as:
With this in mind, we have a base class, Model, that is intended >>> import numpy as np
to be a template for parametric models. It has two main attributes
endog and exog3 and placeholders for fit and predict methods. Then we load the example dataset.
LikelihoodModel is a subclass of Model that is the workhorse for >>> longley = sm.datasets.longley
the regression models. All fit methods are expected to return some
results class. Towards this end, we also have a base class Results The datasets have several attributes, such as descriptives and
and LikelihoodModelResults which inherits from Results. The copyright notices, that may be of interest; however, we will just
result objects have attributes and methods that contain common load the data.
post-estimation results and statistical tests. Further, these are >>> data = longley.load()
computed lazily, meaning that they are not computed until the
user asks for them so that those who are only interested in, say, Many of the Dataset objects have two attributes that are helpful
the fitted parameters are not slowed by computation of extraneous for tests and examples -endog and exog- though the whole dataset
results. Every effort is made to ensure that the constructors of is available. We will use them to construct an OLS model instance.
each sublcass of Model, the call signatures of its methods, and the The constructor for OLS is
post-estimation results are consistent throughout the package. def __init__(self, endog, exog)
94 PROC. OF THE 9th PYTHON IN SCIENCE CONF. (SCIPY 2010)

It is currently assumed that the user has cleaned the dataset and Note that you must specify the distribution family of
that a constant is included, so we first add a constant and then the endogenous variable. The available families in scik-
instantiate the model. its.statsmodels.families are Binomial, Gamma, Gaussian, In-
>>> data.exog = sm.add_constant(data.exog) verseGaussian, NegativeBinomial, and Poisson.
>>> longley_model = sm.OLS(data.endog, data.exog) The above examples also uses the default canonical logit link
for the Binomial family, though to be explicit we could do the
We are now ready to fit the model, which returns a RegressionRe- following
sults class.
>>> links = sm.families.links
>>> longley_res = longley_model.fit() >>> glm_bin = sm.GLM(data.endog, data.exog,
>>> type(longley_res) family=sm.families.Binomial(link=
<class 'sm.regression.RegressionResults'> links.logit))

By default, the least squares models use the pseudoinverse to We fit the model using iteratively reweighted least squares, but we
compute the parameters that solve the objective function. must first specify the number of trials for the endogenous variable
>>> params = np.dot(np.linalg.pinv(data.exog), for the Binomial model with the endogenous variable given as
data.endog) (success, failure).
The instance longley_res has several attributes and methods of >>> trials = data.endog.sum(axis=1)
>>> bin_results = glm_bin.fit(data_weights=trials)
interest. The first is the fitted values, commonly β in the general >>> bin_results.params
linear model, Y = Xβ , which is called params in statsmodels. array([ -1.68150366e-02, 9.92547661e-03,
-1.87242148e-02, -1.42385609e-02,
>>> longley_res.params 2.54487173e-01, 2.40693664e-01,
array([ 1.50618723e+01, -3.58191793e-02, 8.04086739e-02, -1.95216050e+00,
-2.02022980e+00, -1.03322687e+00, -3.34086475e-01, -1.69022168e-01,
-5.11041057e-02, 1.82915146e+03, 4.91670212e-03, -3.57996435e-03,
-3.48225863e+06]) -1.40765648e-02, -4.00499176e-03,
-3.90639579e-03, 9.17143006e-02,
Also available are 4.89898381e-02, 8.04073890e-03,
>>> [_ for _ in dir(longley_res) if not 2.22009503e-04, -2.24924861e-03,
_.startswith('_')] 2.95887793e+00])
['HC0_se', 'HC1_se', 'HC2_se', 'HC3_se', 'aic',
'bic', 'bse', 'centered_tss', 'conf_int', Since we have fit a GLM with interactions, we might be interested
'cov_params', 'df_model', 'df_resid', 'ess', in comparing interquartile differences of the response between
'f_pvalue', 'f_test', 'fittedvalues', 'fvalue', groups. For instance, the interquartile difference between the
'initialize', 'llf', 'model', 'mse_model',
'mse_resid', 'mse_total', 'nobs', 'norm_resid',
percentage of low income households per school district while
'normalized_cov_params', 'params', 'pvalues', holding the other variables constant at their mean is
'resid', 'rsquared', 'rsquared_adj', 'scale', 'ssr',
>>> means = data.exog.mean(axis=0) # overall means
'summary', 't', 't_test', 'uncentered_tss', 'wresid']
>>> means25 = means.copy() # copy means
>>> means75 = means.copy()
All of the attributes and methods are well-documented in the
docstring and in our online documentation. See, for instance, We can now replace the first column, the percentage of low income
help(longley_res). Note as well that all results objects carry an households, with the value at the first quartile using scipy.stats and
attribute model that is a reference to the original model instance likewise for the 75th percentile.
that was fit whether or not it is instantiated before fitting.
>>> from scipy.stats import scoreatpercentile as sap
Our second example borrows from Jeff Gill’s Generalized >>> means25[0] = sap(data.exog[:,0], 25)
Linear Models: A Unified Approach [Gill]. We fit a Generalized >>> means75[0] = sap(data.exog[:,0], 75)
Linear Model where the endogenous variable has a binomial
And compute the fitted values, which are the inverse of the link
distribution, since the syntax differs somewhat from the other
function at the linear predicted values.
models. Gill’s data comes from the 1998 STAR program in Cali-
fornia, assessing education policy and outcomes. The endogenous >>> lin_resp25 = glm_bin.predict(means25)
>>> lin_resp75 = glm_bin.predict(means75)
variable here has two columns. The first specifies the number of
students above the national median score for the math section Therefore the percentage difference in scores on the standardized
of the test per school district. The second column specifies the math tests for school districts in the 75th percentile of low income
number of students below the median. That is, endog is (number households versus the 25th percentile is
of sucesses, number of failures). The explanatory variables for >>> print "%4.2f percent" % ((lin_resp75-
each district are measures such as the percentage of low income lin_resp25)*100)
families, the percentage of minority students and teachers, the -11.88 percent
median teacher salary, the mean years of teacher experience, per-
The next example concerns the testing of joint hypotheses on
pupil expenditures, the pupil -teacher ratio, the percentage of
coefficients and is inspired by a similar example in Bill Greene’s
student taking college credit courses, the percentage of charter
Econometric Analysis [Greene]. Consider a simple static invest-
schools, the percent of schools open year round, and various
ment function for a macro economy
interaction terms. The model can be fit as follows
>>> data = sm.datasets.star98.load() ln It = β1 + β2 lnYt + β3 it + β4 ∆pt + β5t + εt (1)
>>> data.exog = sm.add_constant(data.exog)
>>> glm_bin = sm.GLM(data.endog, data.exog, In this example, (log) investment, It is a function of the interest
family=sm.families.Binomial()) rate, it , inflation, ∆pt , (log) real GDP, Yt , and possibly follows a
STATSMODELS: ECONOMETRIC AND STATISTICAL MODELING WITH PYTHON 95

linear time trend, t. Economic theory suggests that the following of the Google Summer of Code 2010. This includes models for
model may instead be correct time-series analysis, kernel density estimators and nonparametric
regression, panel or longitudinal data models, systems of equation
ln It = β1 + lnYt + β3 (it − ∆pt ) + εt (2) models, and information theory and maximum entropy models.
In terms of the (1) this implies that β3 + β4 = 0, β2 = 1, and β5 = We hope that the above discussion gives some idea of the
0. This can be implemented in statsmodels using the macrodata appoach taken by the project and provides a good overview
dataset. Assume that endog and exog are given as in (1) of what is currently offered. We invite feedback, discussion, or
contributions at any level. If you would like to get involved,
>>> inv_model = sm.OLS(endog, exog).fit()
please join us on our mailing list available at http://groups.google.
Now we need to make linear restrictions in the form of Rβ = q com/group/pystatsmodels or on the scipy-user list. If you would
>>> R = [[0,1,0,0,0],[0,0,1,1,0],[0,0,0,0,1]] like to follow along with the latest development, the project blog
>>> q = [1,0,0] is http://scipystats.blogspots.com and look for release announce-
ments on the scipy-user list.
Rβ = q implies the hypotheses outlined above. We can test the
All in all, we believe that the future for Python and statistics
joint hypothesis using an F test, which returns a ContrastResults
looks bright.
class
>>> Ftest = inv_model.f_test(R,q)
>>> print Ftest Acknowledgements
<F test: F=array([[ 194.4428894]]), In addition to the authors of this paper, many others have con-
p=[[ 1.27044954e-58]], df_denom=197, df_num=3>
tributed to the codebase. Thanks are due to Jonathan Taylor and
Assuming that we have a correctly specified model, given the high contributors to the models code while it was in SciPy and NIPY.
value of the F statistic, the probability that our joint null hypothesis Thanks also go to those who have provided code, constructive
is true is essentially zero. comments, and discussion on the mailing lists.
As a final example we will demonstrate how the SimpleTable
class can be used to generate tables. SimpleTable is also currently R EFERENCES
used to generate our regression results summary. Continuing the
[AltmanMcDonald] M. Altman and M.P. McDonald. 2003. "Replication with
example above, one could do Attention to Numerical Accuracy." Political Analysis,
>>> print inv_model.summary(yname="lninv", 11.3, 302-7.
xname=["const","lnY","i","dP","t"]) [BilinaLawford] R. Bilina and S. Lawford. July 4, 2009. Python for
Unified Research in Econometrics and Statistics, July
To build a table, we could do: 4, 2009. Available at SSRN: http://ssrn.com/abstract=
1429822
>>> gdpmean = data.data['realgdp'].mean() [charlton] Charlton. Available at https://github.com/charlton
>>> invmean = data.data['realinv'].mean() [ChoiratSeri] C. Choirat and R. Seri. 2009. "Econometrics with
>>> gdpstd = data.data['realgdp'].std() Python." Journal of Applied Econometrics, 24.4, 698-
>>> invstd = data.data['realinv'].std() 704.
>>> mydata = [[gdpmean, gdpstd],[invmean, [CryerChan] J.D. Cryer and K.S. Chan. 2008. Time Series Analysis:
invstd]] with Applications in R, Springer.
>>> myheaders = ["Mean", "Std Dev."] [Enthought] Enthought Tool Suite. Available at http://code.enthought.
>>> mystubs = ["Real GDP", "Real Investment"] com/.
>>> tbl = sm.iolib.SimpleTable(mydata, [Gill] J. Gill. 2001. Generalized Linear Models: A Unified
myheaders, mystubs, title = Approach. Sage Publications.
"US Macro Data", data_fmts=['%4.2f']) [Greene] W. H. Greene. 2003. Econometric Analysis 5th ed. Pren-
>>> print tbl tice Hall.
US Macro Data [Gretl] Gnu Regression, Econometrics, and Time-series Library:
================================ gretl. Available at http://gretl.sourceforge.net/.
Mean Std Dev. [Isaac] A.G. Isaac. 2008. "Simulating Evolutionary Games: a
-------------------------------- Python- Based Introduction." Journal of Artificial So-
Real GDP 7221.17 3207.03 cieties and Social Simulation. 11.3.8. Available at http:
Real Investment 1012.86 583.66 //jasss.soc.surrey.ac.uk/11/3/8.html
-------------------------------- [KleiberZeileis] C. Kleiber and A. Zeileis. 2008. Applied Econometrics
with R, Springer.
LaTeX output can be generated with something like [Longley] J.W. Longley. 1967. "An Appraisal of Least Squares
>>> fh = open('./tmp.tex', 'w') Programs for the Electronic Computer from the Point of
>>> fh.write(tbl.as_latex_tabular()) View of the User." Journal of the American Statistical
>>> fh.close() Association, 62.319, 819-41.
[Matplotlib] J. Hunter, et al. Matplotlib. Available at http://matplotlib.
While not all of the functionality of statsmodels is covered in sourceforge.net/index.html.
the above, we hope it offers a good overview of the basic usage
1. The examples reflect the state of the code at the time of writing. The
from model to model. Anything not touched on is available in our main model API is relatively stable; however, recent refactoring has changed
documentation and in the examples directory of the package. the organization of the code. See online documentation for the current usage.
2. Users who wish to learn more about NumPy can do so at http://www.
scipy.org/Tentative_NumPy_Tutorial, http://www.scipy.org/Numpy_Example_
Conclusion and Outlook List, or http://mentat.za.net/numpy/intro/intro.html. For those coming from
Statsmodels is very much still a work in progress, and perhaps R or MATLAB, you might find the following helpful: http://mathesaurus.
sourceforge.net/ and http://www.scipy.org/NumPy_for_Matlab_Users
the most exciting part of the project is what is to come. We
3. The exog attribute is actually optional, given that we are developing
currently have a good deal of code in our sandbox that is being support for (vector) autoregressive processes in which all variables could at
cleaned up, tested, and pushed into the main codebase as part times be thought of as "endogenous".
96 PROC. OF THE 9th PYTHON IN SCIENCE CONF. (SCIPY 2010)

[larry] K.W. Goodman. Larry: Labeled Arrays in Python. Avail-

able at http://larry.sourceforge.net/.
[NIPY] NIPY: Neuroimaging in Python. Available at http://nipy.
org.
[NITIME] Nitime: time-series analysis for neuroscience. Available
at http://nipy.org/nitime
[pandas] W. McKinney. Pandas: A Python Data Analysis Library.
Available at http://pandas.sourceforge.net/.
[PyMC] C. Fonnesbeck, D. Huard, and A. Patil, PyMC: Pythonic
Markov chain Monte Carlo. Available at http://code.
google.com/p/pymc/
[PyMVPA] M. Hanke, et. al. PyMVPA: Multivariate Pattern Analysis
in Python. Available at http://www.pymvpa.org/.
[PyTables] F. Alted, I. Vilata, et al. PyTables: Hierarchical Datasets
in Python. Available at http://www.pytables.org.
[scikits-learn] Pedregosa, F. et al. scikits.learn: machine learning
in Python. Available at http://scikits-learn.sourceforge.
net.
[SciPy] T. Oliphant, et al. SciPy. Available at http://www.scipy.
org.
[Stachurski] J. Stachurski. 2009. Economic Dynamics: Theory and
Computation. MIT Press.
[Vinod] H.D. Vinod. 2008. Hands-on Intermediate Econometrics
Using R, World Scientific Publishing.
[YaltaLucchetti] A.T. Yalta and R. Lucchetti. 2008. "The GNU/Linux Plat-
form and Freedom Respecting Software for Economists."
Journal of Applied Econometrics, 23.3, 279-86.
[YaltaYalta] A.T. Yalta and A.Y. Yalta. 2010. Should Economists Use
Open Source Software for Doing Research. Computa-
tional Economics, 35, 371-94.