Statistical Computing Software Reviews

Multiple Imputation in Practice: Comparison of Software

Packages for Regression Models With Missing Variables
Nicholas J. HORTON and Stuart R. LIPSITZ
differences between the observed and unobserved data (Barnard
and Meng 1999). One approach to incomplete data problems
Missing data frequently complicates data analysis for scientific that addresses these concerns is multiple imputation, which was
investigations. The development of statistical methods to ad- proposed 20 years ago by Rubin (1977) and described in detail
dress missing data has been an active area of research in recent by Rubin (1987) and Schafer (1997). A concise and readable
decades. Multiple imputation, originally proposed by Rubin in primer can be found in Schafer (1999), while Rubin (1996) pro-
a public use dataset setting, is a general purpose method for an- vided an extensive bibliography.
alyzing datasets with missing data that is broadly applicable to a This article reviews multiple imputation, describes assump-
variety of missing data settings. We review multiple imputation tions that it requires, and reviews software packages that im-
as an analytic strategy for missing data. We describe and evaluate plement this procedure. We apply the methods and compare the
a number of software packages that implement this procedure, results using two examples—a child psychopathology dataset
and contrast the interface, features, and results. We compare the with missing outcomes and an artificial dataset with missing co-
packages, and detail shortcomings and useful features. The com- variates. We conclude with some discussion of the strengths and
parisons are illustrated using examples from an artificial dataset weaknesses of these implementations as well as advantages and
and a study of child psychopathology. We suggest additional limitations of imputation.
features as well as discuss limitations and cautions to consider
when using multiple imputation as an analytic strategy for in-
2. MULTIPLE IMPUTATION
complete data settings.
Rubin (1996) described multiple imputation as a three-step
KEY WORDS: Generalized linear models; Incomplete data; process. First, sets of plausible values for missing observations
Markov Chain Monte Carlo; Missing outcomes; Missing pre- are created that reflect uncertainty about the nonresponse model.
dictors. Each of these sets of plausible values can be used to “fill-in” the
missing values and create a “completed” dataset. Second, each
of these datasets can be analyzed using complete-data methods.
Finally, the results are combined, which allows the uncertainty
regarding the imputation to be taken into account.
1. INTRODUCTION The method of multiple imputation was first proposed in a
Missing data is a commonly occurring complication in many public-use survey data setting. Consider an intensive survey of
scientific investigations. Determining the appropriate analytic a small geographical area, where some observations were miss-
approach in the presence of incomplete observations is a major ing or incomplete. Other researchers, whom we denote as users
question for data analysts. The development of statistical meth- of the survey, may have interest in these data. Care needs to be
ods to address missing data has been an active area of research in taken by the creators of the survey to ensure that respondents
recent decades (Rubin 1976; Little and Rubin 1987; Laird 1988; are not uniquely identified based on characteristics included in
Ibrahim 1990; Little 1992; Robins, Rotnitzky, and Zhao 1994, the public dataset. Note that disclosure of the identity of the
1995; Horton and Laird 1999). There are three types of concerns respondents may occur if too much information (such as area,
that typically arise with missing data: (1) loss of efficiency; (2) gender, ethnicity, occupation, age, and so on) is provided regard-
complication in data handling and analysis; and (3) bias due to ing the sample (Fienberg 1994; Fienberg and Willenborg 1998;
Zaslavsky and Horton 1998).
Multiple imputation remains ideally suited to this setting,
Nicholas J. Horton is Assistant Professor, Department of Epidemiology and since the creators can use auxiliary confidential and detailed
Biostatistics, Boston University School of Public Health, and Department of information that would be inappropriate to include in the pub-
Medicine, Boston University School of Medicine, 715 Albany Street, T3E,
Boston, MA 02118 (E-mail: [email protected]). Stuart R. Lipsitz is Professor, lic dataset (Rubin 1996). This comprehensive set of information
Department of Biometry and Epidemiology, Medical University of South Car- about the study can be used to “fill-in” or impute sets of values
olina, Charleston, SC 29425. The authors are grateful for the support provided for incomplete observations. Given the complete datasets, users
by NIMH grant R01-MH546932. We also thank Gwendolyn Zahner for use of
the child psychopathology dataset, which was conducted under contract to the
may utilize existing software to analyse each of the datasets.
Connecticut Department of Children and Youth Services while Dr. Zahner was Finally, given the results for each analysis, an overall summary
on the faculty of the Yale University Child Study Center. is straightforward to calculate.

244 The American Statistician, August 2001, Vol. 55, No. 3

c 2001 American Statistical Association
 Although multiple imputation was initially proposed by Ru- Another important distinction regarding the missing data
bin for public use data, its use has broadened to general pur- refers to the pattern of missing data. If the data matrix can be
pose missing data settings. Here we consider general regression rearranged in such a way that there is a hierarchy of missingness,
models with outcomes (denoted by Y, which may be scalar or such that observing a particular variable Zb for a subject implies
vector valued) and a vector of predictors (denoted by X). For that Za is observed, for a < b, then the missingness is said to be
a given subject, these quantities are either observed or miss- monotone. Simpler imputation methods can be used if the pattern
ing. We denote Yobs as the observed component of the outcome is monotone, though a monotone pattern is uncommon in most
and Xobs as the observed components of the predictors. Simi- complex investigations. It may be possible, however, to create a
larly, we denote Ymis and Xmis as the unobserved components monotone missingness pattern that separates out a small number
of the outcome and predictors, respectively. We will also refer of observations that are non-monotone. In the nonmonotone ex-
to Zmis = (Ymis , Xmis ) and Zobs = (Yobs , Xobs ). ample in Figure 1, all but the first observation can be rearranged
Primary interest revolves around the regression parameters β into a monotone pattern (i.e., 83% of the dataset). This type of
governing the conditional distribution of Y given X: f (Y|X, β), hybrid pattern is exploited by a number of computer packages,
and we are concerned with efficiency issues, complications in since this allows a much simpler model to be estimated.
analysis, and bias. Before describing multiple imputation in de- It is important to note that in many realistic settings, datasets
tail, we will review classifications for the probability distribution may have missing outcomes as well as missing predictors (Little
generating the missing data, using the nomenclature of Little and 1992). Such patterns tend to complicate analysis, particularly for
Rubin (1987). regression models with many predictors.
2.1 Missing Data Classifications 2.2 Steps for Multiple Imputation
Let Z be partially observed, where R is a set of response More formally, the three steps for multiple imputation consist
indicators (i.e., Rj = 1 if the jth element of Z is observed, and of:
equals 0 otherwise), governed by parameters φ. The missing
completely at random (MCAR) assumption is defined as Imputation: Generate a set of m > 1 plausible values for
Zmis = (Ymis , Xmis ).
P (R|Z) = P (R|Zobs , Zmis ) = P (R|φ), Analysis: Analyze the m datasets using complete-case methods.
Combination: Combine the results from the m analyses.
where in addition φ and β are presumed distinct. Heuristically,
this assumption states that missingness is not related to any fac-
tor, known or unknown, in the study. Imputation step—The imputation step is perhaps most criti-
It may be more plausible to posit that missingness is missing cal, since it relies upon assumptions regarding the missingness
at random (MAR), which assumes that law that generated the observed sample. The goal of the imputa-
tion is to account for the relationships between the unobserved
P (R|Z) = P (R|Zobs , φ). and observed variables, while taking into account the uncer-
tainty of the imputation. The MAR assumption (which is gener-
Heuristically, this states that the missingness depends only on
ally assumed for many missing data methods, and as previously
observed quantities, which may include outcomes, and a rich
noted is untestable without additional information) is key to the
set of predictors. It is possible to statistically test the MCAR
validity of multiple imputation. Use of this assumption allows
assumption, against the alternate hypothesis that missingness is
the analyst to generate imputations (Z{1} , Z{2} , . . . Z{m} ) from
MAR (Diggle, Liang, and Zeger 1994; Little 1988).
the distribution f (Zmis |Zobs ), since after conditioning on Zobs ,
Finally, if the missingness law P (R|Z) cannot be simplified
missingness is assumed to be due to chance.
(i.e., it depends on unobserved quantities), the process is termed
There are a variety of imputation models that have been used.
nonignorable. In a nonignorable nonresponse setting, the cor-
When missingness is monotone, simple methods have been pro-
rect specification of the missingness law must be given to yield
posed, including (for continuous variables) propensity methods
consistent estimates of the regression parameters. Without addi-
(Rosenbaum and Rubin 1983), predictive mean matching (Lit-
tional information, it is impossible to test the MAR assumption
tle 1988), and (for discrete variables) discriminant analysis or
against a nonignorable alternative (Little and Rubin 1987).
logistic regression. For more complicated missingness, Markov
Chain Monte Carlo (MCMC) approaches have been suggested.
Both the predictive mean matching and MCMC approaches re-
Monotone Nonmonotone
quire assumptions of multivariate normality, but there is some
Obs Z1 Z2 Z3 Z1 Z2 Z3 evidence (see, e.g., Schafer 1997) that the inferences tend to be
1 O O O O M O robust to minor departures from this assumption.
2 O M M O M M As an example of an imputation model, we will describe the
3 O M M O M M
4 O M M O M M predictive mean matching approach, where a linear regression
5 O O M O O M is postulated for the distribution of a partially observed variable,
6 O O M O O M conditional on other factors. For the predictive mean matching
7 O O M O O M
approach with a variable Zj with missing values, a model is fit
8 O O O O O O
using complete observations for Z1 , . . . , Zj−1 :
Figure 1. Monotone and Nonmonotone Patterns of Missingness
(O = observed, M = missing). E[Zj |φ] = φ0 + φ1 Z1 + φ2 Z2 + · · · + φj−1 Zj−1 . (1)
The American Statistician, August 2001, Vol. 55, No. 3 245
Next, new parameters φ∗ are drawn from the distribution of the “Complete data” analysis step—The next step in the imputa-
parameters (since these values were estimated and not known tion process is to carry out the analysis of interest for each of the
with certainty). Finally, for the ith imputation, the missing values m imputed complete-observation datasets, storing the parameter
are replaced by: vector and standard error estimates.

(i)
Zj = φ∗0 + φ∗1 Z1 + φ∗2 Z2 + · · · + φ∗j−1 Zj−1 + σ ∗ , Combination step—Finally, the results are combined using
results from Rubin (1987), to calculate estimates of the within
where σ ∗ is the estimate of variance from the model and  is a imputation and between imputation variability. These statistics
simulated normal random variate. We refer to this as the regres- account for the variability of the imputations and assuming that
sion method. A variant of this approach imputes the observed the imputation model is correct, provide consistent estimates of
value of Zj that is closest to Ẑj in the dataset; this ensures that the parameters and their standard errors. There has been an ex-
imputed values are plausible, and may be more appropriate if tensive literature regarding the asymptotic behavior of multiple
the normality assumption is violated. We refer to this as the imputation methods (Barnard and Rubin 1999; Meng and Rubin
predictive mean matching method. The predictive mean model 1992; Robins and Wang 2000; Rubin 1996; Wang and Robins
assumes a linear regression model and a monotone structure, 1998) these issues are not further considered here.
otherwise there will be missing predictors in model (1).
The propensity score method uses a different model for im- Notes about imputation—It should be noted that one advan-
putation. Here, values are imputed from observations that are tage of multiple imputation as an analytic approach is that it
equally likely to be missing, by fitting a logistic regression model allows the analyst to incorporate additional information into the
for the missingness indicators. Allison (2000) noted that this imputation model. This auxiliary (or extraneous) information
method can yield serious bias for imputation of missing covari- may not be of interest in the regression model, but may make
ates under some settings. the MAR assumption increasingly plausible (Liu, Taylor, and
For discrete incomplete variables, discriminant analysis (or Belin 2000; Rubin 1996); such information is straightforward
logistic regression for dichotomous variables) can be used to to incorporate into the imputation model.
impute values based on the estimated probability that a miss- A useful quantity in interpreting results from multiple im-
ing value takes on a certain value P (Zjmis = k|Zobs ). Using putation is an estimate of the fraction of missing information
Bayes’s theorem, this can be calculated from estimates of the (Rubin 1987). This quantity, typically denoted by γ̂, denotes
joint distribution of Z. how the missing data influence the uncertainty of estimates of β
Finally, the MCMC method constructs a Markov chain to (Schafer 1997). It has been noted that even with a large fraction
simulate draws from the posterior distribution of f (Zmis |Zobs ). of missing information, a relatively small number of imputa-
This can be implemented using the IP algorithm (Schafer 1997), tions provides estimates of standard errors that are almost fully
where at the tth iteration the steps can be defined as: efficient (Schafer 1997). Schafer (1999) suggested that no more
than 10 imputations are usually required, though this should be
investigated more closely if the fraction of missing information
Imputation-step: Draw Zmis,(t+1) from f (Z|Zobs , φ(t) ).
is large. In any case, the appropriate number of imputations can
Parameter-step: Draw φ(t+1) from f (φ|Zobs , Zmis,(t+1) ). be informally determined by carrying out replicate sets of m
imputations and determining whether the estimates are stable
between sets.
The Markov chain Before the advent of general purpose packages that supported

       multiple imputation, the process of generating imputed datasets,
Z(1) , φ(1) , Z(2) , φ(2) , . . . , Z(t+1) , φ(t+1) , . . . managing the results from each of the m datasets, and combining
the results required specialized programming or use of macros
can be shown to converge to the posterior distribution of inter- that were difficult to use. The packages reviewed in this pa-
est. This method has the advantage that it can handle arbitrary per, though still more complicated than complete case methods,
patterns of missing data. Schafer (1997) provided a complete greatly facilitate the process of using multiple imputation.
exposition of the method in the imputation setting, while Gilks,
Richardson, and Spiegelhalter (1996) described the background
as well as other applications. As a computational tool MCMC 3. SOFTWARE PACKAGES
has some downsides; it requires an assumption of multivari- SOLAS version 3.0
ate normality, it is complicated and computationally expensive. Statistical Solutions (North American office)
Convergence is difficult to determine, and remains more of an Stonehill Corporate Center, Suite 104,
art form than a science. However, MCMC is available in SAS, 999 Broadway, Saugus, MA 01906, USA.
S-Plus, and MICE, and thus is becoming more mainstream. Tel (781) 231-7680
Some practical suggestions for what variables to include in http://www.statsol.ie/solas/solas.htm, [email protected]
the imputation model were given by van Buuren, Boshuizen,
and Knook (1999). They recommended that this set of variables SOLAS is designed specifically for the analysis of datasets
includes those in the complete data model, factors known to be with missing observations. It offers a variety of multiple impu-
associated with missingness, and factors that explain a consid- tation techniques in a single, easy-to-use package with a well-
erable amount of variance for the target variables. designed graphical user interface.
246 Statistical Computing Software Reviews
 SOLAS supports predictive mean model (using the closest ob- tensive examples with numerous screen-shots were useful in
served value to the predicted value) and propensity score mod- introducing the user to SOLAS.
els for missing continuous variables, and discriminant models A single user license for SOLAS 3.0 costs $1,295 ($995 for
for missing binary and categorical variables. Once the multiple academic customers), while an upgrade from previous versions
datasets are created, the package allows the calculation of de- costs $495.
scriptive statistics, t tests and ANOVA, frequency tables, and SAS 8.2 (beta) SAS Institute Inc.
linear regression. SAS Campus Drive
The system automatically provides summary measures by Cary, NC 27513-2414
combining the results from the multiple analyses. These re- (919) 677-8000
ports can be saved as rich text format (rtf) files. It has extensive http://www.sas.com, [email protected]
capabilities to read and write database formats (1-2-3, dBase,
FoxPro, Paradox), spreadsheets (Excel), and statistical packages
(Gauss, Minitab, SAS, S-Plus, SPSS, Stata, Statistica, and Sys- The SAS System is described as an integrated suite of software
tat). Imputed datasets can be exported to other statistical pack- for enterprise-wide information delivery, which includes a ma-
ages, though this is a somewhat cumbersome process, since the jor module for statistical analysis, as implemented in SAS/STAT.
combination of results from the multiple analyses then needs to In release 8.1 two new experimental procedures (PROC MI and
be done manually. PROC MIANALYZE) were made available, which implemented
The new script language facility is particularly useful in doc- multiple imputation. The interface for PROC MI changed sub-
umenting the steps of a multiple imputation run, and for con- stantially in release 8.2. SAS anticipates putting PROC MI and
ducting simulations. It can be set up to automatically record the PROC MIANALYZE into production for release 9.
settings from a menu-based multiple imputation session, and The imputation step is carried out by PROC MI, which al-
store this configuration in a file for later revision and reuse. lows use of either monotone (predictive mean matching, de-
SOLAS includes the ability to view missing data patterns, re- noted by REGRESSION, or propensity, denoted by PROPEN-
view the quantity and positioning of missing values, and classify SITY) or nonmonotone (using MCMC) missingness methods.
them into categories of monotone or nonmonotone missingness. The MCMC methods can also be used in a hybrid model where
Because it does not consolidate observations with the same pat- the dataset is divided into monotone and nonmonotone parts, and
tern of missing data, however, this feature is of limited utility in a regression method is used for the monotone component. Exten-
large datasets. sive control and graphical diagnostics of the MCMC methods are
A nice feature of SOLAS is the fine-grained and intuitive provided. SAS supports transformation and back-transformation
control of the details of the imputation model. As an example, of variables. This may make an assumption of multivariate nor-
incorporating auxiliary information (variables in the imputation mality, needed for the REGRESSION and MCMC methods,
model but not in the regression model of interest) is straightfor- more tenable (Schafer 1997).
ward. Once PROC MI has been run, use of complete data methods
is straightforward; the only addition is the specification of a
There are a number of limitations to SOLAS’ implementa-
BY statement to repeat these methods (i.e., PROC GLM, PROC
tion. It is primarily a package for multiple imputation in linear
PHREG, or PROC LOGISTIC) for each value of the variable
regression models, and has limited data manipulation ability.
Imputation . This approach is attractive, since it allows the
While there are extensive options for linear regression, it lacks
full range of regression models available within SAS to be used
the completeness of general-purpose packages (e.g., specifica-
in imputation.
tion of interactions is cumbersome). Nonlinear regression meth-
The results are combined using PROC MIANALYZE, which
ods, such as logistic or survival models, are not supported. Non-
provides a clear summary of the results. SAS provides an op-
monotone missingness is handled in an ad-hoc fashion. While tion (EDF) to use the adjusted degrees of freedom suggested
this may be acceptable in many applications, it is not always by Barnard and Rubin (1999), and it displays estimates of the
appropriate (support for MCMC methods, not available in SO- fraction of missing information for each parameter.
LAS, are particularly attractive in this setting). By default, SO- A disadvantage of the imputation methods provided by PROC
LAS does not display any estimates of the fraction of missing MI is that the analyst has little control over the imputation model
information (this can be calculated separately from the regres- itself. In addition, for the regression and MCMC methods, SAS
sion model), nor standard error estimates for the intercept. The does not impute an observed value that is closest to the predicted
default behavior uses a fixed seed for the random number gen- value (i.e., there is no support for predictive mean matching using
erator seed. While this can be set to 0 (or left blank) to use clock observed values). Instead, it uses an assumption of multivariate
time as seed (except within the script language), the default seed normality to generate a plausible value for the imputation. SAS
will always yield the same imputation results. Using the clock allows the analyst to specify a minimum and maximum value
time as a pseudo-random seed would seem a more reasonable for imputed values on a variable-by-variable basis, as well as
default. the ability to round imputed values. In addition, a SAS data step
Installation was straightforward, and the interface was clear could be used to generate observed values. In practice, however,
and intuitive. The documentation (which consisted of three man- both these approaches are somewhat cumbersome.
uals, with a total of 487 pages) was well organized (it was the No additional installation was needed for PROC MI/PROC
only documentation with an index), and easy to read. The ex- MIANALYZE, since they are bundled with SAS/STAT. The doc-
The American Statistician, August 2001, Vol. 55, No. 3 247
umentation was terse (69 pages for PROC MI, 31 pages for logistic and polytomous regression, and discriminant analysis.
PROC MIANALYZE), but well organized; a number of exam- Nonmonotone missingness is handled by using chained equa-
ples were provided. tions (MCMC) to loop through all missing values. Extensive
SAS is licensed on an annual basis, on a per-module basis. graphical summaries of the MCMC process are provided. In ad-
An annual license for base SAS and SAS/STAT is $3,900 for the dition, MICE allows users to program their own imputation func-
first year, and $1,900 for subsequent years. Academic discounts tions, which is useful for undertaking sensitivity analyses of dif-
are generally available. ferent (possibly nonignorable) missingness models. The system
allows transformation of variables, and fine-grained control over
Missing Data Library for S-Plus the choice of predictors in the imputation model. The imputation
Insightful (formerly MathSoft) step is carried out using the mice() function. For continuous
(800) 569-0123 missing variables, MICE supports imputation using the norm
http://www.insightful.com, [email protected] function (similar to SAS’ REGRESSION option), and the pmm
function (similar to SOLAS’ predictive mean matching). Com-
pleted datasets can be extracted using the complete() func-
S-Plus 6.0 features a new missing data library, which extends tion, or can be run for each imputation using the lm.mids() or
S-Plus to support model-based missing data models, by use of glm.mids() function. Finally, results can be combined using
the EM algorithm (Dempster, Laird, and Rubin 1977) and data the pool() function.
augmentation (DA) algorithms (Tanner and Wong 1987). DA Although computationally attractive, the chained equation ap-
algorithms can be used to generate multiple imputations. The proach implemented in MICE requires assumptions about the
missing data library provides support for multivariate normal existence of the multivariate posterior distribution used for sam-
data (impGauss), categorical data (impLoglin), and con- pling, however, it is not always certain that such a distribution
ditional Gaussian models (impCgm) for imputations involving exists (van Buuren et al. 1999). Like SOLAS, MICE uses a fixed
both discrete and continuous variables. seed for random number generation, which must be overriden
The package provides a concise summary of missing data during the imputation phase to avoid always generating the same
imputed values. It would be preferable to have this seed vary by
distributions and patterns (further described in the discussion
default, but allow the option to fix the seed to allow replication
of the examples), including both text-based and graphical dis-
of results.
plays. There are good diagnostics provided for the convergence
Installation was straightforward, though automated addition
of the data augmentation algorithms. The printed documenta-
of packages under R is only supported on Unix systems. In ad-
tion, while extensive (164 pages) is light on examples of impu-
dition to the mice package, under R, two additional add-on
tation, instead focusing more on data augmentation and maxi-
packages were required (MASS and nnet). The documentation
mum likelihood (EM) based approaches. It provides an excellent
was short (39 pages), terse (particularly regarding the imputa-
tutorial regarding missing data methods in general.
tion models) but clear. An example using the NHANES dataset
S-Plus has strong support for file input, including the ability
provided a summary of how to use the package. The manual
to connect directly to Excel, and to read files in a variety of included the help pages for each function in the library.
formats (including Access, dBASE, Gauss, Matlab, Paradox, S-Plus was described previously. R is free software for
SAS, SPSS, Stata, and Systat). Unix, Windows, and Macintosh that is distributed under a
The single-user license price for S-Plus 2000 Professional for GNU-style copyleft. More information can be found at the R
Windows is $2500; S-Plus 6.0 is expected to have similar pricing. project web site: www.r-project.org. The MICE library is freely
Discounted academic pricing is available including academic available, and may be downloaded from the www.multiple-
site licenses. imputation.com Web site.

MICE Other Packages and Routines

TNO Prevention and Health
Public Health Other packages that provide some support for imputa-
Wassenaarseweg 56 tion include SPSS, Joseph Schafer’s free software (macros
P.O. Box 2215 for S-Plus and stand-alone windows package NORM), Gary
2301 CE Leiden King’s Amelia program, IVEware, HLM, and LISREL.
The Netherlands Joseph Schafer’s list of multiple imputation software routines
(31) 71 518 18 18 (http://www.stat.psu.edu/∼jls/misoftwa.html) and the list main-
http://www.multiple-imputation.com tained by the Department of Statistics of TNO Prevention and
Health, (http://www.multiple-imputation.com) are helpful in
Multiple Imputation by Chained Equations (MICE) is a li- tracking the developments in this area.
brary distributed for S-Plus (described above) and R, a system SPSS requires the user to run the individual complete
for statistical computation and graphics, whose language and data models and combine the results. Schafer’s software
interface is very similar to S-Plus. are an excellent companion to his book, but they do
MICE provides a variety of imputation models, including not support general purpose regression modeling. Amelia
forms of predictive mean matching and regression methods, (http://gking.harvard.edu/stats.shtml) implements the EMis al-
248 Statistical Computing Software Reviews
> colSums(is.na(allison))
x1 y x2
0 0 5008
> apply(allison, 2, anyMissing)
x1 y x2
F F T
> round(100 * colMeans(is.na(allison)))
x1 y x2
0 0 50
> M <- miss(allison); plot(M); M
Summary of missing values
3 variables, 10000 observations, 2 patterns of missing values
1 variables (33%) have at least one missing value
5008 observations (50%) have at least one missing value
Breakdown by variable
V O name Missing % missing
1 3 x2 5008 50
V = Variable number used below, O = Original number (before sorting)
No missing values for variables: x1 y
Figure 2. Code to Describe Patterns of Missing Data for Artificial Data Example in S-Plus 6.0.

gorithm (King, Honaker, Joseph, and Scheve 2001) and performs covariance given by:
the imputation step, but does not provide support for analysis of  
these imputed datasets or combination of the results. IVEware  1 0.5 0 
(http://www.isr.umich.edu/src/smp/ive) by Raghunathan et al. is Σ = 0.5 1 0 .
 
a SAS version 6.12 callable routine built using the SAS macro 0 0 1
language. It extends multiple imputation to support complex The true regression model is given by:
survey sample designs. HLM (hierarchical linear and nonlinear
modeling) version 5 supports the analysis of multiply-imputed E[Y |X1 , X2 ] = β0 + β1 X1 + β2 X2 .
datasets, where multiple plausible datasets have been previously
created. LISREL 8.20 and later supports multiple imputation, We generated 10,000 observations from the multivariate normal
but the focus of this package is not on the regression models de- distribution: Y = X1 + X2 +  (i.e., β0 = 0, β1 = β2 = 1).
scribed in this article. Because of the existence of more complete Following Allison’s example, we caused approximately half of
implementations, these packages are not further discussed. the values of X2 in this dataset to be missing, according to the
following mechanisms:

MCAR: X2 is missing with probability 0.5

MAR X1 : X2 is missing if X1 < 0
4. EXAMPLE: ARTIFICIAL DATA
MAR Y: X2 is missing if Y < 0
We now replicate an artificial data example with missing co- NI X2 : X2 is missing if X2 < 0.
variates, as described by Allison (2000), to help illustrate how
multiple imputation models are fit using these packages. Let For approximately half the subjects Z = Zobs = (Y, X1 , X2 ),
X1 , X2 and  be multivariate normal with mean 0 and variance while for the balance Zobs = (Y, X1 ) and Zmis = X2 . In this

Table 1. Parameter Estimates (and standard errors) From Artificial Datasets (true parameter values 1.00 and 1.00)

Missing Complete MI MI MI
mechanism Parameter case propensity regression MCMC

MCAR X1 1.00 (0.016) 1.40 (0.014) 1.01 (0.014) 1.01 (0.012)

X2 1.01 (0.016) 0.50 (0.016) 1.00 (0.013) 1.00 (0.015)

MAR X1 X1 1.04 (0.024) 1.54 (0.012) 1.02 (0.016) 1.01 (0.023)

X2 0.98 (0.016) 0.58 (0.019) 0.98 (0.014) 0.99 (0.015)

MAR Y X1 0.70 (0.015) 1.32 (0.013) 1.03 (0.014) 1.02 (0.015)

X2 0.71 (0.015) 0.83 (0.019) 1.00 (0.012) 1.00 (0.014)

NI X2 X1 1.00 (0.017) 1.28 (0.013) 1.15 (0.017) 1.16 (0.017)

X2 1.00 (0.025) 1.15 (0.027) 1.21 (0.026) 1.20 (0.020)

The American Statistician, August 2001, Vol. 55, No. 3 249

Table 2. Estimated Fraction of Missing Information for the MCAR lar when the imputation process was repeated on two occasions
Scenario for Two Replications (each with 10 imputations) of MICE PMM
with 10 imputations each, or with one exception, when different
and SAS REGRESSION
programs (i.e., SOLAS, SAS, S-Plus 6.0, or MICE) were used.
Parameter MICE 1 MICE 2 SAS 1 SAS 2 A bug (which was reported and corrected) in the propensity
X1 0.25 0.20 0.21 0.47 score routines in the beta release of SAS 8.2 yielded inaccurate
X2 0.16 0.26 0.31 0.26 variance estimates under the MAR Y scenario.
We note that for the MCAR missingness scenario, the multiple
scenario, the complete case (CC) estimator (discarding all sub- imputation estimators had slightly smaller standard errors than
jects with X2 missing, the default option for missing data in the complete case estimator. As expected, the complete case es-
most computing packages) is unbiased for the MCAR, MAR timator was unbiased unless missingness depended on Y . When
X1 , and NI X2 mechanisms. For multiple imputation proce- the MAR assumption was violated (NI X2 ), imputation based
dures (assuming MAR missingness), the MCAR, MAR X1 and methods were biased. In addition, the propensity score models
MAR Y missingness scenarios should yield unbiased estimates were biased in all scenarios other than MCAR; this method is
of the regression parameters. Allison (2000) showed that ear- not recommended in this setting.
lier versions of SOLAS (that provided only a propensity score Table 2 displays estimates of the fraction of missing infor-
method) had systematic severe bias when missingness depended mation for the MCAR scenario for two replications of MICE
on X1 , X2 , or Y.
PMM and SAS REGRESSION. Both the parameters have more
Figure 2 displays the commands in the S-Plus 6.0 missing
than 20% missing information. Note that even with 10 imputa-
data library to describe patterns of missing data for the artificial
tions, there remains a great deal of variability in the estimates of
data example. The other packages provided similar exploratory
tools. the fraction of missing information, though there was much less
Table 1 displays the results from the imputation models us- variability in the parameter estimates for the regression model
ing complete case, propensity (using SOLAS), predictive mean in this example.
matching (REGRESSION, using SOLAS), and MCMC meth- Figure 3 displays the code needed to fit the imputation model
ods (using SAS) for each of the four missingness scenarios. The for each of the packages. The data are assumed stored in a dataset
results, which are consistent with those of Allison, were simi- or object named allison.

SAS

proc mi data=allison out=miout nimpute=10 noprint;

monotone method=reg;
var y x1 x2;
proc reg data=miout outest=outreg covout noprint;
model y = x1 x2;
by _Imputation_;
proc mianalyze data=outreg;
var Intercept x1 x2;
run;

S-plus 6.0 Missing Data library

library(missing)
emstart <- emGauss(allison)
worstFraction(emstart)
start <- matrix(rep(emstart$paramIter[2,],10),nrow=10,byrow=T)
imp <- impGauss(allison,start=start,control=list(niter=200))
fit <- miEval(lm(y ˜ x1 + x2,data=imp))
result <- miMeanSE(fit)

MICE

library(mice)
imp <- mice(allison,imputationMethod="pmm",m=10,seed=456)
fit <- lm.mids(y ˜ x1 + x2, imp)
result <- pool(fit)
Figure 3. Code to Fit Models for Artificial Data Example.

250 Statistical Computing Software Reviews

 5. EXAMPLE: CHILD PSYCHOPATHOLOGY research, particularly for measurements of child psychopathol-
We now consider an example with a missing outcome variable, ogy (Fitzmaurice, Laird, Zahner, and Daskalakis 1995; Offord
from a study of child psychopathology in urban and rural Con- et al. 1996; Horton and Laird 1999). We denote this report, based
necticut (Zahner, Jacobs, Freeman, and Trainor 1993; Zahner, on the Child Behavioral Checklist (Achenbach 1991a, CBCL)
Pawelkiewicz, DeFrancesco, and Adnopoz 1992; Zahner and as PXINT. We include in our analysis the 2,501 children with
Daskalakis 1997; Goldwasser and Fitzmaurice 2001). The mea- complete data on parent reports (of which 1,428 had complete
sure of psychopathology used in the study was the internalizing teacher reports). For questions where primary interest involves
problems scale (TXINT) of the Teacher’s Report Form (Achen- the teacher report, this auxiliary information is extraneous to the
bach 1991b, TRF). TXINT will be used as the outcome for a regression model, but it may be associated with the unobserved
linear regression model. TXINT ranges from 33 to 93 and can teacher report. We will consider multiple imputation models that
be considered to be approximately normal (Goldwasser and Fitz- incorporate this auxiliary information.
maurice 2001). Predictors of psychopathology, reported by the parent, in-
In the study, 43% of teacher ratings on children were un- cluded gender of the child (BOY: 1 = boy, 0 = otherwise);
observed. Missingness of this magnitude is not uncommon: a age of the child (OLD: 0 = age 6 to 8, 1 = age 9–11); social
similar rate was reported by Boyle et al. (1993) in their Ontario class (HIGH, MIDDLE, or LOW); area (RURAL, SUBURB,
Child Health Study. There were a variety of causes of missing- SMALL city, or LARGE city); maternal distress (MOMSTRS);
ness for the teacher reports, including school district nonpar- child’s health (HLTHPRO: 0 = good health, 1 = poor health);
ticipation, parental refusal to give consent, and teacher nonre- grade repetition (ACADPRO: 0 = no, 1 = yes); family stress
sponse. Fitzmaurice, Laird, and Zahner (1996) considered the (FAMSTRS: 0 = no, 1 = yes); and belonging to a single parent
question of whether the missingness in this dataset is related to household (MOMSING: 0=father figure present, 1 = no father
the unobserved teacher’s rating, and found no evidence for this figure present).
hypothesis. Thus, the missing at random assumption appears to Here we consider TXINT to be the outcome Y, which is not
be reasonable. fully observed. X is fully observed, and consists of the above
In addition to the teacher reports of psychopathology, par- predictors. PXINT is auxiliary information that may improve
ents also reported on the child, using a parallel instrument. Such the estimation of the missing values of TXINT.
multiple informant reports are commonly collected in services
2.0
2.5
2.0

standard error estimate for MOMSTRS

1.5
parameter estimate for MOMSTRS

1.5

1.0
1.0

2 3 5 10 15 20 25 50

2 3 5 10 15 20 25 50 number of imputations

number of imputations
Figure 5. Distribution of Standard Error Estimates for MOMSTRS
Figure 4. Distribution of Estimates of MOMSTRS Parameter Using Parameter Using Different Number of Imputations (based on 50 replica-
Different Number of Imputations (based on 50 replications). tions).

The American Statistician, August 2001, Vol. 55, No. 3 251

 In this setting (ordinary linear regression with fully observed a Dell Dimension XPS B866, while MICE under R was run on
discrete predictors plus a continuous auxiliary variable), multi- a 400Mhz Sun workstation. Five imputations were used in the
ple imputation is straightforward, due to the monotone nature timing studies, though the timing was approximately linear in
of the missingness. We fit three models: one using just the com- the number of imputations. The models in SAS ran in 2 seconds
plete cases, one that used multiple imputation, but not using the (regression), 3 seconds (propensity), and 8 seconds (MCMC).
auxiliary information, and a model using multiple imputation The S-Plus missing data library using impGauss() required
with the auxiliary information (PXINT). 13 seconds. The models in MICE under S-Plus ran in 13 sec-
In this example, we noticed that with a small number of impu- onds (regression), and 23 seconds (PMM). MICE under R was
tations, some of the parameter estimates were not stable when we slightly slower (11 seconds and 82 seconds, respectively). These
repeated the imputation process with different starting seed val- results may be due to the use of interpreted code under R, which
ues. We explored the variability of multiple imputation for this was compiled to optimize certain functions under S-Plus, and
example by conducting a small simulation study. We repeated the use of different hardware platforms and operating systems.
sets of imputations 50 times for each of the following numbers of SOLAS timing was not comparable; though the script language
imputations (2,3,5,10,15,20,25,50), and described the variabil- automates the imputation process, fitting the regression model
ity of the estimates of the MOMSTRS parameter. Figures 4 and required manual intervention by the analyst. This was unpleas-
5 display boxplots of the results for the MOMSTRS parameter ant for the simulations described in this article, but should not be
and the estimated standard error for this parameter. The median a factor in more standard settings. The predictive mean match-
value for each set of imputations is approximately 1.5, but the ing imputation model required 15 seconds, and approximately
variability of the results for a particular imputation is quite high. 1 minute to describe and fit the regression models and combine
There is also increased variability in estimates of the standard er- the results.
ror for this parameter. For the simulations with five imputations, While these times are significantly slower than complete case
the empirical 95% confidence interval (CI) for the MOMSTRS methods, they are not prohibitive, even in a dataset of this size
parameter from the 50 repetitions was (1.12,1.88). The size of (2,501 observations with 13 variables). We concur with Rubin
this interval is similar to the magnitude of the standard error of (1996) that computational time to carry about multiple imputa-
this parameter, and could potentially affect the interpretation of tion is no longer a serious concern for all but the largest datasets.
the results. Even when 10 imputations were used, the empir- 6. DISCUSSION
ical 95% CI was still large: (1.26,1.73). In contrast, when 50
imputations were conducted, the empirical 95% CI was smaller Earlier, we introduced common concerns associated with
(1.40,1.60). missing data: (1) loss of efficiency; (2) complication in data
The variability of these estimates suggest that more imputa- handling and analysis; and (3) bias due to differences between
the observed and unobserved data. Multiple imputation is an an-
tions be used to decrease the variability in these models. This
alytic approach that addresses these problems. Compared to the
was true despite the fact that the fraction of missing information
complete case (CC) estimator, which discards partially observed
for this parameter was estimated to be in the range of 0.40.
subjects, multiple imputation methods may be more efficient (at
Table 3 displays the parameter estimates from these three
the cost of making assumptions regarding the missingness dis-
models for 50 imputations, using predictive mean matching in
tribution and the imputation model). Complications in data han-
SAS. In general, the parameter estimates are roughly consistent
dling and analysis have been greatly simplified by the existence
in direction and magnitude, with similar estimates of standard
of easy-to-use software packages. If the MAR assumption is ten-
errors. Other packages yielded similar results, though SOLAS
able, then multiple imputation may also provide less bias than
allows a maximum of 10 imputations.
other approaches if the imputation model is correctly specified.
Despite the large number of imputations, the time involved in
Although beyond the scope of this article, additional research is
fitting the multiple imputation models for this example was not needed to investigate the bias resulting from a poorly specified
prohibitive. SAS, SOLAS, and MICE under S-Plus were run on imputation model (e.g., using a normal distribution when the
possibly missing variable is Bernoulli).
Table 3. Parameter Estimates (and standard error) for Child
The previously described packages for multiple imputation,
Psychopathology Example Using Complete Case Analysis and Multiple
Imputation (using predictive mean matching, 50 imputations) though still evolving, are a useful addition to the analytic toolch-
est of practicing statisticians. They facilitate use of multiple im-
Parameter CC model MI (no auxiliary) MI (auxiliary) putation without having to resort to custom programming, te-
dious housekeeping, and additional calculations. Overall, these
INTERCEPT 46.95 (0.80) 46.92 (0.81) 47.08 (0.82) packages provide an easy to use environment for imputation,
LARGE 0.75 (0.73) 0.87 (0.74) 0.74 (0.71)
SMALL −1.51 (0.84) −1.39 (0.83) −1.58 (0.83)
and allow this technique to be applied in many settings.
SUBURB 0.65 (0.84) 0.65 (0.85) 0.47 (0.82) None of the packages is clearly superior, and there is a great
LOW 2.98 (0.97) 3.07 (0.93) 3.05 (0.98) deal of overlap of their support for multiple imputation. SOLAS
MIDDLE 0.81 (0.60) 0.85 (0.59) 0.83 (0.57) provides a nice interface in a package that is geared primarily
MOMSING −0.31 (0.79) −0.35 (0.81) −0.48 (0.81)
MOMSTRS 1.57 (0.74) 1.60 (0.76) 1.58 (0.74) toward imputation, but as a special-purpose package, it is limited
HLTHPRO 0.67 (0.54) 0.65 (0.53) 0.58 (0.53) in its general purpose statistical coverage. It is fine for linear re-
ACADPRO 3.40 (0.57) 3.41 (0.62) 3.34 (0.57) gression but does not currently facilitate imputation analysis for
CSEX −0.07 (0.54) −0.11 (0.54) −0.07 (0.53) nonlinear models. SAS provides a general purpose environment
FAMSTRS 0.55 (0.56) 0.50 (0.58) 0.56 (0.55)
for imputation, but does not provide as fine-grained control of
252 Statistical Computing Software Reviews
the imputation model. The S-Plus missing data library and MICE (1991b), Manual for the Teacher’s Report Form and 1991 Profile, Uni-
are somewhat slower than the other two packages, but provide versity of Vermont: Department of Psychiatry.
support for a wide variety of regression models. Allison, P. D. (2000), “Multiple Imputation for Missing Data: A Cautionary
Tale,” Sociological Methods and Research, 28, 301–309.
Extensive MCMC diagnostics are provided by both SAS and Barnard, J., and Meng, X. L. (1999), “Applications of Multiple Imputation in
MICE. Although these methods are quite attractive in modeling Medical Studies: From AIDS to NHANES,” Statistical Methods in Medical
nonmonotone missingness, they remain in large part a compli- Research, 8, 17–36.
cated black box whose output can be difficult to interpret. We Barnard, J., and Rubin, D. B. (1999), “Small Sample Degrees of Freedom With
Multiple Imputation,” Biometrika, 86, 948–955.
note that because both of the examples used to illustrate the
Boyle, M. H., Offord, D. R., Racine, Y. A., Fleming, J. E., Szatmari, P., and Links,
methods featured a monotone missing data pattern, these rou- P. S. (1993), “Predicting Substance Use in Early Adolescence Based on Parent
tines were not fully explored in this review. and Teacher Assessments of Childhood Psychiatric Disorder: Results From
All of the packages allow the number of imputations to be the Ontario Child Health Study Follow-up,” Journal of Child Psychology and
varied (though SOLAS limits this to 10). Since this number Psychiatry, 34, 535–544.
Dempster, A. P., Laird, N. M., and Rubin, D. B., (1977), “Maximum Likelihood
is under the control of the analyst, and because it affects the From Incomplete Data via the EM Algorithm,” Journal of the Royal Statistical
variability of results, it is important that the sensitivity of the Society, Series B, 39, 1–22.
results to the number of imputations be explored. Given the Diggle, P. J., Liang, K. Y., and Zeger, S. L. (1994), Analysis of Longitudinal
efficiency and speed of current computers, use of more than 10 Data, Oxford: Clarendon Press.
imputations can easily be incorporated into most analyses. Fienberg, S. E. (1994), “Conflicts Between the Needs for Access to Statistical
Information and Demands for Confidentiality,” Journal of Official Statistics,
The existing packages all allow the incorporation of auxiliary 10, 115–132.
information, which may improve the estimation of the impu- Fienberg, S. E., and Willenborg, L. C. R. J. (1998), “Introduction to the Special
tation model. Such models are often useful as an adjunct to a Issue: Disclosure Limitation Methods for Protecting the Confidentiality of
complete case analysis, as an informal test of sensitivity to as- Statistical Data,” Journal of Official Statistics, 14, 337–345.
sumptions required by complete data methods. These packages Fitzmaurice, G. M., Laird, N. M., and Zahner, G. E. P. (1996), “Multivariate
Logistic Models for Incomplete Binary Responses,” Journal of the American
(with the exception of MICE) do not allow the use of sensi- Statistical Association, 91, 99–108.
tivity analysis of nonignorable nonresponse. Such models, as Fitzmaurice, G. M., Laird, N. M., Zahner, G. E. P., and Daskalakis, C. (1995),
described in detail by Rubin (1987, chap. 6), assess the robust- “Bivariate Logistic Regression Analysis of Childhood Psychopathology Rat-
ness of inference to different assumptions about missingness. As ings Using Multiple Informants,” American Journal of Epidemiology, 142,
1194–1203.
various authors have noted, sensitivity analyses are quite com-
Gilks, W. R., Richardson, S., and Spiegelhalter, D. J. E. (1996), Markov Chain
plicated, but incorporation of simple nonignorable missingness Monte Carlo in Practice, New York: Chapman & Hall.
models would be a useful addition to these packages. MICE is Goldwasser, M. A., and Fitzmaurice, G. M. (2001), “Multivariate Linear Re-
the only package that allows the analyst to construct their own gression of Childhood Psychopathology using Multiple Informant Data,” In-
imputation functions, implementing particular nonignorable as- ternational Journal of Methods in Psychiatric Research, 10.
Horton, N. J., and Laird, N. M. (1999), “Maximum Likelihood Analysis of
sumptions, but these new functions require additional program- Generalized Linear Models with Missing Covariates,” Statistical Methods in
ming. Medical Research, 8, 37–50.
As a closing note, we offer the reminder that multiple im- Ibrahim, J. G. (1990), “Incomplete Data in Generalized Linear Models,” Journal
putation is not a panacea. Although it is a powerful and useful of the American Statistical Association, 85, 765–769.
tool applicable to many missing data settings, if not used care- King, G., Honaker, J., Joseph, A., and Scheve, K. (2001), “Analyzing Incomplete
Political Science Data: An Alternative Algorithm for Multiple Imputation,”
fully it is potentially dangerous. The existence of software that American Political Science Review, 95, 49–69.
facilitates its use requires the analyst to be careful about the Laird, N. M. (1988), “Missing Data in Longitudinal Studies,” Statistics in
verification of assumptions, the robustness of imputation mod- Medicine, 7, 305–315.
els, and the appropriateness of inferences. For more complicated Little, R. J. A. (1988), “Missing-Data Adjustments in Large Surveys,” Journal
models (e.g., longitudinal or clustered data), this is even more of Business and Economic Statistics, 6, 287–296.
(1992), “Regression With Missing X’s: A Review,” Journal of the Amer-
important. We reaffirm the sage advice of Barnard and Meng ican Statistical Association, 87, 1227–1237.
(1999): Little, R. J. A., and Rubin, D. B. (1987), Statistical Analysis With Missing Data,
New York: Wiley.
Cautions are needed, however, just as with any statistical methodology. It is clear Liu, M., Taylor, J. M. G., and Belin, T. R. (2000), “Multiple Imputation and
that if the imputation model is seriously flawed in terms of capturing the missing- Posterior Simulation for Multivariate Missing Data in Longitudinal Studies,”
data mechanism, then so will be any analysis based on such imputations. This Biometrics, 56, 1157–1163.
problem can be avoided by carefully investigating each specific application, by Meng, X.-L., and Rubin, D. B. (1992), “Performing Likelihood Ratio Tests With
making the best use of knowledge and data about the missing-data mechanism, Multiply-Imputed Datasets,” Biometrika, 79, 103–111.
and by performing various model checking procedures, in particular, posterior Offord, D. R., Boyle, M. H., Racine, Y., Szatmari, P., Fleming, J. E., Sanford, M.,
predictive checks. This is not an additional burden for using Rubin’s method, and Lipman, E. L. (1996), “Integrating Assessment Data from Multiple In-
but rather a fundamental requirement for any general method that attempts to formants,” Journal of the American Academy of Child Adolescent Psychiatry,
produce statistically and scientifically meaningful results in the presence of 35, 1078–1085.
incomplete data. Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994), “Estimation of Regression
Coefficients When Some Regressors are not Always Observed,” Journal of
the American Statistical Association, 89, 846–866.
[Received April 2001. Revised May 2001.]
(1995), “Analysis of Semiparametric Regression Models for Repeated
Outcomes in the Presence of Missing Data,” Journal of the American Statis-
tical Association, 90, 106–121.
REFERENCES
Robins, J. M., and Wang, N. (2000), “Inference for Imputation Estimators,”
Achenbach, T. M. (1991a), Manual for the Child Behavior Checklist/4-18 and Biometrika, 87, 113–124.
1991 Profile, University of Vermont: Department of Psychiatry. Rosenbaum, P. R., and Rubin, D. B. (1983), “Assessing Sensitivity to an Un-

The American Statistician, August 2001, Vol. 55, No. 3 253

 observed Binary Covariate in an Observational Study with Binary Outcome,” van Buuren, S., Boshuizen, H. C., and Knook, D. L. (1999), “Multiple Imputa-
Journal of the Royal Statistical Society, Series B, 45, 212–218. tion of Missing Blood Pressure Covariates in Survival Analysis,” Statistics in
Rubin, D. B. (1976), “Inference and Missing Data,” Biometrika, 63, 581–590. Medicine, 18, 681–694.
(1977), “Formalizing Subjective Notions About the Effect of Nonre- Wang, N., and Robins, J. M. (1998), “Large-Sample Theory for Parametric
spondents in Sample Surveys,” Journal of the American Statistical Associa- Multiple Imputation Procedures,” Biometrika, 85, 935–948.
tion, 72, 538–543. Zahner, G. E. P., and Daskalakis, C. (1997), “Factors Associated with Mental
(1987), Multiple Imputation for Nonresponse in Surveys, New York: Health, General Health and School-Based Service use for Psychopathology,”
Wiley. American Journal of Public Health, 87, 1440–1448.
Zahner, G. E. P., Jacobs, J. H., Freeman, D. H., and Trainor, K. (1993), “Rural-
(1996), “Multiple Imputation After 18+ Years,” Journal of the Ameri-
Urban Child Psychopathology in a Northeastern U.S. State: 1986–1989,”
can Statistical Association, 91, 473–489.
Journal of the American Academy of Child Adolescent Psychiatry, 32, 378–
Schafer, J. L. (1997), Analysis of Incomplete Multivariate Data, New York: 387.
Chapman & Hall.
Zahner, G. E. P., Pawelkiewicz, W., DeFrancesco, J. J., and Adnopoz, J. (1992),
(1999), “Multiple Imputation: A Primer,” Statistical Methods in Medical “Children’s Mental Health Service Needs and Utilization Patterns in an Urban
Research, 8, 3–15. Community,” Journal of the American Academy of Child Adolescent Psychi-
Tanner, M. A., and Wong, W. H. (1987), “The Calculation of Posterior Distribu- atry, 31, 951–960.
tions by Data Augmentation,” Journal of the American Statistical Association, Zaslavsky, A. M., and Horton, N. J. (1998), “Balancing Disclosure Risk Against
82, 528–540. the Loss of Nonpublication,” Journal of Official Statistics, 14, 411–419.

254 Statistical Computing Software Reviews