BASIC SCIENCE REVIEW

The Risk of Determining Risk with Multivariable Models

John Concato, MD, MS, MPH; Alvan R. Feinstein, MD, MS; and Theodore R. Holford, PhD

• Purpose: To review the principles of multivariable A l t h o u g h most physicians have received no instruction
analysis and to examine the application of multivariable in multivariable methods of statistical analysis, the
statistical methods in general medical literature. methods now commonly appear in medical literature.
• Data Sources: A computer-assisted search of arti- The results of multivariable analyses are often ex-
cles in The Lancet and The New England Journal of pressed in statements such as, "When other risk factors
Medicine identified 451 publications containing multi- are controlled, a decrease of 5 units in substance X
variable methods from 1985 through 1989. A random reduced disease by 10%," or "After adjustment for age
sample of 60 articles that used the two most common and stage of disease, treatment with procedure Y re-
methods—logistic regression or proportional hazards duced mortality by 2 5 % . "
analysis—was selected for more intensive review. Our purpose in the current research was to note the
• Data Extraction: During review of the 60 randomly frequency with which multivariable analyses now ap-
selected articles, the focus was on generally accepted pear in general medical journals, to identify some com-
methodologic guidelines that can prevent problems mon problems and desirable precautions in the analy-
affecting the accuracy and interpretation of multivari- ses, and to determine how well the challenges are being
able analytic results. met. The investigation also provided a framework for a
• Results: From 1985 to 1989, the relative frequency brief review—intended for clinical readers—of com-
of multivariable statistical methods increased annually monly used multivariable statistical methods.
from about 10% to 18% among all articles in the two
journals. In 44 (73%) of 60 articles using logistic or
General Principles
proportional hazards regression, risk estimates were
quantified for individual variables ("risk factors"). Vio- Format of Multivariable Analysis
lations and omissions of methodologic guidelines in
these 44 articles included overfitting of data; no test of In the types of multivariable analyses discussed here,
conformity of variables to a linear gradient; no mention the mathematical expressions described in Appendix 1
of pertinent checks for proportional hazards; no report are used to relate two or more independent variables to
of testing for interactions between independent vari- an outcome or dependent variable. In those expres-
ables; and unspecified coding or selection of indepen- sions, a linear regression coefficient indicates the impact
dent variables. These problems would make the re- of each independent variable on the outcome in the
ported results potentially inaccurate, misleading, or context of (or "adjusting for") all other variables. The
difficult to interpret. values of the regression coefficients are obtained as the
• Conclusions: The findings suggest a need for im- best mathematical fit for the specified model, although
provement in the reporting and perhaps conducting of the selected model and the multivariable analysis may
multivariable analyses in medical research. or may not provide a good absolute fit for the data.
The four main multivariable methods in medical liter-
ature have many mathematical similarities but differ in
the expression and format of the outcome expressed as
the dependent variable:
1. In multiple linear regression, the outcome variable
is a continuous quantity, such as blood pressure in
millimeters of mercury or sodium concentration in mil-
liequivalents per liter.
2. In multiple logistic regression, the observed out-
come variable is usually a binary event, such as alive
versus dead or case versus control. The event occurs at
a fixed point in time, such as mortality 1 year after
surgery.
3. In discriminant function analysis, the outcome
variable is a category or group to which a subject be-
longs. For example, patients may be classified as having
obstructive, restrictive, vascular, or "other" forms of
pulmonary dysfunction. The analytic results are often
converted to a " s c o r e " used to classify observations
Annals of Internal Medicine. 1993;118:201-210. into one of the categorical groups. For only two cate-
gories (such as healthy or diseased), this form of mul-
From Yale University School of Medicine, New Haven, Con- tivariable analysis produces results similar to logistic
necticut. For current author addresses, see end of text. regression.

© 1993 American College of Physicians 201

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 4. In proportional hazards regression, which is also consuming to obtain and difficult to assimilate. In such
known as Cox regression (1), the outcome variable is situations, multivariable analysis can be used as an ini-
the duration of time to the occurrence of a binary "fail- tial screening process. "Important" variables can be
ure" event during a follow-up period of observation. suggested by the screening, with thresholds of quanti-
The most common event in such analyses is death, but tative and statistical importance (significance) deter-
other failures can also be modeled. For example, each mined by the researcher; more detailed analyses can
person's final state can be classified as either dead at a then be performed. Software programs typically have
specified time or as "censored" if lost to follow-up or "default values" that can be modified for levels of
alive at the end of the study period. The results of a statistical significance, but quantitative significance is
Cox model may be considered as an instantaneous in- usually ignored by the computer and must be recog-
cidence rate for the failure event. nized and specified by the analyst.
The technical details of all four methods have been
described elsewhere (2-6), and the current review is Creating Risk Scores
limited to the way two of the methods—multiple logistic The multiple variables that seem important may be
regression and proportional hazards analysis—are used assigned simple rating scores and combined into a single
in medical literature. These multivariable methods have risk score used for predicting outcomes of individual
become particularly popular because binary dependent patients. The choice of these ratings can be aided by
variables, such as survival state, are frequently used in the regression coefficients found in the multivariable
clinical research. (Although the term "multivariate" is analysis. A well-known simple, single combination of
often used interchangeably with "multivariable," the arbitrary ratings is the Apgar score, created by Virginia
former does not apply to the most common medical Apgar, who assigned integer ratings of 0, 1, or 2 to each
situations, where a single outcome variable is studied.) of five variables in the assessment of a newborn infant
(10). Although Apgar's initial decisions were made us-
Purposes of Multivariable Analysis ing clinical judgment and were not related to infants'
outcome, a multivariable analysis might have been used
Multivariable methods can be used for at least five to suggest suitable "weights" for the ratings.
major purposes.
Quantifying Risk of Individual Variables
Bivariate Confirmation In many instances, particularly in studies of risk fac-
Before multivariable analyses are initiated, most in- tors, the impact of an individual variable is quantified
vestigators have previously done simple bivariate anal- for its specific effect among the other independent vari-
yses in which each independent variable is evaluated, ables. For example, proportional hazards analysis was
one at a time, for its association with the outcome used to determine that "a 19% reduction in coronary
variable. (Because of the one-at-a-time examination, the heart disease risk was associated with each decrement
analyses are often called "univariate," although two of 8% of total plasma cholesterol" (11), adjusting for
variables are being associated.) After the important in- age, systolic blood pressure, cigarette smoking, and
dependent variables have been identified in the simple other factors. In these situations, the regression coeffi-
analyses, the multivariable examination is usually done cients found in the multivariable analysis are converted
to confirm that the independent variables retain their to expressions of relative risks or hazards (with Cox
importance in the simultaneous context of the other regression) or odds ratios (with multiple logistic regres-
variables. For example, the effect of smoking on car- sion), indicating the individual variable's effect on the
diovascular mortality can be evaluated along with the outcome.
concomitant effects of age, systolic blood pressure, se-
rum cholesterol level, and serum triglyceride value. If
the initial bivariate impact of smoking is changed in the Assumptions and Limitations
multivariable context, specific relationships can be in-
In the first three purposes just cited, the multivariable
vestigated further.
analyses are used mainly to confirm results that have
Multivariable Confirmation already been documented with other methods or to
A second purpose of the multivariable methods is to suggest important variables that will be evaluated later
confirm the results of non-regression analyses such as in greater detail. In the fourth purpose, the risk score
adjustment by cross-stratification (7) or standardization will combine the different variables and will demon-
(8). For example, data on survival after prostatectomy, strate the impact of the combinations rather than each
independent of type of surgery, can be examined for variable alone. Although the assumptions and limita-
subjects cross-stratified in categorical groups of age and tions of multivariable models are important, they may
severity of illness. After suitable composite strata are be considered as having a secondary role when the
formed, long-term mortality can then be examined in combined scores are formed.
each stratum for transurethral compared with open In the fifth purpose, when individual variables are
prostatectomy, and the results can be checked with a examined to determine the magnitude of their impact or
multivariable regression model (9). risk, the estimates will vary with the structure of the
mathematical model and the coding of variables. The
Screening assumptions and limitations of the multivariable meth-
With large administrative data bases, the total num- ods then become especially important for ensuring ac-
ber of variables may make bivariate analyses too time- curate results and valid interpretations.

202 1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 Methods
Selection of Articles
The BRS Colleague complete text computer data base (12)
was used to identify the use of the four cited multivariable
methods in articles from The Lancet and The New England
Journal of Medicine from 1985 to 1989. The search was limited
to these two journals because they include a variety of general
clinical topics, have large circulations, and have complete texts
(not just keywords) available in the data base.
The computer search was designed to focus on original and
special articles while excluding editorials, reviews, letters, and
so forth. The search process included the text but not the
references of published articles. The main terms used in the
search included "multiple linear," "discriminant function,"
"logistic," "proportional hazards," or " C o x . " Each main
term was paired with "regression," "analysis," "function,"
"model," and "method." The accuracy of the computer
search was checked with a manual inspection of articles pub-
lished in each journal during a 6-month period.
After the initial search confirmed that the use of logistic
regression and proportional hazards analysis was particularly
frequent, these two methods were selected for further review.
For this purpose, a random, stratified sample of 60 articles Figure 1. Nonproportional "hazards." Crossing of survival
(28-87) was selected to provide three articles for each of the curves for groups A and B, violating the assumption of pro-
two methods, for each of the two journals, for each of the 5 portional hazards.
years 1985 through 1989. With information about each usage
excerpted and recorded on standardized forms, the 60 articles
were reviewed to document the purpose for which each mul-
tivariable method was used and to evaluate the application and of outcome events. Underfitting occurs when the power for
reporting of results. detecting important relationships is low, and important vari-
If regression coefficients, relative risks, or odds ratios were ables may be omitted from a model. For example, consider a
listed for individual variables, the multivariable method was longitudinal study of 200 persons in whom 2 develop lung
classified as having risk quantification as a purpose, and math- cancer after 10 years of follow-up. In this situation, the asso-
ematical details were checked. The other multivariable analytic ciation of cigarette smoking and lung cancer would probably be
purposes were noted and classified according to the foregoing undetectable, although analysis with a larger number of out-
taxonomy, but the articles were not evaluated further. come events might identify a relationship between the indepen-
dent variable and the outcome. Thus, a study with a relative
M a n a g e m e n t of P r o b l e m s a n d P r e c a u t i o n s paucity of outcome events may have misleading results due to
overfitting or underfitting.
Various guidelines (2-6, 13) have been suggested to encour-
age the appropriate execution, interpretation, and reporting of Nonconformity to a Linear Gradient
multivariable methods. In logistic and Cox regression, inatten- When a linear regression coefficient is estimated for a vari-
tion to the guidelines can cause unreliable results in estimates able X, the implication is that regardless of the value of X, a
of risk when the impact of an individual variable is quantified unit change in X should always have the same effect on the
from its regression coefficient. In examining the published re- outcome. If the independent variable is binary, then the re-
sults for logistic and Cox regression, we checked to see how gression coefficient represents only a single gradient as the
the investigators had managed and reported the six problems variable " m o v e s " from being absent to present. With ranked
and precautions that follow. We reviewed the relevant data variables, however, several or many gradients may occur as
when available or accepted even a brief statement by the the variable moves through its series of ordinal categories or
authors that a potential problem was evaluated. continuous measurements. The value of the regression coeffi-
cient is calculated to be accurate as the average effect of X,
Overfitting but the result will be misleading if X has different effects in
The risk estimates may be unreliable if the multivariable data different zones. The implication may be particularly misleading
contain too few outcome events relative to the number of if the average value of X does not occur in any of the zones.
independent variables. For example, consider a cohort study of For example, the impact of left ventricular ejection fraction
1000 persons in which 5 deaths occur during 6 months of on mortality is not linear: A decrease of 10% from 30% to 20%
follow-up. The factors associated with death are determined carries greater risk than a decrease from 50% to 40%. A single
from only five observations, although numerous baseline char- risk estimate may therefore be misleading unless conformity to
acteristics might be included in a multivariable analysis of the assumption of a linear gradient is evaluated. Various meth-
6-month mortality. Because outcome events are sparse, the ods are available to check the assumption of a linear gradient,
resulting regression coefficients for individual variables may including assessing the impact of each variable separately in
not be trustworthy; they may represent spurious associations, zones of the ranked data (2).
or the effects may be estimated with low precision. (Because
the analysis depends on the smaller number of the two com- Violation of Proportional Hazards
plementary outcome events, the problem is not corrected by Another potential problem occurs in the Cox regression
focusing on the survival in 995 persons.) method, in which the risk or "hazard" of an independent
Consequently, a large number of outcome events is needed if variable is assumed to be constantly proportional (the relative
many independent variables are included in the analysis. In risk does not change with time). The problem can be illustrated
general, the results of models having fewer than 10 outcome by considering survival curves with a binary variable that
events per independent variable are thought to have question- identifies patients in groups A and B, representing two forms
able accuracy (13, 14), and the usual tests of statistical signif- of treatment. If the hazard is proportional, the survival curve
icance may be invalid. Large confidence intervals associated of one group will not cross the survival curve of the other
with individual risk estimates may indicate an overfitted model group.
under these circumstances. When crossover (as shown in Figure 1) or other nonpropor-
A counterpart problem, underfitting, is also due to a scarcity tional survival patterns occur, the corresponding risk estimates

1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3 203

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 may be inaccurate (3, 13, 14). For example, the effects of an "change-in-estimate" selection procedure has a forward direc-
early survival advantage followed by late excess mortality may tion but allows variables to be deleted after they are entered in
cancel for group A and group B, so that the Cox method may a model if a subsequent variable improves overall prediction
indicate that group membership has no effect. Therefore, when (16). The "principal components" method involves a reduction
the Cox method is used, the independent variables should be in the number of candidate variables before the modeling pro-
evaluated for adherence to the assumption of proportional haz- cess begins (13).
ards. If nonproportional hazards are found, the Cox method Although intended to be used as tools in exploring data, the
can be applied as an analysis stratified by the offending vari- various automated procedures may not be recognized and fully
able or by including time-dependent covariates (15). understood by researchers. None of the automated multivari-
able methods uses clinical judgments or consider biological
No Report of Tests for Interactions sensibility in the analysis. The fundamental issue for readers to
An interaction occurs between independent variables if the understand is that the final model will depend on the chosen
impact of one variable on the outcome event depends on the selection process and that variables may be retained or ex-
level of another variable. For example, consider a logistic or cluded merely by mathematical details of the analysis.
Cox regression model with two independent binary variables:
smoking and use of oral contraceptives, each given possible Additional Issues
values of 0 (no exposure) or 1 (exposure). If interactions are
not considered, the regression coefficient for oral contracep- Three other issues that are also important considerations in
tives represents the impact of oral contraceptive use on the the application of multivariable analyses are not easily evalu-
outcome event for both levels of smoking. If oral contraceptive ated from published reports.
use and cigarette smoking have a significant interaction, how-
ever, the impact of oral contraceptives depends on whether Collinearity of Variables
exposure to smoking occurs. Without attention to interaction, A problem occurs if independent variables have a high cor-
the oral contraceptive coefficient would offer a misleading relation with one another. For example, measurements of ven-
quantitative estimate of the impact of oral contraceptive use. tricular ejection fraction and ventricular contractility may con-
Interactions may be checked either because suspicions are tain redundant information regarding the risk for mortality
raised by clinical judgment before the analysis is done or by a (unless the relationship between the two variables is itself
specific statistical examination whenever multivariate meth- under study). The results of a multivariable analysis including
ods are applied. The regression methods do not automatically both variables might identify one or the other as important but
examine interactions, which can be evaluated only if appropri- is not likely to include both variables if they are highly corre-
ate interaction terms are explicitly included in the model. (A lated. Furthermore, with a high correlation the quantitative
potential problem arises because of the increased opportunity risk estimates for each variable may be imprecise. Although
for overfitting when interaction terms are added to a model.) software packages include tests to examine the data for colline-
Despite the absence of a universal rule dictating appropriate arity, investigators have noted that " . . .it is possible for
tests for interactions in all circumstances, readers of published variables to pass these tests and have the program run but
results will not know whether any tests for interactions were yield output that is clearly nonsense" (17). As a precaution,
conducted unless the testing is mentioned in the report. the most clinically relevant (among similar) variables can be
chosen for inclusion in a multivariable analysis, or the princi-
Unspecified Coding of Variables pal components selection method can be used to choose among
The apparent effect of an independent variable will depend the variables.
on the corresponding units of measurement and coding for that
variable. For example, the regression coefficient for the impact Influential Observations
of age on long-term mortality will be different if age is coded in A small number of observations for individual subjects can
1-year increments, in 10-year intervals ( . . .; 50 to 59 years; 60 have a substantial effect on the final analytic results if their
to 69 years; . . .), or dichotomously as < 65 versus > 65 corresponding values are distinctly different from all others.
years. If the values of regression coefficients are cited without Such "outliers" produce a problem similar to the example of
concomitant citation of the units of coding for independent inadvertently including a child's age in the calculation of mean
variables, then readers will be unable to interpret the actual age for patients seen in a geriatric clinic. Unlike this simple
magnitude or effect of the risk estimates. situation, however, the problem of "influential" observations
Independent variables expressed in ordinal or binary catego- is often obscure in a multivariable context, where the relation-
ries must receive arbitrary numerical codings for a multivari- ship among numerous independent variables determines outlier
able analysis. For example, binary variables can be coded (2) status.
as - 1 / + 1 ("marginal method") or 0/1 ("partial method"). Ac- Although mathematical methods exist for dealing with influ-
cording to the selected code, the magnitude of the regression ential observations (mostly by deleting observations from the
coefficient will vary by a factor of 2 in logistic and Cox re- data set), the optimal scientific approach to this situation can-
gression. In addition, ordinal variables can be coded with in- not be specified in advance, particularly because outliers may
teger values or with "dummy" variables (2) that compare all accurately reflect an important biologic relationship rather than
other categories to a single reference category. Because the representing a mathematical aberration. The best precaution
particular measurement or coding scheme can have substantial for this potential problem is to insist on high-quality data for
effects on the numerical values and interpretation of the re- the multivariable analysis (to avoid spurious outliers) or to
gression coefficients, readers should always be notified of how analyze the data by other methods such as stratified analysis.
the coding was used in a multivariable analysis.
Validation of a Model
Unspecified Selection of Variables A multivariable analysis assigns coefficients to independent
The choice of independent variables included in a final mul- variables selected as important predictors of outcome. As with
tivariable analysis is not a simple task. Although candidate all statistical models, the results require validation to ensure
variables may be chosen from previous research results or protection against unrecognized problems and limitations.
from clinical experience, automated algorithms exist for select- Common methods for validating models include 1) performing
ing among variables thought to have possible prognostic value. a test analysis on a subsample of patients followed by a sub-
Most software programs offer "forward" or "backward" se- sequent validation analysis on the remaining patients; 2) re-
lection of variables. These procedures include or delete vari- peating the analysis on an independent sample of patients; 3)
ables one at a time until a specified threshold of statistical using "jackknife" or "bootstrap" procedures (18) in which the
significance is met (16). For example, a forward model might same analysis is performed multiple times on a series of sub-
include all variables for which the associated regression coef- sets from the same data set to investigate the stability of
ficient has P < 0.05. Yet another process investigates "all coefficients and predictive ability of the model.
possible subsets" of candidate variables (16). An alternative A related issue is the mathematical fit of the final model.

204 1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 Table 1. Multivariable Methods: Number of Annea ranees in The Lancet and The New England Journal of Medicine

Multivariate Method Year

1985 1986 1987 1988 1989

Multiple logistic regression, n 32 23 43 40 56

Proportional hazards analysis, n 19 24 35 30 40
Multiple linear regression, n 10 21 19 17 22
Discriminant function analysis, n 2 7 4 5 2
Total, n 63 75 101 92 120
Number of original articles, n 661 661 661 644 660
Frequency of any of the four methods, % 10 11 15 14 18

Indexes of "goodness-of-fit" evaluate how effectively the cal- garded as reasonably accurate frequencies of the cited
culated model fits the actual data for estimating the outcome multivariable methods in these two journals.
variable. Although various indexes have been developed (2), a
consensus is lacking even among statisticians about which in- Because the logistic regression and proportional haz-
dex is most appropriate. (Details of the methods and discus- ards methods were particularly common, they were the
sions are beyond the scope of this review.) As a fundamental focus of further evaluation in the random sample of 60
principle, including additional independent variables in a model articles.
will enhance mathematical goodness of fit but can cause prob-
lems such as overfitting of data or collinearity of variables.
Authors' Purpose in Using Multivariable Analysis
In 44 (73%) of the 60 publications using logistic and
Results
Cox regression, the multivariable methods were applied
Frequency of Use to quantify risk estimates reported as regression coeffi-
cients, odds ratios, or relative risks for individual vari-
Table 1 shows results for the four multivariable meth- ables. For example, in a study of prenatal X-ray expo-
ods in the two journals from 1985 to 1989. The number sure and childhood cancer in twins (31), the relative risk
of annual citations has increased steadily, although the for cancer, adjusting for birth weight, was 2.4 for ex-
total number of pertinent articles has remained essen- posed compared with nonexposed children; and when
tially constant. The frequency of the multivariable type A behavior was related to outcome of coronary
methods has thus increased from 10% to 18% over the heart disease (79), the mortality for Type A persons,
5-year period; and in 1989, one of the four methods after adjustment for other risk variables, was 58% that
appeared on average at least once per week in each of Type B persons.
journal. The remaining 16 (27%) of the 60 articles used mul-
The manual inspection of articles from July to De- tivariable methods as follows: Thirteen studies con-
cember 1989 determined that the computer search had firmed the results of other forms of analysis (such as
correctly identified "original articles," "special arti- simple bivariate analysis [40] or a Mantel-Haenszel
cles," "medical intelligence," "medical progress," and analysis [62]); one study screened data for important
"case records" in The New England Journal of Medi- variables (to identify risk factors for gastric cancer after
cine (n = 191); and "original articles," "preliminary gastric surgery for benign disease [65]); one report cre-
communications," and "methods and devices" in The ated a risk score (to predict relapse among patients with
Lancet (n = 135). Eleven additional publications in The testicular teratoma [72]); and one investigation checked
Lancet, however, were mistakenly identified as "origi- for interactions only (in a report of tamoxifen therapy
nal," including two articles on medicine and the law, a for breast cancer [58]).
letter to the editor, a correction, and so forth. Thus, the
computer search identified a proportionate excess of 11
Problems in Reporting and Application
of 326, or 3%, of the desired articles for the checked
period in both journals. The error rate seemed too small The pertinent statistical "package" or program—such
to warrant corrections for the reported frequency as SAS (19) or BMDP (20)—used to perform a multi-
counts. variable analysis should be reported to the reader. Cit-
The manual inspection was also used to evaluate the ing the information is analogous to a laboratory re-
computer citations of the multivariable methods. In searcher indicating the particular experimental protocol
three articles, the search term appeared in the discus- used for physiologic measurements. Yet, the pertinent
sion (such as describing a logistic analysis done in an- program was mentioned in only 17 (39%) of the 44
other publication) rather than in the study methods. articles reviewed.
Several articles using major modifications of classical In addition to this general consideration, the six cited
multivariable methods were not identified, but these principles were evaluated in the 44 studies where logis-
techniques were not an intended subject of our investi- tic regression and proportional hazards analysis meth-
gation. Furthermore, in five articles the authors did not ods were applied for quantifying risk of individual vari-
distinguish between simple linear regression and multi- ables. Potential problems involving collinear variables,
ple linear regression when reporting the use of "linear influential observations, and model validation were not
regression." Thus, the results in Table 1 can be re- evaluated in the articles under review.

1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3 205

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 Overfitting tions sometimes discussed interactions that were sus-
The criterion for overfitting data was violated in 19 pected before the analysis (for example, aspirin and
(42%) of the 44 studies. For example, in an investiga- sulfinpyrazone interacting on the risk of cardiac death
tion (49) of coronary restenosis after dietary supplemen- among patients with unstable angina [63]). In other in-
tation with n-3 fatty acids, 11 independent variables stances, the investigators evaluated pairwise interac-
were included in a logistic model containing 29 outcome tions of all independent variables. Although testing for
events for 85 patients. The ratio of 2.6 (29/11) events interactions may or may not have been important for
per independent variable is much smaller than the sug- the clinical and statistical context, the reader would at
gested ratio of 10, and the overfitting could affect the least be aware of the testing that was done. In the
quantitative assertion that "therapy reduced the likeli- remaining articles, interactions may have been assessed
hood of restenosis by 77%." Of note, the 95% confi- during the research, but in the absence of a published
dence interval for the effect of the intervention was statement, readers evaluating the results have no assur-
quite wide (10% to 92% reduction in restenosis), con- ance that the assessments were even considered.
sistent with overfitting.
Unspecified Coding of Variables
Nonconformity to a Linear Gradient The coding classification of pertinent independent
The criterion for nonconformity to a linear gradient variables was not reported in 37 (84%) of the 44 arti-
did not apply to 30 articles where the analyses used cles. Such omissions prevent the reader from interpret-
only binary independent variables. Of the 14 articles ing the quantitative results. In the remaining 7 (16%)
with ranked independent variables, however, 4 (29%) articles, the coding scheme was described in the meth-
gave no indication of checking for nonconformity to a ods section, in the table of results for the multivariable
linear gradient. For example, when calcium intake (78) model, or in an appendix. The description occupied
seemed to have a protective effect on incidence of hip only a modest amount of space in the publication.
fracture, the main result was reported as a relative
risk = 0.6 "per 198 mg calcium/1000 kcal." (The value Unspecified Selection of Variables
of 198 mg was chosen because it was the standard The selection process for choosing independent vari-
deviation of calcium intake.) The result implies that an ables was described in 38 (86%) of publications. This
increase in calcium intake of about 200 mg from any finding suggests that most investigators (or their statis-
level lowers the risk for hip fracture by 40%, because a tical consultants) are aware of the impact of the selec-
relative risk of 0.6 is a proportionate decrease of 0.4. A tion mechanism on the results of multivariable analyses,
concomitant graphic display of the data, however, sug- and that the particular mechanism is considered impor-
gested otherwise: The difference in hip fracture rates tant enough to mention.
was minimal for persons with " l o w " compared with
" m i d " calcium intake but was substantial for persons Discussion
with " m i d " compared with "high" calcium intake. The
40% risk reduction, representing an average risk among Multivariable methods have become increasingly pop-
all patients, could therefore be due mainly to the very ular forms of data analysis in medical research; and two
low risk of hip fracture in subjects with "high" calcium of the methods—logistic regression and proportional
intake. hazards analysis—appeared in 15% (96 of 660) of arti-
In the remaining 10 studies, ranked variables were cles published in two leading general medical journals
appropriately divided and checked in several ordinal during 1989. These two regression methods were fre-
zones of data delineated by the investigators. Individual quently used to identify the effect of individual variables
risk estimates were then calculated separately for each in a multivariable context and to offer a quantitative
of the zones. In one study, for example, the risk for estimate of risk for the individual effect. Nevertheless,
ulcerative keratitis was calculated for duration of con- certain important precautions were often omitted or not
tact lens use divided in zones of 1 day, 2 days to 1 reported when the methods were applied.
week, more than 1 but not more than 2 weeks, and The six problems that were reviewed (and three more
more than 2 weeks (57); in another study the risk for discussed) in this paper are listed in Table 2. The cri-
cataract formation was determined separately for ultra- teria represent our minimum standards for the conduct
violet radiation divided in quartiles of exposure (51). and reporting of research using multivariable analysis.
We do not expect everyone to agree with these stan-
Violation of Proportional Hazards dards, but a consensus cannot be reached unless the
A check of the assumption of proportional hazards issues are elaborated and investigated.
over time was not mentioned in 17 (81%) of the 21 The problem of unstable risk estimates and inappro-
studies using Cox regression. The proportionality crite- priate P values produced by overfitted data can be
ria may have been violated in these instances, but quan- prevented by ensuring an adequate number of outcome
titative risk estimates were nevertheless reported. The events. Although inadequacy cannot be formally tested
accuracy of the estimates is therefore uncertain. in a manner analogous to power calculations for type 2
error (21), a low (less than 10:1) ratio of outcome events
No Report of Tests for Interactions to independent variables makes the risk estimates un-
Tests for possible interactions between independent certain. In the earlier cited example in which a protec-
variables were not mentioned in 32 (73%) of the 44 tive effect of dietary n-3 fatty acids (49) was based on
articles. When this criterion was satisfied, the publica- an overfitted logistic regression model, the findings

206 1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 Table 2. Problems and Issues in the Application and Reporting of Logistic Regression and Proportional Hazards
Analysis

Problem or Issue Description

Problem
Overfitting of data Fewer than 10 outcome events per independent variable in the model
Nonconformity to linear gradient Nonconstant impact of variables in different zones of ranked data
Nonproportional risk Violation of assumption of proportional hazard function over time (in the
proportional hazards method)
No report of tests for interactions Check not mentioned for interactions between independent variables
Unspecified coding of variables Unknown classification or codings for independent variables
Unspecified selection of variables Unknown method of selecting among candidate independent variables
Issue
Collinear variables Independent variables with high correlation to each other
Influential observations "Outlier" observations that have a substantial effect on results
Validation of model Separate method of confirming analytic results

were not confirmed in a subsequent, similar investiga- sponding to < 200 mg/dL versus > 200 mg/dL). Rather
tion (22). Although the inconsistent results were attrib- than using a dichotomous classification, continuous
uted to various elements of study design, limitations of variables may be converted into an array of ordinal
the data analysis itself were not considered. zones or transformed into " d u m m y " variables (2) ap-
The problems of overfitted models have been re- propriate for the clinical context of each analysis. Al-
ported (2,13, 23) in the statistical literature but are not though the ideal number of zones cannot be specified in
widely recognized. The key issue in the overfitting is an advance and often requires judgment, clinicians should
ample number of outcome events, not just a large sam- be aware of this issue in multivariable modeling.
ple size. When numerous variables are included in an The problem of nonproportionality in Cox regression
attempt to "control" or "adjust" the data, accuracy of can be avoided if hazard functions are suitably checked
results can be threatened by overfitting or by other and reported. Although criteria for identifying " s e v e r e "
mechanisms (24). The number of variables selected for violations are lacking, a rigorous scientific analysis
analysis should therefore be parsimonious, based on should include evaluating methodologic assumptions
clinical sensibility and suitable data quality. and reporting the results. Techniques such as checking
In checking the problem of nonlinearity when ranked for proportional hazards using logarithmic graphs (15)
variables are used directly, the analyst can compare the may not be familiar to all readers but, when described,
observed and the multivariable model's predicted values would indicate that the proportional hazards assumption
for the outcome over the range of each variable. A had been evaluated.
single risk estimate is inappropriate if the pattern of The interaction problem is illustrated by the associa-
" e r r o r s " is nonrandom. For example, if arterial carbon tion of asbestos exposure and cigarette smoking with
dioxide tension (PC02) is included as a predictor in a lung cancer, initially thought (25) to interact: The risk
multivariable analysis of death from chronic obstructive for asbestos-exposed persons who also smoked ciga-
lung disease, the corresponding linear regression coef- rettes was substantially greater than the risk anticipated
ficient will represent the average impact of PC02 on merely from combining the risks calculated individually
mortality. If the actual mortality substantially differs for asbestos exposure and for cigarette smoking. Al-
from predicted mortality for "high" values of P C 0 2 , though subsequent data (26) did not confirm these re-
then the analysis will incorrectly estimate the true risk sults, such interactions represent another threat to the
for such patients. constancy implied by the reporting of regression coef-
In "nonlinear" circumstances the risks should be ficients. A variable whose impact is linear when acting
quantitatively estimated not as a single value but in alone may be nonlinear when acting jointly with other
zones or categories of the data. Although checking for a variables.
linear gradient is not a trivial exercise, a common The fifth principle requires an explicit statement of
method available in software packages involves visual the way the independent variables are analytically clas-
inspection of appropriate data. Alternatively, the ana- sified and coded. This statement can be easily incorpo-
lyst can use other forms of multivariable analysis, such rated in the text, tables, or appendix to allow the reader
as cross-stratification (7), to evaluate whether the vari- to interpret the quantitative results. Such disclosure is
ables conform to a linear gradient. obviously crucial for interpreting the numerical magni-
In the papers under review, the problem of noncon- tude of a cited risk factor.
formity to a linear gradient was frequently avoided by Similarly, a statement indicating the method of select-
the strategy of using binary independent variables—a ing among candidate independent variables is desirable.
tactic found in 30 of the 44 pertinent articles. The true Readers should be aware that some variables may have
impact of continuous or ordinal variables, however, minimal impact on the outcome despite achieving "sta-
may be masked when two binary zones are created. For tistical significance," whereas other variables that fail
example, the "J-shape" relationship of serum choles- to achieve the threshold of " P < 0.05" may still have a
terol and mortality cannot be described by binary zones substantial effect on the outcome. (This distinction be-
such as < 5.20 mmol/L versus > 5.20 mmol/L (corre- tween quantitative and statistical significance occurs in

1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3 207

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 all forms of analysis [27] and is not unique to multivari- Grant Support: Dr. Concato was a Postdoctoral Fellow in the Robert
Wood Johnson Clinical Scholars Program, supported by the Department
a t e procedures.) of Veterans Affairs, when this study was conducted.
The remaining principles relating to collinear vari-
ables, influential observations, and model validation re- Requests for Reprints: John Concato, MD, Medical Service/111, West
Haven Veterans Affairs Medical Center, 950 Campbell Avenue, West
quire raw data for evaluation. Readers of the published Haven, CT 06516.
reports therefore cannot easily determine if a problem Current Author Addresses: Dr. Concato: Yale University School of
exists. Investigators who are aware of the principles, Medicine, Department of Medicine and Medical Service/111, West Ha-
ven Veterans Affairs Medical Center, 950 Campbell Avenue, West Ha-
however, can publish descriptive statements to indicate ven, CT 06516.
that suitable precautions were taken during the analysis. Dr. Feinstein: Yale University School of Medicine, Departments of
Medicine and Epidemiology and Public Health, P.O. Box 3333, 333
Because accurate and understandable results are re- Cedar Street, New Haven, CT 06510.
quired in communicating medical research, the findings Dr. Holford: Yale University School of Medicine, Department of Epi-
of this review suggest a need for substantial improve- demiology and Public Health, P.O. Box 3333, 60 College Street, New
Haven, CT 06510.
ments in reporting and perhaps in conducting multivari-
able analyses.
References
[Note: The first 27 citations indicate sources of comment or other
Appendix 1. Mathematical Expressions of Multivariate pertinent references. Citations 28 through 87 indicate the 60 articles
Analysis that were selected for special review. They are cited by method and in
chronological order for both journals.]
Multivariate analysis relates independent variables 1. Cox DR. Regression methods and life tables (with discussion). J R
Stat Soc B. 1972;34:187-220.
Xj, X 2 , X 3 , . . . X n to an outcome variable via a model 2. Hosmer DW, Lemeshow S. Applied Logistic Regression. New York:
expressed as a combination: G = b 0 + b ^ + b 2 X 2 + Wiley; 1989.
b 3 X 3 -I- . . . +e, where G is a function arranged in 3. Kalbfleisch JD, Prentice RL. The Statistical Analysis of Failure
Time Data. New York: Wiley; 1980.
various mathematical forms (see below); bj is a regres- 4. Lee ET. Statistical Methods for Survival Data Analysis. Belmont,
sion coefficient indicating the impact of each Xj variable California: Lifetime Learning Publications; 1980.
5. Kleinbaum DG, Kupper LL, Muller KE. Applied Regression Anal-
on the outcome; and b 0 is an intercept term, which is ysis and Other Multivariable Methods. Boston: PWS-Kent Publish-
usually included in the model. If a particular bj = 0, ing Co.; 1988.
then variable Xj has no impact on the outcome; a pos- 6. Breslow NE, Day NE. Statistical Methods in Cancer Research, v I
(1980) and v II (1987). Lyon: International Agency for Research on
itive value of bj indicates that higher values of Xj are Cancer.
associated with an increase in the outcome expressed as 7. Feinstein AR. Prognostic stratification. In: Clinical Biostatistics. St.
G; and negative values have the reverse effect. A ran- Louis: CV Mosby; 1977:385-443.
8. Armitage P, Berry G. Statistical Methods in Medical Research.
dom variable e is an "error" term representing the Boston: Blackwell; 1987:399-405.
increment by which any individual G value deviates 9. Concato J, Horwitz RI, Feinstein AR, Elmore JG, Schiff SF. Prob-
from the calculated value of G. lems of comorbidity in mortality after prostatectomy. JAMA. 1992;
267:1077-82.
The function G is arranged in different mathematical 10. Apgar V. A proposal for a new method of evaluation of the new-
forms for each multivariable method. born infant. Curr Res Anesth. 1953;32:260-7.
11. Lipid Research Clinics Program Investigators. The lipid research
1. Multiple linear regression: G = outcome variable clinics coronary primary prevention trial results. JAMA. 1984;251:
2. Multiple logistic regression: G = In [P/(l-P)]; 365-74.
12. BRS Colleague Users Manual. Latham, New York: BRS Informa-
where P is probability of the outcome event occurring tion Technologies; 1988.
and G is the "logit" or log odds of the outcome. 13. Harrell FE Jr, Lee KL, Matchar DB, Reichert TA. Regression mod-
3. Discriminant function analysis: G = relative prob- els for prognostic prediction: advantages, problems, and suggested
solutions. Cancer Treat Rep. 1985; 69:1071-7.
ability of each category. 14. Harrell F. The PHGLM procedure. In: SUGI Supplemental Library
4. Proportional hazards analysis: mortality or inci- User's Guide. Cary, North Carolina: SAS Institute Inc.; 1983:267-94.
dence rate as "hazard function" H(t,x) = H 0 (t)e G , 15. Christensen E. Multivariate survival analysis using Cox's regression
model. Hepatology. 1987;7:1346-58.
where t is elapsed time after a starting point (zero time) 16. Greenland S. Modeling and variable selection in epidemiologic anal-
and Ho(t) is an underlying hazard when all Xj are zero. ysis. Am J Pub Health. 1989;79:340-9.
17. Hosmer DW, Lemeshow S. Applied Logistic Regression. New York:
The following comments can be added about the two Wiley; 1989:131.
main methods under review. 18. Efron B, Gong G. A leisurely look at the bootstrap, the jackknife,
Logistic regression may be done as a "conditional" and cross-validation. American Statistician. 1983;37:36-48.
19. SAS Institute Inc. SAS Guide for Personal Computers. Version
analysis for studies with small numbers of observations 6.03. Cary, North Carolina: SAS Institute Inc.; 1988.
within strata, or as an "unconditional" analysis for 20. Dixon WJ. BMDP Statistical Software. University of California
unstratified studies or studies with large numbers of Press; 1987.
21. Freiman JA, Chalmers TC, Smith H Jr, Kuebler RR. The impor-
observations within strata. The method has also been tance of beta, the type II error and sample size in the design and
adapted for modeling outcome variables having three or interpretation of the randomized control trial. N Engl J Med. 1978;
more ordinal categories. 299:690-4.
22. Reis GJ, Boucher TM, Sipperly ME, Silverman DI, McCabe CH,
Proportional hazard analysis is sometimes reported in Bairn DS, et al. Randomised trial of fish oil for prevention of
terms of the hazard function, indicating the probability restenosis after coronary angioplasty. Lancet. 1989;2:177-81.
23. Harrell FE Jr, Lee KL, Califf RM, Pryor DB, Rosati RA. Regression
of a binary outcome event occurring at an instant in modelling strategies for improved prognostic prediction. Stat Med.
time, conditional on the subject surviving up to that 1984;3:143-52.
instant. For pragmatic usage, survival is estimated as 24. Walter SD, Feinstein AR, Wells CK. A comparison of multivariable
mathematical models in predicting survival—II. Statistical selection
S(t) e x p ( G ) , where S(t) is the summary survival curve for of prognostic variables. J Clin Epidemiol. 1990;43:349-59.
the group (possibly hypothetical) where G = 0. 25. Selikoff U, Hammond EC, Churg J. Asbestos exposure, smoking,
and neoplasia. JAMA. 1968;204:106-12.
Presented in part at the annual meeting of the American Federation for 26. Rom WN; ed. Environmental and Occupational Medicine. Boston:
Clinical Research, Baltimore, Maryland, 6 May 1991. Little, Brown; 1983:157-82.

208 1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 27. Feinstein AR. Clinical Epidemiology. Philadelphia: W.B.Saunders; haemophiliacs with human immunodeficiency virus infection. Lan-
1985:396-400. cet. 1989;2:63-6.
Logistic Regression 54. Wahrendorf J, Chang-Claude J, Liang QS, Rei YG, Munoz N, Cre-
28. Rahman M, Rahaman MM, Wojtyniak B, Aziz KM. Impact of spi M, et al. Precursor lesions of oesophageal cancer in young
environmental sanitation and crowding on infant mortality in rural people in a high-risk population in China. Lancet. 1989;2:1239-41.
Bangladesh. Lancet. 1985;2:28-31. 55. Sheinfeld J, Schaefifer AJ, Cordon-Cardo C, Rogatko A, Fair WR.
29. Ludlam CA, Tucker J, Steel CM, Tedder RS, Cheingsong-Popov R, Association of the Lewis blood-group phenotype with recurrent
Weiss RA, et al. Human T-lymphotropic virus type III (HTLV-III) urinary tract infections in women. N Engl J Med. 1989;320:773-7.
infection in seronegative haemophiliacs after transfusion of factor 56. Hayes EB, Matte TD, O'Brien TR, McKinley TW, Logsdon GS, Rose
VIII. Lancet. 1985;2:233-6. JB, et al. Large community outbreak of cryptosporidiosis due to
30. Stadel BV, Rubin GL, Webster LA, Schlesselman JJ, Wingo PA. contamination of a filtered public water supply. N Engl J Med.
Oral contraceptives and breast cancer in young women. Lancet. 1989;320:1372-6.
1985;2:970-3. 57. Schein OD, Glynn RJ, Poggio EC, Seddon JM, Kenyon KR, and the
31. Harvey EB, Boice JD Jr, Honeyman M, Flannery J T . Prenatal X-ray Microbial Keratitis Study Group. The relative risk of ulcerative
exposure and childhood cancer in twins. N Engl J Med. 1985;312: keratitis among users of daily-wear and extended-wear contact
541-5. lenses. N Engl J Med. 1989;321:773-8.
32. Lesko SM, Rosenberg L, Kaufman DW, Helmrich SP, Miller DR, Proportional Hazards Analysis
Strom B, et al. Cigarette smoking and the risk of endometrial 58. Controlled trial of tamoxifen as single adjuvant agent in manage-
cancer. N Engl J Med. 1985;313:593-6. ment of early breast cancer. Analysis at six years by Nolvadex
33. Hurwitz ES, Barrett MJ, Bregman D, Gunn WJ, Schonberger LB, Adjuvant Trial Organisation. Lancet. 1985;1:836-40.
Fairweather WR, et al. Public Health Service study on Reye's 59. Haybittle JL, Hayhoe FG, Easterling MJ, Jelliffe AM, Bennett MH,
syndrome and medications. N Engl J Med. 1985;313:849-57. Vaughan Hudson G, et al. Review of British National Lymphoma
34. Francis CW, Marder VJ, Evarts CM. Lower risk of thromboembolic Investigation studies of Hodgkin's disease and development of prog-
disease after total hip replacement with non-cemented than with nostic index. Lancet. 1985;1:967-72.
cemented prostheses. Lancet. 1986;1:769-71. 60. Harris KR, Digard N, Gosling DC, Tate DG, Campbell MJ, Gardner
35. Knutsson A, Akerstedt T, Jonsson BG, Orth-Gomer K. Increased B, et al. Azathioprine and cyclosporin: different tissue matching
risk of ischaemic heart disease in shift workers. Lancet. 1986;2:89- criteria needed? Lancet. 1985;2:802-5.
61. Geresh BJ, Kronmal RA, Schaff HV, Frye RL, Ryan TJ, Mock MB,
92.
et al. Comparison of coronary artery bypass surgery and medical
36. Witteman JC, Kok FJ, van Saase JL, Valkenburg HA. Aortic calci-
therapy in patients 65 years of age or older. N Engl J Med. 1985;
fication as a predictor of cardiovascular mortality. Lancet. 1986;2:
313:217-24.
1120-2.
62. The EC/IC Bypass Study Group. Failure of extracranial-intracranial
37. Kreiss JK, Koech D, Plummer FA, Holmes KK, Lightfoote M, Piot arterial bypass to reduce the risk of ischemic stroke. N Engl J Med.
P, et al. AIDS virus infection in Nairobi prostitutes. N Engl J Med. 1985;313:1191-200.
1986;314:414-8. 63. Cairns JA, Gent M, Singer J, Finnie KJ, Froggatt GM, Holder DA,
38. Bowden RA, Sayers M, Flournoy N, Newton B, Banaji M, Thomas et al. Aspirin, sulfinpyrazone, or both in unstable angina. N Engl J
ED, et al. Cytomegalovirus immune globulin and seronegative blood Med. 1985;313:1369-75.
products to prevent primary cytomegalovirus infection after marrow 64. Holmberg LH, Tabar L, Adami HO, Bergstrom R. Survival in breast
transplantation. N Engl J Med. 1986;314:1006-10. cancer diagnosed between mammographic screening examinations.
39. Freedman DS, Srinivasan SR, Shear CL, Franklin FA, Webber LS, Lancet. 1986;2:27-30.
Berenson GS. The relation of apolipoproteins A-I and B in children 65. Viste A, Bjornestad E, Opheim P, Skarstein A, Thunold J, Hartveit
to parental myocardial infarction. N Engl J Med. 1986;315:721-6. F, et al. Risk of carcinoma following gastric operations for benign
40. Cohen J, Moore RH, Al Hashimi S, Jones L, Apperley J F , Aber VR. disease. Lancet. 1986;2:502-5.
Antibody titres to a rough-mutant strain of Escherichia coli in 66. Amery A, Birkenhager W, Brixko R, Bulpitt C, Clement D, Deruyt-
patients undergoing allogeneic bone-marrow transplantation. Lan- tere M, et al. Efficacy of antihypertensive drug treatment according
cet. 1987;1:8-11. to age, sex, blood pressure, and previous cardiovascular disease in
41. McManus C, Richards P. Choice and ordering of medical school patients over the age of 60. Lancet. 1986;2:589-92.
applications: cause for concern. Lancet. 1987;2:33-5. 67. Brain Resuscitation Clinical Trial I Study Group. Randomized clin-
42. Krauss RM, Lindgren FT, Williams FT, Kelsey SF, Brensike J, ical study of thiopental loading in comatose survivors of cardiac
Vranizan K, et al. Intermediate-density lipoproteins and progression arrest. N Engl J Med. 1986;314:397-403.
of coronary artery disease in hypercholesterolaemic men. Lancet. 68. Storb R, Deeg HJ, Whitehead J, Applebaum F, Beatty P, Bensinger
1987;2:62-6. W, et al. Methotrexate and cyclosporine compared with cyclospo-
43. Mandel EM, Rockette HE, Bluestone CD, Paradise JL, Nozza RJ. r i n alone for prophylaxis of acute graft versus host disease after
Efficacy of amoxicillin with and without decongestant-antihistamine marrow transplantation for leukemia. N Engl J Med. 1986;314:729-
for otitis media with effusion in children. N Engl J Med. 1987;316: 35.
432-7. 69. Ettinger B, Tang A, Citron JT, Livermore B, Williams T. Random-
44. Louik C, Mitchell AA, Werler MM, Hanson JW, Shapiro S. Mater- ized trial of allopurinol in the prevention of calcium oxalate calculi.
nal exposure to spermicides in relation to certain birth defects. N Engl J Med. 1986;315:1386-9.
N Engl J Med. 1987;317:474-8. 70. Flegel KM, Shipley MJ, Rose G. Risk of stroke in non-rheumatic
45. Daling JR, Weiss NS, Hislop TG, Maden C, Coates RJ, Sherman KJ, atrial fibrillation. Lancet. 1987;1:526-9.
et al. Sexual practices, sexually transmitted diseases, and the inci- 71. Riou G, Barrois M, Le MG, George M, Le Doussal V, Haie C.
dence of anal cancer. N Engl J Med. 1987;317:973-7. C-myc proto-oncogene expression and prognosis in early carcinoma
46. National Heart Foundation of Australia Coronary Thrombolysis of the uterine cervix. Lancet. 1987;1:761-3.
Group. Coronary thrombolysis and myocardial salvage by tissue 72. Freedman LS, Parkinson MC, Jones WG, Oliver RT, Peckham MJ,
plasminogen activator given up to 4 hours after onset of myocardial Read G, et al. Histopathology in the prediction of relapse of patients
infarction. Lancet. 1988;1:203-8. with stage I testicular teratoma treated by orchidectomy alone.
47. Marwick TH, Case C, Siskind V, Woodhouse SP. Adverse effect of Lancet. 1987;2:294-8.
early high-dose adrenaline on outcome of ventricular fibrillation. 73. Philip T, Armitage J O , Spitzer G, Chauvin F, Jagannath S, Cahn
Lancet. 1988;2:66-8. JY, et al. High-dose therapy and autologous bone marrow trans-
48. Yudkin JS, Forrest RD, Jackson CA. Microalbuminuria as predictor plantation after failure of conventional chemotherapy in adults with
of vascular disease in non-diabetic subjects. Lancet. 1988;2:530-3. intermediate-grade or high-grade non-Hodgkin's lymphoma. N Engl
49. Dehmer GJ, Popma J J , van den Berg EK, Eichhorn EJ, Prewitt JB, J Med. 1987;316:1493-8.
Campbell WB, et al. Reduction in the rate of early restenosis after 74. Willett WC, Green A, Stampfer MJ, Speizer FE, Colditz GA, Rosner
coronary angioplasty by a diet supplemented with n-3 fatty acids. B, et al. Relative and absolute excess risks of coronary heart dis-
N Engl J Med. 1988;319:733-40. ease among women who smoke cigarettes. N Engl J Med. 1987;
50. Ron E, Modan B, Boice JD Jr, Alfandary E, Stovall M, Chetrit A, et 317:1303-9.
al. Tumors of the brain and nervous system after radiotherapy in 75. Coates A, Gebski V, Bishop J F , Jeal PN, Woods RL, Snyder R, et al.
childhood. N Engl J Med. 1988;319:1033-9. Improving the quality of life during chemotherapy for advanced
51. Taylor HR, West SK, Rosenthal FS, Munoz B, Newland HS, Abbey breast cancer. N Engl J Med. 1987;317:1490-5.
H, et al. Effect of ultraviolet radiation on cataract formation. 76. AIMS Trial Study Group. Effect of intravenous APSAC on mortal-
N Engl J Med. 1988;319:1429-33. ity after acute myocardial infarction: preliminary report of a place-
52. Price WH, Morris SW, Kitchin AH, Wenham PR, Burgon PR, bo-controlled clinical trial. Lancet. 1988;1:545-9.
Donald PM. DNA restriction fragment length polymorphisms as 77. Gill M, McCarthy M, Murrells T, Silcocks P. Chemotherapy for the
markers of familial coronary heart disease. Lancet. 1989;1:1407-11. primary treatment of osteosarcoma: population effectiveness over
53. Webster A, Lee CA, Cook DG, Grundy JE, Emery VC, Kernoff PB, 20 years. Lancet. 1988;1:689-92.
et al. Cytomegalovirus infection and progression towards AIDS in 78. Holbrook TL, Barrett-Connor E, Wingard DL. Dietary calcium and

1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3 209

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015

 risk of hip fracture: 14-year prospective population study. Lancet. in infancy and death from ischaemic heart disease. Lancet. 1989;2:
1988;2:1046-9. 577-80.
79. Ragland DR, Brand RJ. Type A behavior and mortality from cor- 84. Spyratos F, Maudelonde T, Bouillet JP, Brunet M, Defrenne A,
onary heart disease. N Engl J Med. 1988;318:65-9. Andrieu C, et al. Cathepsin D: an independent prognostic factor for
80. Gelmers HJ, Goiter K, de Weerdt CJ, Wiezer HJ. A controlled trial metastasis of breast cancer. Lancet. 1989;2:1115-8.
of nimodipine in acute ischemic stroke. N Engl J Med. 1988;318: 85. Fogelfeld L, Wiviott MB, Shore-Freed man E, Blend M, Bekerman C,
203-7. Pinsky S, et al. Recurrence of thyroid nodules after surgical removal
in patients irradiated in childhood for benign conditions. N Engl J
81. Ekelund LG, Haskell WL, Johnson JL, Whaley FS, Criqui MH,
Med. 1989;320:835-40.
Sheps DS. Physical fitness as a predictor of cardiovascular mortality
86. Justice AC, Feinstein AR, Wells CK. A new prognostic staging
in asymptomatic North American men. N Engl J Med. 1988;319: system for the acquired immunodeficiency syndrome. N Engl J
1379-84. Med. 1989;320:1388-93.
82. International Bone Marrow Transplant Registry. Effect of metho- 87. Scott GB, Hutto C, Makuch RW, Mastrucci MT, O'Connor T,
trexate on relapse after bone-marrow transplantation for acute lym- Mitchell CD, et al. Survival in children with perinatally acquired
phoblastic leukaemia. Lancet. 1989;1:535-7. human immunodeficiency virus type 1 infection. N Engl J Med.
83. Barker DJ, Winter PD, Osmond C, Margetts B, Simmons SJ. Weight 1989;321:1791-6.

As to your method of work, I have a single bit of advice, which I give with the
earnest conviction of its paramount influence in any success which may have
attended my efforts in life—Take no thought for the morrow. Live neither in the
past nor in the future, but let each day's work absorb your entire energies, and
satisfy your widest ambition. This was the singular but very wise answer which
Cromwell gave to Bellevire—"No one rises so high as he who knows not wither
he is going," and there is much truth in it. The student who is worrying about the
future, anxious over the examinations, doubting his fitness for the profession, is
certain not to do so well as the man who cares for nothing but the matter in hand,
and who knows not wither he is going!

William Osier, to his students

The Practical Cogitator
Dell, New York, 1962

Submissions from readers are welcomed. If the quotation is published, the sender's name will be
acknowledged. Please include a complete citation, as done for any reference.—The Editors

210 1 February 1993 • Annals of Internal Medicine • Volume 118 • Number 3

Downloaded From: http://annals.org/ by a Penn State University Hershey User on 05/24/2015