Frank E. Harrell, Jr.
Regression Modeling
Strategies
With Applications to
Linear Models,
Logistic Regression,
and Survival Analysis
With 141 Figures
Springer
Contents
Preface
vii
Typographical Conventions
1 Introduction
1.1
1.2
1.3
1.4
1.5
Hypothesis Testing, Estimation, and Prediction
Examples of Uses of Predictive Multivariable Modeling
Planning for Modeling
1.3.1
Emphasizing Continuous Variables
Choice of the Model
Further Reading
2 General Aspects of Fitting Regression Models
2.1 Notation for Multivariable Regression Models
2.2 Model Formulations
2.3 Interpreting Model Parameters
2.3.1 Nominal Predictors
2.3.2 Interactions
xxiii
1
1
3
4
6
6
8
11
11
12
13
14
14
xiv
Contents
2.3.3
2.4
Example: Inference for a Simple Model
Relaxing Linearity Assumption for Continuous Predictors
15
16
2.4.1
Simple Nonlinear Terms
16
2.4.2
Splines for Estimating Shape of Regression Function and Determining Predictor Transformations
18
2.4.3
Cubic Spline Functions
19
2.4.4
Restricted Cubic Splines
20
2.4.5
Choosing Number and Position of Knots
23
2.4.6
Nonparametric Regression
24
2.4.7
Advantages of Regression Splines over Other Methods . . . .
26
2.5
Recursive Partitioning: Tree-Based Models
26
2.6
Multiple Degree of Freedom Tests of Association
27
2.7
Assessment of Model Fit
29
2.7.1
Regression Assumptions
29
2.7.2
Modeling and Testing Complex Interactions
32
2.7.3
Fitting Ordinal Predictors
34
2.7.4
Distributional Assumptions
35
2.8
Further Reading
36
2.9
Problems
37
Missing Data
41
3.1
Types of Missing Data
41
3.2
Prelude to Modeling
42
3.3
Missing Values for Different Types of Response Variables
43
3.4
Problems with Simple Alternatives to Imputation
43
3.5
Strategies for Developing Imputation Algorithms
44
3.6
Single Conditional Mean Imputation
47
3.7
Multiple Imputation
47
3.8
Summary and Rough Guidelines
48
3.9
Further Reading
50
3.10
Problems
51
Multivariable Modeling Strategies
4.1
Prespecification of Predictor Complexity Without
Later Simplification
53
53
Contents
4.2
Checking Assumptions of Multiple Predictors Simultaneously
4.3
4.4
Variable Selection
Overfitting and Limits on Number of Predictors
56
60
4.5
Shrinkage
61
4.6
4.7
Collinearity
Data Reduction
64
66
4.8
4.9
4.10
4.11
...
5.1
5.2
5.3
5.4
5.5
56
4.7.1
4.7.2
4.7.3
4.7.4
Variable Clustering
Transformation and Scaling Variables Without Using Y . . .
Simultaneous Transformation and Imputation
Simple Scoring of Variable Clusters
66
67
69
70
4.7.5
4.7.6
Simplifying Cluster Scores
How Much Data Reduction Is Necessary?
72
73
Overly Influential Observations
74
Comparing Two Models
Summary: Possible Modeling Strategies
4.10.1 Developing Predictive Models
77
79
79
4.10.2
82
Developing Models for Effect Estimation
4.10.3 Developing Models for Hypothesis Testing
Further Reading
5 Resampling, Validating, Describing, and Simplifying the Model
xv
The Bootstrap
83
84
87
87
Model Validation
5.2.1
Introduction
5.2.2
Which Quantities Should Be Used in Validation?
90
90
91
5.2.3
5.2.4
5.2.5
91
93
94
Data-Splitting
Improvements on Data-Splitting: Resampling
Validation Using the Bootstrap
Describing the Fitted Model
Simplifying the Final Model by Approximating It
5.4.1
Difficulties Using Full Models
5.4.2
Approximating the Full Model
Further Reading
S-Plus Software
97
98
98
99
101
105
xvi
Contents
6.1
The S Modeling Language
106
6.2
User-Contributed Functions
107
6.3
The Design Library
108
6.4
Other Functions
119
6.5
Further Reading
120
Case Study in Least Squares Fitting and Interpretation
of a Linear Model
121
7.1
Descriptive Statistics
122
7.2
Spending Degrees of Freedom/Specifying Predictor Complexity . . 127
7.3
Fitting the Model Using Least Squares
128
7.4
Checking Distributional Assumptions
131
7.5
Checking Goodness of Fit
135
7.6
Overly Influential Observations
135
7.7
Test Statistics and Partial R
136
7.8
Interpreting the Model
137
7.9
Problems
142
Case Study in Imputation and Data Reduction
147
8.1
Data
147
8.2
How Many Parameters Can Be Estimated?
150
8.3
Variable Clustering
151
8.4
Single Imputation Using Constants or Recursive Partitioning
8.5
Transformation and Single Imputation Using transcan
157
8.6
Data Reduction Using Principal Components
160
8.7
Detailed Examination of Individual Transformations
168
8.8
Examination of Variable Clusters on Transformed Variables . . . .
169
8.9
Transformation Using Nonparametric Smoothers
170
8.10
Multiple Imputation
172
8.11
Further Reading
175
8.12
Problems
176
. . . 154
Overview of Maximum Likelihood Estimation
179
9.1
General NotionsSimple Cases
179
9.2
Hypothesis Tests
183
Contents
9.3
xvii
9.2.1
Likelihood Ratio Test
183
9.2.2
Wald Test
184
9.2.3
Score Test
184
9.2.4
Normal DistributionOne Sample
185
General Case
186
9.3.1
Global Test Statistics
187
9.3.2
Testing a Subset of the Parameters
187
9.3.3
Which Test Statistics to Use When
189
9.3.4
Example: BinomialComparing Two Proportions
190
9.4
Iterative ML Estimation
192
9.5
Robust Estimation of the Covariance Matrix
193
9.6
Wald, Score, and Likelihood-Based Confidence Intervals
194
9.7
Bootstrap Confidence Regions
195
9.8
Further Use of the Log Likelihood
202
9.9
9.8.1
Rating Two Models, Penalizing for Complexity
202
9.8.2
Testing Whether One Model Is Better than Another
9.8.3
Unitless Index of Predictive Ability
203
9.8.4
Unitless Index of Adequacy of a Subset of Predictors . . . .
205
....
Weighted Maximum Likelihood Estimation
203
206
9.10
Penalized Maximum Likelihood Estimation
207
9.11
Further Reading
210
9.12
Problems
212
10 Binary Logistic Regression
10.1
10.2
Model
215
215
10.1.1
Model Assumptions and Interpretation of Parameters . . . .
217
10.1.2
Odds Ratio, Risk Ratio, and Risk Difference
220
10.1.3
Detailed Example
221
10.1.4
Design Formulations
227
Estimation
228
10.2.1
Maximum Likelihood Estimates
228
10.2.2
Estimation of Odds Ratios and Probabilities
228
10.3
Test Statistics
229
10.4
Residuals
230
xviii
Contents
10.5
10.6
10.7
10.8
10.9
Assessment of Model Fit
Collinearity
Overly Influential Observations
Quantifying Predictive Ability
Validating the Fitted Model
10.10 Describing the Fitted Model
10.11
S-PLUS Functions
10.12 Further Reading
10.13 Problems
11 Logistic Model Case Study 1: Predicting Cause of Death
11.1 Preparation for Modeling
11.2
230
244
245
247
249
253
257
264
265
269
269
11.4
11.5
Regression on Principal Components, Cluster Scores, and Pretransformations
Fit and Diagnostics for a Full Model, and Interpreting Pretransformations
Describing Results Using a Reduced Model
Approximating the Full Model Using Recursive Partitioning . . . .
276
285
291
11.6
Validating the Reduced Model
294
11.3
12 Logistic Model Case Study 2: Survival of Titanic Passengers
271
299
12.1
Descriptive Statistics
12.2
12.3
12.4
12.5
12.6
12.7
12.8
Exploring Trends with Nonparametric Regression
305
Binary Logistic Model With Casewise Deletion of Missing Values . 305
Examining Missing Data Patterns
'
312
Single Conditional Mean Imputation
316
Multiple Imputation
320
Summarizing the Fitted Model
322
Problems
326
13 Ordinal Logistic Regression
13.1
13.2
13.3
Background
Ordinality Assumption
Proportional Odds Model
13.3.1 Model
13.3.2 Assumptions and Interpretation of Parameters
300
331
331
332
333
333
333
Contents
xix
13.3.3
Estimation
334
13.3.4
Residuals
334
13.3.5
Assessment of Model Fit
335
13.3.6
Quantifying Predictive Ability
335
13.3.7
Validating the Fitted Model
337
13.3.8
S-PLUS Functions
337
13.4
13.5
13.6
Continuation Ratio Model
13.4.1 Model
13.4.2 Assumptions and Interpretation of Parameters
13.4.3 Estimation
338
338
338
339
13.4.4
Residuals
339
13.4.5
Assessment of Model Fit
339
13.4.6
Extended CR Model
339
13.4.7
13.4.8
Role of Penalization in Extended CR Model
Validating the Fitted Model
340
341
13.4.9
S-PLUS Functions
341
Further Reading
Problems
342
342
14 Case Study in Ordinal Regression, Data Reduction, and Penalization
345
14.1
14.2
Response Variable
Variable Clustering
14.3
14.4
Developing Cluster Summary Scores
Assessing Ordinality of Y for each X, and Unadjusted Checking of
PO and CR Assumptions
A Tentative Full Proportional Odds Model
Residual Plots
Graphical Assessment of Fit of CR Model
Extended Continuation Ratio Model
Penalized Estimation
Using Approximations to Simplify the Model
Validating the Model
Summary
Further Reading
14.5
14.6
14.7
14.8
14.9
14.10
14.11
14.12
14.13
. 346
347
349
351
352
355
357
357
359
364
367
369
371
xx
Contents
14.14 Problems
15 Models Using Nonparametric Transformations of X and Y
15.1 Background
15.2 Generalized Additive Models
15.3 Nonparametric Estimation of ^-Transformation
15.4 Obtaining Estimates on the Original Scale
15.5 S-PLUS Functions
15.6 Case Study
371
375
375
376
376
377
378
379
16 Introduction to Survival Analysis
389
16.1 Background
389
16.2 Censoring, Delayed Entry, and Truncation
391
16.3 Notation, Survival, and Hazard Functions
392
16.4 Homogeneous Failure Time Distributions
398
16.5 Nonparametric Estimation of 5 and A
400
16.5.1 Kaplan-Meier Estimator
400
16.5.2 Altschuler-Nelson Estimator
403
16.6 Analysis of Multiple Endpoints
404
16.6.1 Competing Risks
404
16.6.2 Competing Dependent Risks
405
16.6.3 State Transitions and Multiple Types of Nonfatal Events . . 406
16.6.4 Joint Analysis of Time and Severity of an Event
407
16.6.5 Analysis of Multiple Events
407
16.7 S-PLUS Functions
408
16.8 Further Reading
410
16.9 Problems
411
17 Parametric Survival Models
17.1 Homogeneous Models (No Predictors)
17.1.1 Specific Models
17.1.2 Estimation
17.1.3 Assessment of Model Fit
17.2 Parametric Proportional Hazards Models
17.2.1 Model
413
413
413
414
416
417
417
Contents
17.3
17.4
xxi
17.2.2
Model Assumptions and Interpretation of Parameters . . . .
418
17.2.3
Hazard Ratio, Risk Ratio, and Risk Difference
419
17.2.4
Specific Models
421
17.2.5
Estimation
422
17.2.6
Assessment of Model Fit
Accelerated Failure Time Models
423
426
17.3.1
Model
426
17.3.2
Model Assumptions and Interpretation of Parameters . . . .
427
17.3.3
Specific Models
427
17.3.4
Estimation
428
17.3.5
Residuals
429
17.3.6
Assessment of Model Fit
430
17.3.7
Validating the Fitted Model
434
Buckley-James Regression Model
435
17.5
Design Formulations
435
17.6
Test Statistics
435
17.7
Quantifying Predictive Ability
436
17.8
S-PLUS Functions
436
17.9
Further Reading
17.10 Problems
441
441
18 Case Study in Parametric Survival Modeling and Model Approximation
443
18.1 Descriptive Statistics
443
18.2
Checking Adequacy of Log-Normal Accelerated Failure Time Model 448
18.3
18.4
Summarizing the Fitted Model
Internal Validation of the Fitted Model Using the Bootstrap . . . .
18.5
Approximating the Full Model
458
18.6
Problems
464
19 Cox Proportional Hazards Regression Model
19.1
Model
454
454
465
465
19.1.1
Preliminaries
465
19.1.2
Model Definition
466
19.1.3
Estimation of/?
466
xxii
Contents
19.1.4 Model Assumptions and Interpretation of Parameters . . . .
19.1.5 Example
19.1.6 Design Formulations
19.1.7 Extending the Model by Stratification
19.2 Estimation of Survival Probability and Secondary Parameters . . .
468
468
470
470
472
19.3
474
Test Statistics
19.4
19.5
Residuals
476
Assessment of Model Fit
476
19.5.1 Regression Assumptions
477
19.5.2 Proportional Hazards Assumption
483
19.6 What to Do When PH Fails
489
19.7 Collinearity
491
19.8 Overly Influential Observations
492
19.9 Quantifying Predictive Ability
492
19.10 Validating the Fitted Model
493
19.10.1 Validation of Model Calibration
493
19.10.2 Validation of Discrimination and Other Statistical Indexes . 494
19.11 Describing the Fitted Model
496
19.12
S-PLUS Functions
19.13. Further Reading
499
506
20 Case Study in Cox Regression
509
20.1 Choosing the Number of Parameters and Fitting the Model . . . . 509
20.2
20.3
20.4
20.5
20.6
20.7
Checking Proportional Hazards
Testing Interactions
Describing Predictor Effects
Validating the Model
Presenting the Model
Problems
513
516
517
517
519
522
Appendix
523
References
527
Index
559
