0% found this document useful (0 votes)
55 views16 pages

Handbook of Statistical Methods For Randomized Controlled Trials, 1st Edition Final Version Download

The 'Handbook of Statistical Methods for Randomized Controlled Trials' provides a comprehensive overview of statistical methodologies applicable to randomized controlled trials, edited by experts in the field. It covers a range of topics including binary and ordinal outcomes, continuous outcomes, and time-to-event data, integrating theory with practical examples. This first edition, published in 2021, is part of the Chapman & Hall/CRC Handbooks of Modern Statistical Methods series.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
55 views16 pages

Handbook of Statistical Methods For Randomized Controlled Trials, 1st Edition Final Version Download

The 'Handbook of Statistical Methods for Randomized Controlled Trials' provides a comprehensive overview of statistical methodologies applicable to randomized controlled trials, edited by experts in the field. It covers a range of topics including binary and ordinal outcomes, continuous outcomes, and time-to-event data, integrating theory with practical examples. This first edition, published in 2021, is part of the Chapman & Hall/CRC Handbooks of Modern Statistical Methods series.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

Handbook of Statistical Methods for Randomized Controlled

Trials, 1st Edition

Visit the link below to download the full version of this book:

https://medidownload.com/product/handbook-of-statistical-methods-for-randomized-
controlled-trials-1st-edition/

Click Download Now


Chapman & Hall/CRC
Handbooks of Modern Statistical Methods

Series Editor
Garrett Fitzmaurice, Department of Biostatistics, Harvard School of Public Health, Boston, MA,
U.S.A.

The objective of the series is to provide high-quality volumes covering the state-of-the-art in
the theory and applications of statistical methodology. The books in the series are thoroughly
edited and present comprehensive, coherent, and unified summaries of specific methodologi-
cal topics from statistics. The chapters are written by the leading researchers in the field and
present a good balance of theory and application through a synthesis of the key methodological
developments and examples and case studies using real data.

Published Titles
Handbook of Neuroimaging Data Analysis
Hernando Ombao, Martin Lindquist, Wesley Thompson, and John Aston
Handbook of Statistical Methods and Analyses in Sports
Jim Albert, Mark E. Glickman, Tim B. Swartz, and Ruud H. Koning
Handbook of Methods for Designing, Monitoring, and Analyzing Dose-Finding Trials
John O’Quigley, Alexia Iasonos, and Björn Bornkamp
Handbook of Quantile Regression
Roger Koenker, Victor Chernozhukov, Xuming He, and Limin Peng
Handbook of Statistical Methods for Case-Control Studies
Ørnulf Borgan, Norman Breslow, Nilanjan Chatterjee, Mitchell H. Gail, Alastair Scott, and
Chris J. Wild
Handbook of Environmental and Ecological Statistics
Alan E. Gelfand, Montserrat Fuentes, Jennifer A. Hoeting, and Richard L. Smith

Handbook of Approximate Bayesian Computation


Scott A. Sisson, Yanan Fan, and Mark Beaumont
Handbook of Graphical Models
Marloes Maathuis, Mathias Drton, Steffen Lauritzen, and Martin Wainwright
Handbook of Mixture Analysis
Sylvia Frühwirth-Schnatter, Gilles Celeux, and Christian P. Robert

Handbook of Infectious Disease Data Analysis


Leonhard Held, Niel Hens, Philip O’Neill, and Jacco Walllinga
Handbook of Forensic Statistics
David L. Banks, Karen Kafadar, David H. Kaye, and Maria Tackett

Handbook of Meta-Analysis
Christopher H. Schmid, Theo Stijnen, and Ian White
Handbook of Statistical Methods for Randomized Controlled Trials
KyungMann Kim, Frank Bretz, Ying Kuen K. Cheung, and Lisa Hampson
For more information about this series, please visit: https://www.crcpress.com/Chapman--HallCRC-Handbooks-of-
Modern-Statistical-Methods/book-series/CHHANMODSTA
Handbook of Statistical
Methods for Randomized
Controlled Trials

KyungMann Kim
Frank Bretz
Ying Kuen K. Cheung
Lisa V. Hampson
First edition published 2021
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742

and by CRC Press


2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN

© 2021 Taylor & Francis Group, LLC

The right of KyungMann Kim, Frank Bretz, Ying Kuen K. Cheung, and Lisa Hampson
to be identified as the author(s) of the editorial material, and of the authors for their individual chapters, has been as-
serted in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.

Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot as-
sume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have
attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders
if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please
write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or
utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including pho-
tocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission
from the publishers.

For permission to photocopy or use material electronically from this work, access www.copyright.com or contact the
Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. For works that are
not available on CCC please contact [email protected]

Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for iden-
tification and explanation without intent to infringe.

Library of Congress Cataloging-in-Publication Data

Names: Kim, KyungMann, editor.


Title: Handbook of statistical methods for randomized controlled trials /
edited by KyungMann Kim, Frank Bretz, Ying Kuen K. Cheung, and Lisa
Hampson.
Description: First edition. | Boca Raton : CRC Press, 2021. | Series:
Chapman & Hall/CRC handbooks of modern statistical methods | Includes
bibliographical references and index.
Identifiers: LCCN 2020055560 (print) | LCCN 2020055561 (ebook) | ISBN
9781498714624 (hardback) | ISBN 9781315119694 (ebook)
Subjects: LCSH: Clinical trials--Statistical methods--Handbooks, manuals,
etc.
Classification: LCC R853.C55 H365 2021 (print) | LCC R853.C55 (ebook) |
DDC 610.72/4--dc23
LC record available at https://lccn.loc.gov/2020055560
LC ebook record available at https://lccn.loc.gov/2020055561

ISBN: 9781498714624 (hbk)


ISBN: 9781032009100 (pbk)
ISBN: 9781315119694 (ebk)

Typeset in CMR10 font


by KnowledgeWorks Global Ltd.
To our families
for their love and support
Contents

Preface xix

List of Figures xxi

List of Tables xxv

Contributors xxxi

I Introduction to Randomized Controlled Trials 1


1 Introduction 3
KyungMann Kim
1.1 Historical Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Statistical Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Organization of the Handbook . . . . . . . . . . . . . . . . . . . . . . . . 5
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

II Analytic Methods for Randomized Controlled Trials 9


2 Binary and Ordinal Outcomes 11
Garrett M. Fitzmaurice and Stuart R. Lipsitz
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Analysis of 2 × 2 Contingency Tables . . . . . . . . . . . . . . . . . . . . . 13
2.3 Analysis of R × C Contingency Tables . . . . . . . . . . . . . . . . . . . . 17
2.4 Analysis of Stratified 2 × 2 Contingency Tables . . . . . . . . . . . . . . . 20
2.5 Regression Models for Binary Outcomes . . . . . . . . . . . . . . . . . . . 22
2.5.1 Logistic regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5.2 Estimation and inference for logistic regression . . . . . . . . . . . . 26
2.5.3 Exact logistic regression . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.4 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.6 Regression Models for Ordinal Outcomes . . . . . . . . . . . . . . . . . . 28
2.6.1 Proportional odds model . . . . . . . . . . . . . . . . . . . . . . . . 29
2.6.2 Some alternative models for ordinal outcomes . . . . . . . . . . . . 33
2.6.3 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.7 Adjustment for Baseline Response . . . . . . . . . . . . . . . . . . . . . . 37
2.8 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3 Continuous Outcomes 45
Fang-Shu Ou
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.2 The t-Test (One Population) . . . . . . . . . . . . . . . . . . . . . . . . . 46

vii
viii Contents

3.3 The t-Test (Two Populations) . . . . . . . . . . . . . . . . . . . . . . . . . 47


3.4 Mann-Whitney U-Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.5 Paired Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.5.1 Paired t-test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.5.2 Wilcoxon signed rank test . . . . . . . . . . . . . . . . . . . . . . . 51
3.6 Multiple Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.7 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.7.1 Residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.7.2 Inference for linear regression . . . . . . . . . . . . . . . . . . . . . 56
3.7.3 ANCOVA models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.7.4 Nonlinear regression . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4 Time to Event Data 63


Daniel Scharfstein, Yuxin Zhu and Anastasios Tsiatis
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.2 ACTG 320 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.3 Mathematical Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.3.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.3.2 Hazard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3.3 Censoring and observed data . . . . . . . . . . . . . . . . . . . . . . 66
4.4 Estimation of Survival Distribution . . . . . . . . . . . . . . . . . . . . . . 67
4.5 Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.6 Cox Regression Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.7 Informative Censoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5 Count Data 81
Xin He and Jianguo “Tony” Sun
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.2 Regression Analysis of Simple Count Data . . . . . . . . . . . . . . . . . . 82
5.2.1 Poisson regression for count . . . . . . . . . . . . . . . . . . . . . . 83
5.2.2 Negative binomial regression for count . . . . . . . . . . . . . . . . 84
5.2.3 Poisson and negative binomial regression for rate . . . . . . . . . . 85
5.2.4 Other models for simple count data . . . . . . . . . . . . . . . . . . 86
5.3 Regression Analysis of Correlated Count Data: Likelihood-Based Ap-
proaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.3.1 Maximum pseudo-likelihood estimation for the Poisson model . . . 87
5.3.2 Maximum likelihood estimation for the Poisson model . . . . . . . 88
5.3.3 Maximum likelihood estimation for the negative binomial model . . 88
5.4 Regression Analysis of Correlated Count Data: Distribution-Free Ap-
proaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.4.1 Conditional estimating equation method . . . . . . . . . . . . . . . 89
5.4.2 Unconditional estimating equation method . . . . . . . . . . . . . . 90
5.4.3 Analysis of the National Cooperative Gallstone Study . . . . . . . . 92
5.5 Discussion and Concluding Remarks . . . . . . . . . . . . . . . . . . . . . 92
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Contents ix

6 Longitudinal Data 97
Soeun Kim and Myunghee Cho Paik
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.2 Generalized Linear Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.3 Generalized Estimating Equations . . . . . . . . . . . . . . . . . . . . . . 98
6.3.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.3.2 Asymptotic properties . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.3.3 Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.3.4 Model selection criterion in GEE . . . . . . . . . . . . . . . . . . . 101
6.4 Generalized Linear Mixed Models . . . . . . . . . . . . . . . . . . . . . . . 102
6.4.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6.4.2 Population average versus subject-specific model . . . . . . . . . . 102
6.4.3 Estimation procedures . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.4.3.1 Marginal likelihood . . . . . . . . . . . . . . . . . . . . . . 103
6.4.3.2 Conditional likelihood . . . . . . . . . . . . . . . . . . . . 104
6.5 Test Statistics Under Randomization . . . . . . . . . . . . . . . . . . . . . 104
6.5.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.5.2 Score-type test for GEE under randomization . . . . . . . . . . . . 105
6.5.3 Score test for GLMMs under randomization . . . . . . . . . . . . . 106
6.6 Handling Missing Data in Clinical Trials . . . . . . . . . . . . . . . . . . . 107
6.6.1 Missing data in GEE . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.6.2 Missing data in GLMMs . . . . . . . . . . . . . . . . . . . . . . . . 108
6.7 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

7 Recurrent Events 117


Yujie Zhong and Richard Cook
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.1.1 Recurrent event data . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.1.2 Data from a cystic fibrosis Trial . . . . . . . . . . . . . . . . . . . . 118
7.2 Notation and Model Formulation . . . . . . . . . . . . . . . . . . . . . . . 119
7.2.1 Analysis considerations with recurrent event data . . . . . . . . . . 119
7.2.2 Methods based on rate and mean functions . . . . . . . . . . . . . . 120
7.2.3 Censoring, Likelihood, and Marginal Methods . . . . . . . . . . . . 123
7.2.4 Assessment based on exacerbations in cystic fibrosis . . . . . . . . . 124
7.3 Sample Size Based on Proportional Rate Functions . . . . . . . . . . . . . 126
7.3.1 Derivations under a negative binomial model . . . . . . . . . . . . . 126
7.3.2 Illustrative sample size calculation . . . . . . . . . . . . . . . . . . . 128
7.4 Other Considerations in Recurrent Event Analyses . . . . . . . . . . . . . 128
7.4.1 Issues regarding causal inference . . . . . . . . . . . . . . . . . . . . 128
7.4.2 Marginal multivariate failure times models . . . . . . . . . . . . . . 129
7.4.3 Adaptive two-stage sample size estimation . . . . . . . . . . . . . . 130
7.4.4 Recurrent and terminal events . . . . . . . . . . . . . . . . . . . . . 131
7.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

III Design of Randomized Controlled Trials 137


x Contents

8 Cross-Over Designs 139


Stephen Senn
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
8.2 Some Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
8.2.1 Example 1 : An AB/BA design . . . . . . . . . . . . . . . . . . . . 141
8.2.2 Example 2: A design in three treatments, three periods, and six
sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
8.2.3 Example 3: An incomplete blocks design with fewer periods than
treatments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
8.2.4 Example 4: A replicate cross-over design with more periods than
treatments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
8.2.5 Example 5: A replicate bioequivalence study comparing two formu-
lations in four periods . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.3 General Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.3.1 Phase of drug development . . . . . . . . . . . . . . . . . . . . . . . 145
8.3.2 Suitable indications . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.4 Issues in Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
8.4.1 Models for cross-over trials . . . . . . . . . . . . . . . . . . . . . . . 146
8.4.2 Patient effects and variance structures . . . . . . . . . . . . . . . . 147
8.4.3 Carry-over effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
8.4.4 Residual degrees of freedom and error estimation . . . . . . . . . . 148
8.5 Examples of Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
8.5.1 Basic estimator approach . . . . . . . . . . . . . . . . . . . . . . . . 150
8.5.2 Two-sample t-test approach . . . . . . . . . . . . . . . . . . . . . . 152
8.5.3 Linear and mixed models . . . . . . . . . . . . . . . . . . . . . . . . 152
8.5.4 Testing for carry-over . . . . . . . . . . . . . . . . . . . . . . . . . . 153
8.5.5 8.5.5 An unbiased estimate of the treatment effect . . . . . . . . . . 154
8.5.6 The two-stage procedure . . . . . . . . . . . . . . . . . . . . . . . . 154
8.6 Issues in Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.6.1 Choosing sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.6.2 Other issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
8.6.3 Planning the sample size . . . . . . . . . . . . . . . . . . . . . . . . 157
8.7 N-of-1 trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
8.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.9 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.10 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

9 Factorial Designs 163


Bibhas Chakraborty and Palash Ghosh
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
9.2 Different Usages of Factorial Designs . . . . . . . . . . . . . . . . . . . . . 164
9.2.1 Efficiency of confirmatory trials: Evaluation of more than one Inter-
vention in a single study . . . . . . . . . . . . . . . . . . . . . . . . 165
9.2.2 Screening trials: Developing multicomponent interventions . . . . . 166
9.2.3 Situations where factorial designs are not suitable . . . . . . . . . . 169
9.3 Full Factorial Designs: A Theoretical Background . . . . . . . . . . . . . 170
9.4 Fractional Factorial Designs . . . . . . . . . . . . . . . . . . . . . . . . . . 174
9.5 Analysis Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
9.6 Follow-up Studies: Developing Multicomponent Interventions . . . . . . . 179
9.7 Power and Sample Size Considerations . . . . . . . . . . . . . . . . . . . . 180
Contents xi

9.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184


Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

10 Cluster Randomized Designs 191


Martin Bland and Mona Kanaan
10.1 What is a Cluster Randomized Trial? . . . . . . . . . . . . . . . . . . . . 191
10.2 The Problem of Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . 192
10.3 Summary Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
10.4 The Intra-Cluster Correlation Coefficient and the Design Effect . . . . . . 196
10.5 Baseline and Other Adjustments . . . . . . . . . . . . . . . . . . . . . . . 198
10.6 Robust Standard Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
10.7 Multilevel Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
10.8 Generalized Estimating Equations (GEE) Models . . . . . . . . . . . . . . 203
10.9 Stepped Wedge Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
10.10 Sample Size Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
10.11 Practical Problems of Cluster Randomized Trials . . . . . . . . . . . . . . 208
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

11 Randomization, Stratification, and Outcome-Adaptive Allocation 215


Oleksandr Sverdlov and Yevgen Ryeznik
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
11.2 Simple and Restricted Randomization . . . . . . . . . . . . . . . . . . . . 218
11.3 Stratified and Covariate-Adaptive Randomization . . . . . . . . . . . . . . 225
11.4 Outcome-Adaptive Randomization . . . . . . . . . . . . . . . . . . . . . . 233
11.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238

12 Background to Sample Size Calculations 243


Jo Rothwell, Cindy Cooper, Steven Julious and Mike Campbell
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
12.2 Types of Trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
12.2.1 Parallel group trials . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
12.2.2 Cross-over trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
12.3 Continuous Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
12.3.1 Superiority trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
12.3.1.1 Parallel group trials . . . . . . . . . . . . . . . . . . . . . 247
12.3.1.2 Quick results . . . . . . . . . . . . . . . . . . . . . . . . . 248
12.3.1.3 Worked example 1 . . . . . . . . . . . . . . . . . . . . . . 249
12.3.1.4 Cross-over trials . . . . . . . . . . . . . . . . . . . . . . . 249
12.3.1.5 Quick results . . . . . . . . . . . . . . . . . . . . . . . . . 250
12.3.1.6 Worked example 2 . . . . . . . . . . . . . . . . . . . . . . 251
12.3.2 Equivalence trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
12.3.2.1 Parallel group trials . . . . . . . . . . . . . . . . . . . . . 253
12.3.2.2 Worked example 3 . . . . . . . . . . . . . . . . . . . . . . 255
12.3.2.3 Cross-over trials . . . . . . . . . . . . . . . . . . . . . . . 255
12.3.2.4 Worked example 4 . . . . . . . . . . . . . . . . . . . . . . 257
12.3.3 Non-inferiority trials . . . . . . . . . . . . . . . . . . . . . . . . . . 259
12.3.3.1 Parallel group trials . . . . . . . . . . . . . . . . . . . . . 259
12.3.3.2 Worked example 5 . . . . . . . . . . . . . . . . . . . . . . 259
12.3.3.3 Cross-over trials . . . . . . . . . . . . . . . . . . . . . . . 260
12.3.3.4 Worked example 6 . . . . . . . . . . . . . . . . . . . . . . 260
xii Contents

12.4 Binary Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262


12.4.1 Superiority trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
12.4.1.1 Parallel group trials . . . . . . . . . . . . . . . . . . . . . 262
12.4.1.2 Method 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
12.4.1.3 Worked example 7 . . . . . . . . . . . . . . . . . . . . . . 264
12.4.1.4 Cross-over trials . . . . . . . . . . . . . . . . . . . . . . . 266
12.4.1.5 Worked example 8 . . . . . . . . . . . . . . . . . . . . . . 267
12.4.2 Equivalence trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
12.4.2.1 Parallel group trials . . . . . . . . . . . . . . . . . . . . . 269
12.4.2.2 Worked example 9 . . . . . . . . . . . . . . . . . . . . . . 269
12.4.2.3 Cross-over trials . . . . . . . . . . . . . . . . . . . . . . . 270
12.4.3 Non-inferiority trials . . . . . . . . . . . . . . . . . . . . . . . . . . 270
12.4.3.1 Parallel group trials . . . . . . . . . . . . . . . . . . . . . 270
12.4.3.2 Worked example 10 . . . . . . . . . . . . . . . . . . . . . . 271
12.5 Final Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

13 Sample Size Estimation and Power Analysis: Time to Event Data 275
Oliver Bautista and Keaven Anderson
13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
13.2 Methods for Sample Size Estimation and Power Analysis . . . . . . . . . 276
13.2.1 Approaches relating to acquisition of events . . . . . . . . . . . . . 276
13.2.2 Estimation of required number of events: no accounting of other
design parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
13.2.3 Estimation of required number of events: with accounting of other
design parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
13.3 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
13.3.1 Rare events with non-proportional hazard ratio . . . . . . . . . . . 279
13.3.1.1 The study as designed . . . . . . . . . . . . . . . . . . . . 279
13.3.1.2 The study as it unfolded . . . . . . . . . . . . . . . . . . . 279
13.3.1.3 Insights gleaned from the study . . . . . . . . . . . . . . . 280
13.3.1.4 Alternative strategies . . . . . . . . . . . . . . . . . . . . . 281
13.3.1.5 Alternative strategy example . . . . . . . . . . . . . . . . 282
13.3.2 An oncology study . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
13.3.3 A diabetes noninferiority study . . . . . . . . . . . . . . . . . . . . 289
13.4 Special Topics and Recent Developments . . . . . . . . . . . . . . . . . . . 293
13.4.1 Treatment effects beyond hazard ratios . . . . . . . . . . . . . . . . 293
13.4.2 Sample size re-estimation . . . . . . . . . . . . . . . . . . . . . . . . 294
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297

14 Sample Size Estimation and Power Analysis: Longitudinal Data 301


Sin-Ho Jung
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
14.2 Generalized Estimating Equations (GEE) Method . . . . . . . . . . . . . 302
14.2.1 Continuous outcome variable case . . . . . . . . . . . . . . . . . . . 303
14.2.2 Binary outcome variable case . . . . . . . . . . . . . . . . . . . . . 304
14.3 Power Analysis and Sample Size Estimation . . . . . . . . . . . . . . . . . 305
14.3.1 Continuous outcome variable case . . . . . . . . . . . . . . . . . . . 306
14.3.2 Binary outcome variable case . . . . . . . . . . . . . . . . . . . . . 306
14.4 Modelling Missing Pattern and Correlation Structure . . . . . . . . . . . . 307
14.4.1 Missing pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Contents xiii

14.4.2 Correlation structure . . . . . . . . . . . . . . . . . . . . . . . . . . 308


14.5 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
14.5.1 Labor pain study (Continuous outcome case) . . . . . . . . . . . . 309
14.5.2 Design of an RCT based on GENISOS (binary outcome case) . . . 309
14.6 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

IV Monitoring of Randomized Controlled Trials 315


15 Group Sequential Methods 317
Michael Proschan
15.1 Group Sequential Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 317
15.1.1 A unified framework . . . . . . . . . . . . . . . . . . . . . . . . . . 318
15.1.2 Boundaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
15.2 The Effect of Monitoring on Power . . . . . . . . . . . . . . . . . . . . . . 329
15.3 Futility/Stochastic Curtailment . . . . . . . . . . . . . . . . . . . . . . . . 330
15.4 Problems with Post-Trial Inference . . . . . . . . . . . . . . . . . . . . . . 334
15.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336

16 Sample Size Re-Estimation 339


Tobias Mütze and Tim Friede
16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
16.2 Nuisance Parameter Based Sample Size Re-Estimation . . . . . . . . . . . 342
16.2.1 Sample size re-estimation for normal data . . . . . . . . . . . . . . 342
16.2.1.1 Motivating example . . . . . . . . . . . . . . . . . . . . . 342
16.2.1.2 Statistical model and sample size re-estimation . . . . . . 342
16.2.1.3 Unblinded nuisance parameter estimation . . . . . . . . . 343
16.2.1.4 Blinded nuisance parameter estimation . . . . . . . . . . . 344
16.2.1.5 Comparison of sample size re-estimation procedures . . . 345
16.2.2 Sample size re-estimation for count data . . . . . . . . . . . . . . . 349
16.2.2.1 Motivating example . . . . . . . . . . . . . . . . . . . . . 350
16.2.2.2 Negative binomial outcomes . . . . . . . . . . . . . . . . . 351
16.2.3 Further issues and recent developments . . . . . . . . . . . . . . . . 352
16.2.3.1 Non-inferiority trials . . . . . . . . . . . . . . . . . . . . . 352
16.2.3.2 Controlling the type I error rate . . . . . . . . . . . . . . 353
16.2.3.3 Size of the internal pilot study . . . . . . . . . . . . . . . 353
16.2.3.4 Covariates . . . . . . . . . . . . . . . . . . . . . . . . . . . 354
16.2.3.5 Other endpoints and more complex designs . . . . . . . . 355
16.2.3.6 Multi-arm trials . . . . . . . . . . . . . . . . . . . . . . . . 355
16.2.3.7 Incorporating historical data into the sample size re-
estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
16.3 Effect-Based Sample Size Re-Estimation . . . . . . . . . . . . . . . . . . . 356
16.3.1 Controlling the type I error rate . . . . . . . . . . . . . . . . . . . . 357
16.3.2 Sample size adaptation . . . . . . . . . . . . . . . . . . . . . . . . . 361
16.3.3 Further issues and recent developments . . . . . . . . . . . . . . . . 361
16.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
xiv Contents

17 Adaptive Designs 371


Gernot Wassmer, Franz Koenig and Martin Posch
17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
17.2 General Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372
17.2.1 The combination testing principle . . . . . . . . . . . . . . . . . . . 373
17.2.2 The closed testing principle . . . . . . . . . . . . . . . . . . . . . . 375
17.2.3 Adaptive designs for multiple hypotheses . . . . . . . . . . . . . . . 375
17.2.4 Assessing the performance of an adaptive design . . . . . . . . . . . 377
17.3 Treatment Arm Selection Designs . . . . . . . . . . . . . . . . . . . . . . . 378
17.3.1 The procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
17.3.2 Binary and survival endpoints . . . . . . . . . . . . . . . . . . . . . 380
17.3.3 Case studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382
17.4 Population Enrichment Designs . . . . . . . . . . . . . . . . . . . . . . . . 382
17.4.1 The procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
17.4.2 Effect specification . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
17.4.3 Binary and survival endpoints . . . . . . . . . . . . . . . . . . . . . 385
17.4.4 Case studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
17.5 Discussion and Further Developments . . . . . . . . . . . . . . . . . . . . 386
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388

V Practical Issues in Analysis of Randomized Controlled


Trials 395
18 Multiple Testing 397
Yi Liu, Jason Hsu and Szu-Yu Tang
18.1 Error Rates in Multiple Comparisons . . . . . . . . . . . . . . . . . . . . . 398
18.2 Principles of Multiple Testing . . . . . . . . . . . . . . . . . . . . . . . . . 399
18.2.1 Partitioning principle . . . . . . . . . . . . . . . . . . . . . . . . . . 400
18.2.2 Closed testing principle . . . . . . . . . . . . . . . . . . . . . . . . . 401
18.3 A Simple Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
18.4 Shortcutting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
18.4.1 Holm’s method is a shortcut . . . . . . . . . . . . . . . . . . . . . . 405
18.4.2 Hochberg’s method is also a shortcut . . . . . . . . . . . . . . . . . 405
18.5 Paths in Decision-Making . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
18.5.1 Decision path respecting principle . . . . . . . . . . . . . . . . . . . 408
18.5.2 A specific dose × endpoint example . . . . . . . . . . . . . . . . . . 409
18.6 Setting Priorities in Multiple Testing for Each Study . . . . . . . . . . . . 411
18.6.1 The graphical approach . . . . . . . . . . . . . . . . . . . . . . . . . 413
18.7 Logical Relationships Among Parameters Tested . . . . . . . . . . . . . . 415
18.7.1 Logic induced in multiple test construction . . . . . . . . . . . . . . 416
18.7.2 Logic inherent in scientific parameters . . . . . . . . . . . . . . . . 418
18.8 Going Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

19 Subgroup Analysis 423


Rui Wang
19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
19.2 Methods for Conducting Subgroup Analyses . . . . . . . . . . . . . . . . . 424
19.2.1 Commonly used methods . . . . . . . . . . . . . . . . . . . . . . . . 424
19.2.2 Qualitative interaction . . . . . . . . . . . . . . . . . . . . . . . . . 428
Contents xv

19.2.3 Graphical methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 431


19.2.4 Multivariate tests of interaction . . . . . . . . . . . . . . . . . . . . 434
19.3 Power Consideration of Subgroup Analysis . . . . . . . . . . . . . . . . . 437
19.4 Subgroup Analysis Reporting and Interpretation . . . . . . . . . . . . . . 437
19.5 Final Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438

20 Competing Risks 443


Haesook Kim
20.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
20.2 Cumulative Incidence Function in the Presence of Competing Risks . . . 444
20.2.1 Cumulative incidence function . . . . . . . . . . . . . . . . . . . . . 445
20.2.2 Estimation of CIF in the presence of competing risks . . . . . . . . 445
20.3 Testing for Differences between Cumulative Incidence Curves in the Pres-
ence of Competing Risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
20.3.1 Gray test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
20.3.2 Estimation of Gray statistic . . . . . . . . . . . . . . . . . . . . . . 449
20.4 Competing Risks Regression Analysis . . . . . . . . . . . . . . . . . . . . 451
20.4.1 Cause-specific hazard regression model . . . . . . . . . . . . . . . . 452
20.4.2 Fine and Gray model . . . . . . . . . . . . . . . . . . . . . . . . . . 452
20.4.3 Klein and Andersen model . . . . . . . . . . . . . . . . . . . . . . . 453
20.4.4 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
20.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
20.6 Computing Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458

21 Joint Models for Longitudinal and Time to Event Data 463


Hélène Jacqmin-Gadda, Loı̈c Ferrer and Cécile Proust-Lima
21.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463
21.2 Illustrative Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
21.3 Joint Shared Random-Effect Models . . . . . . . . . . . . . . . . . . . . . 467
21.3.1 Model definition for Gaussian markers . . . . . . . . . . . . . . . . 467
21.3.2 Model definition for discrete markers . . . . . . . . . . . . . . . . . 469
21.3.3 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
21.3.3.1 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
21.3.3.2 Bayesian estimation . . . . . . . . . . . . . . . . . . . . . 470
21.3.3.3 Model diagnostic . . . . . . . . . . . . . . . . . . . . . . . 471
21.3.4 Joint shared random-effect models for clinical trials . . . . . . . . . 471
21.3.4.1 Distinguishing direct and indirect treatment effects . . . . 471
21.3.4.2 Incomplete data . . . . . . . . . . . . . . . . . . . . . . . 473
21.4 Joint Latent Class Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
21.4.1 Model definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
21.4.2 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
21.4.2.1 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
21.4.2.2 Model diagnostic . . . . . . . . . . . . . . . . . . . . . . . 476
21.4.3 Joint latent class models for clinical trials . . . . . . . . . . . . . . 477
21.5 Conclusion and Recent Developments . . . . . . . . . . . . . . . . . . . . 479
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
xvi Contents

VI Miscellaneous Topics in Randomized Controlled Trials 485


22 Design and Analysis Methods for Developing Personalized Treatment
Rules 487
Emily Butler and Michael Kosorok
22.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487
22.2 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
22.3 Analysis Techniques: Single Stage . . . . . . . . . . . . . . . . . . . . . . . 492
22.4 Analysis Techniques: Multiple Stages . . . . . . . . . . . . . . . . . . . . . 496
22.5 Related Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
22.5.1 Variable selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501
22.5.2 Multiple outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
22.5.3 DTRs for observational data . . . . . . . . . . . . . . . . . . . . . . 504
22.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506

23 Safety Evaluation in Clinical Trials 509


H. Amy Xia, Brenda J. Crowe and Jesse A. Berlin
23.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509
23.2 Elements of a Systematic Approach to Clinical Trial Safety Data Evaluation 511
23.2.1 The program safety analysis plan (PSAP) . . . . . . . . . . . . . . 511
23.2.2 Facilitating combining data across studies, including planning meta-
analyses (be prepared) . . . . . . . . . . . . . . . . . . . . . . . . . 512
23.3 Approaches to Characterizing the Product Safety Profile . . . . . . . . . . 512
23.3.1 Known or pre-specified safety issues . . . . . . . . . . . . . . . . . . 513
23.3.1.1 Specific safety issues that should always be considered for
all products . . . . . . . . . . . . . . . . . . . . . . . . . . 513
23.3.1.2 Product-specific adverse events of special interest (AESIs) 513
23.3.1.3 Adverse events not specified in advance . . . . . . . . . . 513
23.3.2 Data sources for safety evaluation including specific safety studies . 514
23.4 Planning for Clinical Data Collection and Standardization . . . . . . . . . 515
23.4.1 Definition of safety outcomes and adjudication . . . . . . . . . . . . 515
23.4.2 Standardization of safety data collection . . . . . . . . . . . . . . . 516
23.5 Safety Data Analysis and Reporting . . . . . . . . . . . . . . . . . . . . . 517
23.5.1 Considerations for individual studies . . . . . . . . . . . . . . . . . 518
23.5.1.1 Defining the safety analysis set . . . . . . . . . . . . . . . 518
23.5.1.2 Accounting for time on or off treatment . . . . . . . . . . 518
23.5.2 Meta-analysis of adverse event data . . . . . . . . . . . . . . . . . . 519
23.5.3 Multiplicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
23.5.4 Signal detection for common events . . . . . . . . . . . . . . . . . 521
23.5.5 Descriptive analysis of infrequent adverse events . . . . . . . . . . 521
23.5.6 Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
23.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523

24 Non-Inferiority Trials 527


Brian L. Wiens
24.1 Background and History . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
24.2 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
24.2.1 Historical studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
24.2.2 Parameters and margins . . . . . . . . . . . . . . . . . . . . . . . . 529

You might also like