Statistical Significance and the PHC Curve

Visit the link below to download the full version of this book:

https://medipdf.com/product/statistical-significance-and-the-phc-curve/

Click Download Now

Hideki Toyoda

Statistical Significance
and the PHC Curve
Hideki Toyoda
Department of Psychology
Waseda University
Shinjuku City, Tokyo, Japan

ISBN 978-981-97-7747-1 ISBN 978-981-97-7748-8 (eBook)

https://doi.org/10.1007/978-981-97-7748-8

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
Singapore Pte Ltd. 2024

This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse
of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore

If disposing of this product, please recycle the paper.

Preface

In March 2019, two impactful papers on statistics were published. One paper [1],
“Moving to a World Beyond ‘p < 0.05’”, was featured in the scholarly journal The
American Statistician, overseen by the American Statistical Association. The title of
the first chapter is “Don’t Say ‘Statistically Significant’”, and it uses the imperative
form to clearly forbid the use of significance testing. Continuing to use p-values is
equivalent to driving a car that the leading manufacturer has recalled and banned
from use.
Another paper [2], “Retire statistical significance”, was published in the presti-
gious scientific journal Nature. This commentary was endorsed by over 800 scientists,
advocating for the statement, “We agree, and call for the entire concept of statistical
significance to be abandoned.” Continuing to teach p-values is like driving a car
which the leading consumer advocacy group has recommended against using.
The p-value is a probability, yet it is difficult to interpret intuitively. Numerical
values used as analysis results are more user-friendly when they are intuitive and
easily understood by statistical novices. Fortunately, such indicators are already in
practical use and widely established in societal infrastructure. For example, they
are implemented in the “posterior probability of being spam” filter used in our
daily emails. Unlike p-values, this probability directly indicates the chance that an
incoming email is spam. It is expressed in terms everyone can understand and feel,
making it a tangible probability. How about adopting this methodology? Consider a
study comparing the duration of hospital stays between treatments A and B. Previ-
ously, research conclusions were typically stated as: “There was a statistically signif-
icant difference at the 5% level in the average duration of hospital stays.” This
phrasing is quite abstract. Instead, we suggest adapting to the specific context of the
research objectives, setting flexible research questions, and responding accordingly.
For example,
• The average duration of hospital stays for Group A is at least half a day shorter
than for Group B.
• 71% of patients in Group A have shorter hospital stays than the average for
Group B.

v
vi Preface

• Group A has an average hospital stay that is, on average, no more than 94% of
that of Group B.
Wouldn’t it be better to express conclusions in terms that are understandable not
only to physicians but also to patients? And like an email filter, this expression would
indicate the correct probability of the statement. Both analysts and readers can feel
the effectiveness of this approach. This book explains the importance of using the
Probability that Hypothesis is Correct (PHC), an intuitive measure that anyone
can understand, as an alternative to the p-value.
All the source code and data in each chapter are freely available on this
book’s GitHub repository: https://github.com/TOYODA-Hideki/Statistical-Signif
icance-and-the-PHC-Curve.
The computing environment under which this book was written was Windows 11
(64bit), R 4.2.2, Stan 2.31.0, CmdStanR 0.5.3.
The script that executes the contents of Chaps. 1–4 of the textbook is “post p01.R”.
The script that executes the contents of Chaps. 5–7 of the textbook is “post p02.R”.
The working directory in R language is “SSandPHC”.

Tokyo, Japan Hideki Toyoda

July 2024
Contents

Part I The Brink of Statistical Science

1 “Statistical Significance” Is Merely a Necessary Condition . . . . . . . . . 3
1.1 Evidence of Predictive Ability Discovered!? . . . . . . . . . . . . . . . . . . . 3
1.2 The Purpose of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 The Procedure of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.1 Sampling Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.2 p-Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.3 Significance Level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.4 Rejection and Acceptance Regions . . . . . . . . . . . . . . . . . . . . 10
1.3.5 Interval Estimation Using Confidence Intervals . . . . . . . . . . 12
1.4 Test Results for “Feeling the Future” . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.5 Let’s Try a “Telekinesis Experiment” . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Column: Statement of the American Statistical Association . . . . . . . . . . . . 17
2 The Invisible Hand of God . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.1 Is There an Effect in Dieting Methods?/Crash Course
in Statistical Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.1.1 Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.1.2 Numerical Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.1.3 Representative Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.1.4 Dispersion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 Test of the Mean Difference of Paired Two Groups . . . . . . . . . . . . . 23
2.3 Statistically Significant but Scientifically Meaningless . . . . . . . . . . 27
2.4 The Null Hypothesis Is Scientifically False . . . . . . . . . . . . . . . . . . . . 28
2.5 The p-Value Question in the National Medical Examination . . . . . 29
2.6 The p-Value Is Probabilistic but Abstract . . . . . . . . . . . . . . . . . . . . . . 30
2.7 The Invisible Hand of God . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.7.1 Focusing Only on Profit Leads to Appropriate Pricing . . . . 32

vii
viii Contents

2.7.2 Focusing Only on Statistical Significance Leads

to a High and Stable Proportion of Meaningless
Papers in Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3 Tiger at the Front Gate, Wolf at the Back Gate . . . . . . . . . . . . . . . . . . . . 37
3.1 Efficacy of Antibacterial Drug A . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 Test for Mean Difference Between Two Independent Groups . . . . 39
3.2.1 Difference in Average Hospitalization Days
Required for Statistical Significance . . . . . . . . . . . . . . . . . . . 41
3.3 Two Types of Errors Associated with Decision-Making . . . . . . . . . 42
3.3.1 Reasons Not to Further Reduce the Risk Level . . . . . . . . . . 42
3.3.2 Power of a Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.4 The Null Hypothesis Cannot Be Adopted . . . . . . . . . . . . . . . . . . . . . 44
3.4.1 Significance Means Finding an Alibi . . . . . . . . . . . . . . . . . . . 45
3.4.2 Attitude Reservation if No Alibi Is Found . . . . . . . . . . . . . . 46
3.4.3 The Null Hypothesis Is False Before Data Collection . . . . . 46
3.5 Tiger at the Front Gate, Wolf at the Back Gate . . . . . . . . . . . . . . . . . 47
3.6 Power Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.6.1 Determining n for Tests of Proportions . . . . . . . . . . . . . . . . . 48
3.6.2 Determining the Sample Size n for the Test of Mean
Differences Between Two Independent Groups . . . . . . . . . . 49
3.6.3 Determining n for the Test of Mean Differences
Between Two Related Groups . . . . . . . . . . . . . . . . . . . . . . . . 50
3.7 Limitations of Power Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
Column: The Multiple Comparisons Problem . . . . . . . . . . . . . . . . . . . . . . . 54
4 The Zombie Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.1 Testing Every Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 The Sample Distribution Relies on Unrecorded Intent . . . . . . . . . . . 62
4.2.1 Intent 1: Fixing n and Observing x . . . . . . . . . . . . . . . . . . . . . 62
4.2.2 Intent 2: Fixing x and Observing n . . . . . . . . . . . . . . . . . . . . . 63
4.2.3 Intent 3: Fixing Research Resources, Observing n,
and Then x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.2.4 Intent 4: Increasing n While Monitoring
x and p-Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3 The Zombie Problem in Significance Testing . . . . . . . . . . . . . . . . . . 67
4.3.1 Philosophical Zombies and Zombie-Like Tests
(Definition of Terms) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.3.2 Eradicating Zombies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4 Confidence Intervals Do Not Save Significance Testing . . . . . . . . . 69
4.5 Pre-registration System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.5.1 Benefits of Pre-registration . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Contents ix

4.5.2 Problems with Pre-registration in Significance

Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
Interlude: Problems Revealed in Part I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Interlude: Issues Revealed in Part I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

Part II Saving the Field of Statistics!

5 Bayes’ Theorem—“The Probability of a Research Hypothesis
Being Correct” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.1 The Probability of Being Spam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.1.1 Joint Probability and Conditional Probability . . . . . . . . . . . 81
5.1.2 Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.1.3 Probability of Not Being Spam . . . . . . . . . . . . . . . . . . . . . . . . 84
5.1.4 Transformation of Bayes’ Theorem 1 . . . . . . . . . . . . . . . . . . 84
5.1.5 Transformation of Bayes’ Theorem 2 . . . . . . . . . . . . . . . . . . 85
5.1.6 Bayesian Updating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.1.7 Comparison with p-Values . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.2 Bayes’ Theorem for Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.3 Likelihood Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.3.1 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.3.2 Likelihood Does Not Depend on Intent . . . . . . . . . . . . . . . . . 89
5.3.3 Prior and Posterior Distributions . . . . . . . . . . . . . . . . . . . . . . 90
5.4 Estimating the Population Proportion for “Premonition
of the Future” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.4.1 Probability that a Research Hypothesis Is True . . . . . . . . . . 92
5.4.2 Peer Review with Metrics Convincing to Those
with Domain Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.4.3 Determining the Baseline c . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.4.4 It Is Not Necessary to Set a Baseline c/PHC Curve . . . . . . . 94
5.4.5 Actively Presenting Claims Equivalent to the Null
Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.4.6 Clarification of Hypothesis Truth as the Number
of Observations Increases . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.4.7 The Binary Judgment of Significance
or Non-significance Involves Great Social
Disadvantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6 Concluding Remarks with Sincerity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.1 Inference of the Difference Between Two Independent
Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.1.1 Modeling the Data Generation Process . . . . . . . . . . . . . . . . . 99
6.1.2 Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
x Contents

6.1.3 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

6.2 Breaking Free from Zombies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
6.3 Prior Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6.3.1 Uniform Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6.3.2 Determining the Parameters of the Uniform
Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.3.3 The Upper and Lower Limits of the Uniform
Distribution Can Be Determined Loosely . . . . . . . . . . . . . . . 103
6.3.4 Unique Properties of the Uniform Distribution
as a Prior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.4 Inference of the Difference Between Two Independent
Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.4.1 EAP, post.sd, Credible Intervals . . . . . . . . . . . . . . . . . . . . . . . 106
6.5 Generative Quantities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.6 Posterior Distribution of the Difference in Population Means . . . . 107
6.6.1 Probability that the Hypothesis of a Difference Is
Correct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.6.2 Probability that the Hypothesis of No Difference Is
Correct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.7 Standardized Mean Difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.7.1 Power Analysis May Not Always Be Valid . . . . . . . . . . . . . 111
6.7.2 Posterior Distribution of the Standardized Mean
Difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
6.7.3 Probability that the Hypothesis δcon > c Is Correct . . . . . . . 113
6.8 Non-overlap Degree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.9 Threshold Exceedance Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.10 Inference of Ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Interlude: Resolved Issues from Part I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Part III Education and Malice

7 Seligman’s Dogs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.1 The Diet Problem Revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.1.1 Test for the Difference Between Two Corresponding
Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.1.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.1.3 In the Case of a Learner Who Memorized Test
Statistics Without Understanding . . . . . . . . . . . . . . . . . . . . . . 128
7.2 Likelihood Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.2.1 Freedom from the Multiplicity Problem . . . . . . . . . . . . . . . . 130
Contents xi

7.2.2 In the Case of a Learner Who Memorized

that the Posterior Distribution Can Be Obtained
by MCMC Without Understanding . . . . . . . . . . . . . . . . . . . . 131
7.3 PHC Makes the Meaning of Analytical Results
More Tangible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
7.4 Properties of PHC Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.5 The Future of Statistical Education . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7.5.1 Top-Down Approach and Learned Helplessness . . . . . . . . . 137
7.5.2 Top-Down Approach and Sense of Control/Efficacy . . . . . . 138
7.5.3 Fostering a Sense of Competence and Control
in Modeling the Data Generation Process . . . . . . . . . . . . . . . 138
8 Tampering and Cover-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
8.1 Tampering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
8.2 Cover-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
8.2.1 Cover-Up by Thinning Out Variables . . . . . . . . . . . . . . . . . . 142
8.2.2 Cover-Up by Selecting Periods . . . . . . . . . . . . . . . . . . . . . . . . 143
8.3 HARKing and Cover-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
8.4 Rethinking the Pre-registration System . . . . . . . . . . . . . . . . . . . . . . . 144
8.4.1 Review of Pre-registration in the Absence
of Cover-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
8.4.2 Differences Between p-Values and PHC
in Pre-registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.4.3 Publication Bias and Peer-Reviewed Pre-registration
System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
8.4.4 PHC and Pre-registration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
8.5 Saving Dying Statistics! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Q & A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
 Part I
The Brink of Statistical Science
Chapter 1
“Statistical Significance” Is Merely
a Necessary Condition

Criteria for Peer Review Unrelated to Scholarly Value.

1.1 Evidence of Predictive Ability Discovered!?

This intriguing story begins with a peculiar paper. Dr. Daryl Bem, an emeritus pro-
fessor at Cornell University and a renowned social psychologist, is well-known for
his EBE theory (a theory on how feelings of sexual orientation diversity transform
into erotic love), which is featured in many textbooks.
It was in 2011. In JPSP, the most prestigious journal in social psychology, Bem
suddenly published a paper titled “Feeling the Future” [5] (see Fig. 1.1). The paper,
with its somewhat strange and romantic title, contained numerous experiments and
their analyses, but the first experiment was particularly noteworthy. The experiment,
which involved nude photographs, went roughly as follows:

On a computer screen, two curtains are displayed side by side. Participants are
instructed: “Behind one of the curtains is a nude photo of the opposite sex, and
behind the other, there is none. Please click on the curtain you think has the
photo.” After the participant clicks on the curtain they feel has the nude photo,
a random number generator sets the photo behind only one curtain. The curtain
is then opened to check if the photo is there. This counts as one trial. Out of
1560 trials, 829 correctly guessed the location of the nude photo.

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024 3
H. Toyoda, Statistical Significance and the PHC Curve,
https://doi.org/10.1007/978-981-97-7748-8_1
4 1 “Statistical Significance” Is Merely a Necessary Condition

Fig. 1.1 Cover of Bem’s paper titled “Feeling the Future”

Bem analyzed this data using a statistical method called significance testing,
and applied for publication in JPSP based on the statistical significance of “Feeling
the Future”. In academic journals, papers are usually subjected to a review process,
known as peer review, to determine their scholarly value. Many academic papers that
use experiments and surveys have been published based on the results of significance
tests in statistics being deemed significant.
In significance testing, a probability known as the p-value is typically used, and
a result is considered “statistically significant” when this value falls below 5%. In
many fields of research, regardless of the analysis purpose or subject matter, the
criterion of “whether the p-value is less than 0.05” has been used as a standard for
value judgment.
The p-value of Bem’s precognition experiment was 1.3%, satisfying the con-
dition p < 0.05. The reviewers at JPSP were in a dilemma. Publishing a paper
claiming the existence of precognition could tarnish the reputation of a prestigious
academic journal. However, rejecting Bem’s paper solely because it seemed implau-
sible, despite meeting the p < 0.05 criterion, would not be fair or impartial. After
much deliberation, the reviewers accepted Bem’s argument, and “Feeling the Future”
was eventually published in JPSP.
As expected, Bem’s paper caused a significant stir and faced intense criticism
from various quarters. Of course, psychologists immediately began replication exper-
iments, but no one could reproduce the results. Although statistically significant, the
paper was psychologically meaningless.
1.2 The Purpose of Testing 5

1.2 The Purpose of Testing

The reason “Feeling the Future” was published in a prestigious journal of social
psychology was that it met the condition of p < 0.05 required by the significance
test. But what exactly is a significance test? There are many types of significance
tests. Here, we will take the binomial test used by Bem as an example and concretely
explain the general idea behind significance testing.
In statistics, an attempt that can result in different outcomes probabilistically is
called a trial. Suppose we conduct ten trials to predict the location of a nude photo.
The possible outcomes observed are called events. Hits and misses are events. For
example, let’s say the location was correctly predicted seven times. The number of
trials is called the number of trials n, and the number of hits is called the number
of successes x. The ratio of the number of successes to the number of trials is called
the sample proportion, and is calculated as

Number of Successes 7
Sample Proportion = = = 0.7. (1.1)
Number of Trials 10
The sample proportion is a ratio calculated from the data.
Some people might consider a 70% success rate in predicting the location of
a nude photo as evidence of precognitive ability. But does this sample proportion
indicate the existence of precognitive ability? No, it does not necessarily indicate
that.
Because even flipping a fair, unbiased coin, it’s not unusual to get heads 7 out
of 10 times. In trials like a coin toss, where there are only two possible outcomes
(heads or tails), the probability of getting heads is represented by a probability dis-
tribution called the binomial distribution (Fig. 1.2). A probability distribution
shows how likely different values are to be observed. A probability distribution is
sometimes simply referred to as a distribution. The binomial distribution is one type
of probability distribution.
According to Fig. 1.2, the probability of getting heads 7 out of 10 times in a coin
toss is 11.7%. This can be observed more than once in every 10 trials if repeated.
This time, the location of the nude photo was correctly predicted 7 times, but like
a coin toss, it might not be so accurate next time. Therefore, just considering the
sample proportion leads to endless debates like “It’s high enough, so there must be
precognitive ability!” or “No, it’s just a coincidence, like a coin toss.”
The most academically important thing is the reproducibility of the phenomenon.
What matters is that the average ratio in an infinite number of replication experiments
under the same conditions is sufficiently high. If that is demonstrated, it becomes
clear to anyone that precognitive ability exists.
The average ratio mentioned here is different from the sample proportion calcu-
lated from the current data, so to distinguish it, let’s call it the population proportion
π (pi). Academically, it’s the value of the population proportion, not the sample pro-
portion, that is of interest.