The future of fMRI in cognitive

neuroscience

Russell Poldrack

Department of Psychology
Stanford University

poldracklab.org
# of PubMed abstracts mentioning fMRI

1999:
1552 abstracts
2002:
4052 abstracts

1996:
364 abstracts
 Do we really know an order of magnitude
more now than we did in 2002?

Will 40,000 more of the same kind of fMRI

papers tell us twice what we already know?

poldracklab.org
The future of fMRI in cognitive neuroscience

• Selective inference
• Predictive fMRI
• Individual variability
• Reproducible research practices

poldracklab.org
What does the anterior cingulate cortex do?

• “anterior cingulate” and fMRI

• 3683 abstracts in PubMed

poldracklab.org
The standard imaging paradigm

Manipulate Observe
mental activation
process in region X

Forward Inference: Amygdala is

involved in fear

poldracklab.org
What does the ACC do?
maintenance pain phonology interference

difficulty conflict errors attention

forward inference Z estimated using neurosynth.org

poldracklab.org
Neural correlates

Manipulate some Observe associated

mental process brain activation

working memory
maintenance

working memory is sufficient to activate ACC

working memory is necessary to activate ACC
ACC is necessary or sufficient for working memory
poldracklab.org
Some alternatives

• There is some confound driving all of these (such as

autonomic arousal or breathing)

• These are all truly distinct functions performed by subsets of

neurons in the ACC

• These are all truly distinct functions subserved by ACC in

different neural contexts

• These are not truly distinct functions

• We are chopping up mental function in the wrong way

• Thought experiment: What if the phrenologists had fMRI?

poldracklab.org
What if the phrenologists had fMRI?

- Neural Evidence for Suavity

poldracklab.org
The reverse inference paradigm

Observe Predict the mental

activation process engaged by
in region X the subject

Reverse Inference: The person is

experiencing fear

poldracklab.org
Do you really love your iPhone?

• “Earlier this year, I carried out an fMRI experiment to find out whether
iPhones were really, truly addictive, no less so than alcohol, cocaine, shopping
or video games. In conjunction with the San Diego-based firm MindSign
Neuromarketing, I enlisted eight men and eight women between the ages of
18 and 25. Our 16 subjects were exposed separately to audio and to video of
a ringing and vibrating iPhone...most striking of all was the flurry of activation
in the insular cortex of the brain, which is associated with feelings of love and
compassion. The subjects’ brains responded to the sound of their phones as
they would respond to the presence or proximity of a girlfriend, boyfriend or
family member. In short, the subjects didn’t demonstrate the classic brain-
based signs of addiction. Instead, they loved their iPhones.

poldracklab.org
To the Editor:
“You Love Your iPhone. Literally,” by Martin Lindstrom (Op-Ed, Oct. 1), purports to show,
using brain imaging, that our attachment to digital devices reflects not addiction but
instead the same kind of emotion that we feel for human loved ones.
However, the evidence the writer presents does not show this.
The brain region that he points to as being “associated with feelings of love and
compassion” (the insular cortex) is active in as many as one-third of all brain imaging
studies.
Further, in studies of decision making the insular cortex is more often associated with
negative than positive emotions.
The kind of reasoning that Mr. Lindstrom uses is well known to be flawed, because there is
rarely a one-to-one mapping between any brain region and a single mental state; insular
cortex activity could reflect one or more of several psychological processes.
We find it surprising that The Times would publish claims like this that lack scientific
validity.
RUSSELL POLDRACK
Austin, Tex., Oct. 3, 2011
The writer is a professor of psychology and neurobiology at the University of Texas at
Austin. His letter was signed by 44 other neuroscientists.
Does reverse inference work?

effort

Insula
love
activity

pain

poldracklab.org
Insula activation is weakly selective

Some voxels active in more than 20% of studies

Yarkoni et al., 2011
poldracklab.org
Reverse inference

• Informal reverse
inference provides
relatively weak
evidence

TICS, 2006
poldracklab.org
How to make better predictions

poldracklab.org
Decoding mental states using machine learning

}
train to
classify Cross-validation:
mental •Train for each split of
size k
fMRI states
•Compute average
dataset test accuracy predictive accuracy on
of decoding left-out data
on untrained
data

poldracklab.org
An example: Classifying the HCP data

poldracklab.org
Task contrasts are (mostly) easily classified

poldracklab.org
• What we really care about is decoding
psychological functions, not tasks

poldracklab.org
Ontology-based decoding
Experimental conditions

Forward
Reverse

Ontology
terms

Terms maps

Schwartz et al., in prep

poldracklab.org
 L

R
R L R
Visual atlas z = ≠15 z=0 Temporal atlas

R R

L L
R R

y = ≠30 z = ≠27 z = 40 z = 60
Motor atlas Parietal atlas

poldracklab.org
Decoding cognitive functions across subjects
A Vision
B Action Execution
C Decision Making
1.0 D Orthography
SVM
0.8 E Shape Vision
Logistic
Accuracy

0.6
F Audition
Ridge
G Phonology
0.4 Popularity H Conflict
0.2 I Semantics
0.0 J Reinforcement Learning
A B C D E F G H I J K L M N O P Q R S T U V K Working Memory
L Feedback
M Response Inhibition
N Reward
O Stimulus-driven Attention

• Multilabel classifier trained P Speech

Q Emotion Regulation
using OpenfMRI data and R Mentalizing
S Punishment
Cognitive Atlas labels T Error Processing
U Memory Encoding
V Spatial Attention
Koyejo & Poldrack, 2013
poldracklab.org
How might predictive neuroimaging be useful?

Will I get Alzheimer’s disease?

Which depression drug is
best for this patient?

Will this person commit

another crime?
Does this person have
terrorist intent?
poldracklab.org
What do we mean by “prediction”?
• Pubmed search: “predict”[TI] AND “fMRI”
• 96 results
Presurgery  res)ng-­‐state  local  graph-­‐theory  measures  predict  neurocogni)ve  outcomes  a7er  brain  surgery  in  temporal  lobe  epilepsy.

Recollec)on-­‐related  increases  in  func)onal  connec)vity  predict  individual  differences  in  memory  accuracy.

Structural  Connec)vity  Fingerprints  Predict  Cor)cal  Selec)vity  for  Mul)ple  Visual  Categories  across  Cortex.

Individual  differences  in  symptom  severity  and  behavior  predict  neural  ac)va)on  during  face  processing  in  adolescents  with  au)sm.

Res)ng-­‐state  networks  predict  individual  differences  in  common  and  specific  aspects  of  execu)ve  func)on.

Dissociated  signals  in  human  dentate  gyrus  and  CA3  predict  different  facets  of  recogni)on  memory.

Network  measures  predict  neuropsychological  outcome  a7er  brain  injury.

First  steps  in  using  machine  learning  on  fMRI  data  to  predict  intrusive  memories  of  trauma)c  film  footage.

Childhood  maltreatment  and  combat  posLrauma)c  stress  differen)ally  predict  fear-­‐related  fronto-­‐subcor)cal  connec)vity.

Global  gene)c  varia)ons  predict  brain  response  to  faces.

Balancing  reward  and  work:  an)cipatory  brain  ac)va)on  in  NAcc  and  VTA  predict  effort  differen)ally.

Buffering  social  influence:  neural  correlates  of  response  inhibi)on  predict  driving  safety  in  the  presence  of  a  peer.

Pre-­‐exis)ng  brain  states  predict  risky  choices.

Supplementary  motor  area  ac)va)ons  predict  individual  differences  in  temporal-­‐change  sensi)vity  and  its  illusory  distor)ons.

Individual  differences  in  intrinsic  brain  connec)vity  predict  decision  strategy.

poldracklab.org
Prediction as correlation

“in all three experiments the magnitude of connectivity increases correlated across
individuals with recollection accuracy in areas diffusely distributed throughout the
brain”

poldracklab.org
Prediction as prospective correlation

poldracklab.org
Observed correlation ≉ predictive accuracy

“Since any assessment of retrospective fit "uses the data twice", it is

obvious that it gives too optimistic a picture of the validation fit likely to
be obtained on new data.”

poldracklab.org
Testing prediction using cross-validation

poldracklab.org
 “The present analysis shows that hemodynamic activity within
the brain prospectively predicted rearrest in an offender sample.”
Kiehl et al., 2013
 Prediction error using crossvalidation

Integrated Brier scores:

Reference: 0.214
dACC: 0.207

http://www.russpoldrack.org/2013/04/how-well-can-we-predict-future-criminal.html
https://github.com/poldrack/criminalprediction
“Diagnosing” psychiatric disorders

“Here we show that the

synchronous neural interactions
IOP PUBLISHING JOURNAL OF NEURAL ENGINEERING
(SNI) test which assesses the
J. Neural Eng. 7 (2010) 016011 (7pp) doi:10.1088/1741-2560/7/1/016011
functional interactions among
The synchronous neural interactions test neural populations derived from
as a functional neuromarker for magnetoencephalographic (MEG)
post-traumatic stress disorder (PTSD): a recordings can successfully
robust classification method based on the differentiate PTSD patients from
bootstrap healthy control subjects. ...
A P Georgopoulos1,2,4,5,6,9 , H-R M Tan1,2,8 , S M Lewis1,3 ,
A C Leuthold1,2 , A M Winskowski1,6 , J K Lynch1,2 and B Engdahl6,7
Altogether, these findings
1
Brain Sciences Center, US Department of Veterans Affairs Medical Center (11B), Minneapolis,
MN 55417, USA
document robust differences in
2
3
4
Department of Neuroscience, University of Minnesota Medical School, Minneapolis, MN 55455, USA
Department of Neurology, University of Minnesota Medical School, Minneapolis, MN 55455, USA
Department of Psychiatry, University of Minnesota Medical School, Minneapolis, MN 55455, USA
brain function between the PTSD
5
6
Center for Cognitive Sciences, University of Minnesota, Minneapolis, MN 55455, USA
Psychology Section, US Department of Veterans Affairs Medical Center (11B), Minneapolis,
MN 55417, USA
and control groups that can be
7
8
Department of Psychology, University of Minnesota, Minneapolis, MN 5545, USA
Centre for Cognitive Neuroimaging, University of Glasgow, 58 Hillhead Street, Glasgow G12 8QB, UK used for differential
E-mail: [email protected]

Received 31 August 2009

diagnosis...” (my italics)
Accepted for publication 25 November 2009
Published 20 January 2010
Online at stacks.iop.org/JNE/7/016011

Abstract
Traumatic experiences can produce post-traumatic stress disorder (PTSD) which is a
debilitating condition and for which no biomarker currently exists (Institute of Medicine (US)
2006 Posttraumatic Stress Disorder: Diagnosis and Assessment (Washington, DC: National
Academies)). Here we show that the synchronous neural interactions (SNI) test which assesses
the functional interactions among neural populations derived from magnetoencephalographic
(MEG) recordings (Georgopoulos A P et al 2007 J. Neural Eng. 4 349–55) can successfully
differentiate PTSD patients from healthy control subjects. Externally cross-validated,
poldracklab.org
Detection versus differential diagnosis
• The ability of a biomarker to discriminate between patient
and normal does not demonstrate its ability to diagnose the
disorder versus other disorders

• “brain schmutz” (R. Bilder)

BPD

TBI

Normal PTSD ?
GAD

SZ

poldracklab.org
Disease “diagnosis”: A cautionary tale

Challenge: Diagnose typical controls from ADHD, and diagnose

ADHD subtypes (attentive vs combined)

Training data (released 3/1/2011):

- 491 typical controls
- 285 ADHD
Test data (released 6/1/2011):
- 197 unlabeled datasets

poldracklab.org
ADHD-200 results

64% accuracy

Sensitivity: ability to identify true ADHD cases

Specificity: ability to correctly identify healthy cases
J-statistic: sensitivity + specificity -1

poldracklab.org
How did Alberta win?

•Used only age, sex, handedness, and IQ

•Capitalized on differences in gender and IQ between
patient and control groups

•Brings results of other groups into question: where they also

capitalizing on these confounds?
poldracklab.org
 a effect of prior medication use. Similarly, nonabuse levels of
d
r
Why decoding DSM diagnosesNIHmay
Publicbe a fool’s errand
smoking, alcohol, or other drug use cannot be excluded as ex-
Access
planatory factors. 85 Second, comparisons between indi- Author Manuscript
- vidual
Research
diagnoses may fail to yield evidence of distinct defi- Lancet. Author manuscript; available in PMC 2014 April 20.

g cits by virtue of insufficient power due to limited sample size Published in final edited form as:

NIH-PA Author Manuscript

Original Investigation | META-ANALYSIS Lancet. 2013 April 20; 381(9875): 1371–1379. doi:10.1016/S0140-6736(12)62129-1.
- for smaller magnitude
Identification of a Common effects.Neurobiological Substrate
n, for Mental Illness Identification of risk loci with shared effects on five major
t psychiatric disorders: a genome-wide analysis
Madeleine Goodkind, PhD; Simon B. Eickhoff, DrMed; Desmond J. Oathes, PhD; Ying Jiang, MD; Andrew Chang, BS;
r Laura B. Jones-Hagata, MA; Brissa N. Ortega, BS; Yevgeniya V. Zaiko, BA; Erika L. Roach, BA; Mayuresh S. Korgaonkar, PhD;

f
Conclusions
Stuart M. Grieve, DPhil; Isaac Galatzer-Levy, PhD; Peter T. Fox, MD; Amit Etkin, MD, PhD Cross-Disorder Group of the Psychiatric Genomics Consortium*
Supplemental content at
Summary
n Our findings
IMPORTANCE suggest
Psychiatric diagnoses are that
currentlya general
distinguished basedmapping
on sets of specific exists between a jamapsychiatry.com
Our findings show that specific SNPs are
symptoms. However, genetic and clinical analyses find similarities across a wide variety of Background—Findings from family and twin studies suggest that genetic contributions to
l, broad
diagnoses,range
suggestingofthatsymptoms andsubstrate
a common neurobiological the integrity
may exist across of an anterior insula/
mental
associated with
psychiatric disorders a
do not range
in of
all cases map topsychiatric disorders
present diagnostic categories. We aimed ofto iden
illness.
d dACC–based network across a wide variety of neuropsychiat-
specific variants underlying genetic effects shared between the five disorders in the Psychiatric
OBJECTIVE To conduct a meta-analysis of structural neuroimaging studies across multiple
childhood onset
Genomics Consortium: or
autism adult
spectrum onset.
disorder, In
attention particular,
deficit-hyperactivity disorder, bipol
r ric illnesses.
psychiatric diagnoses, These results
followed by parallel analysesdo not imply
of 3 large-scale thatdata
healthy participant phenotypic dif- disorder, major depressive disorder, and schizophrenia.
sets to help interpret structural findings in the meta-analysis. variation
Methods—We inanalysed
calcium-channel activity genes seems
e ferences between diagnoses are negligible. Rather, they provide
genome-wide single-nucleotide polymorphism (SNP) data for the fiv

NIH-PA Author Manuscript

DATA SOURCES PubMed was searched to identify voxel-based morphometry studies through to disorders
haveonineach 33 332 cases and 27 888 controls of European ancestory. To characterise allelic
pleiotropic effects on logistic
psychopathology.
e anJulyorganizing model
2012 comparing psychiatric that
patients emphasizes
to healthy the
control individuals for import of shared en-
the meta-analysis.
effects disorder, we applied a multinomial regression procedure with model
selection to identify the best-fitting model of relations between genotype and phenotype. We
e dophenotypes across
These results provide evidence relevant to the goal
of activationpsychopathology,
foci across thousands of neuroimaging which is not cur-
The 3 parallel healthy participant data sets included resting-state functional magnetic
examined cross-disorder effects of genome-wide significant loci previously identified for bipol
resonance imaging, a database

e rently an explicit component of psychiatric nosology. This

experiments, and a data set with structural imaging and cognitive task performance data. of disorder
moving
broader
and schizophrenia, and used polygenic risk-score analysis to examine such effects from
beyond
set of common variants.descriptive
We undertook pathway syndromes inthe biological
analyses to establish
associations underlying genetic overlap for the five disorders. We used enrichment analysis of
g transdiagnostic perspective is consistent,
patients with an Axishowever,
DATA EXTRACTION AND SYNTHESIS Studies were included in the meta-analysis if they
reported voxel-based morphometry differences between I diagnosis with newer psychiatry, andtrait
expression quantitative towards a nosology
loci (eQTL) data to assess whetherinformed by
SNPs with cross-disorder

t dimensional models such as the National Institute of Mental

and control individuals in stereotactic coordinates across the whole brain, did not present association were enriched for regulatory SNPs in post-mortem brain-tissue samples.
predominantly in childhood, and had at least 10 studies contributing to that diagnosis (or
disease cause. −8
Findings—SNPs at four loci surpassed the cutoff for genome-wide significance (p<5×10 ) in
r Health’s Research Domain Criteria Project.86 Although this
across closely related diagnoses). The meta-analysis was conducted on peak voxel
coordinates using an activation likelihood estimation approach.
the primary analysis: regions on chromosomes 3p21 and 10q24, and SNPs within two L-type

s shared neural substrate suggests common brain structural

voltage-gated calcium channel subunits, CACNA1C and CACNB2. Model selection analysis
MAIN OUTCOMES AND MEASURES We tested for areas of common gray matter volume increase
supported effects of these loci for several disorders. Loci previously associated with bipolar
d changes at some
or decrease across level,
Axis I diagnoses, itareas
as well as is differing
likely that
between these
diagnoses. changes reflect a
Follow-up
analyses on other healthy participant data sets tested connectivity related to regions arising
d diverse set ofand
from the meta-analysis etiologies. Nonetheless,
the relationship of gray matter volume to cognition.the fact that common Correspondence to: Dr Jordan W Smoller, Simches Research Building, Massachusetts General Hospital, Boston, MA 02114
[email protected].

structural changes
the voxel-based are seen despite of 193potentially differing eti-
*Members listed at end of paper
NIH-PA Author Manuscrip

RESULTS Based on morphometry meta-analysis studies comprising

Contributors Overall coordination: Jordan W Smoller, Kenneth Kendler, Nicholas Craddock. Writing committee: Jordan W
15 892 individuals across 6 diverse diagnostic groups (schizophrenia, bipolar disorder,
n ologies
depression,raises the possibility disorder,that some interventions that tar-
(lead), Nicholas Craddock, Kenneth Kendler, Phil Hyoun Lee, Benjamin M Neale, John I Nurnberger, Stephan Ripke, Susan
addiction, obsessive-compulsive and anxiety), we found that gray Santangelo, Patrick F Sullivan. Statistical analysis: Stephan Ripke (lead), Kenneth Kendler, Phil Hyoun Lee, Benjamin M N
matter loss converged across diagnoses in 3 regions: the dorsal anterior cingulate, right
- getinsula,
the andanterior insula there and
were fewdACC mayeffects,
prove of broad use across
Shaun Purcell. Editorial revisions: Richard Anney, Jan Buitelaar, Ayman Fanous, Stephen V Faraone, Witte Hoogendijk, Kl
left insula. By contrast, diagnosis-specific distinguishing Lesch, Douglas F Levinson, Roy H Perlis, Shaun Purcell, Marcella Rietschel, Brien Riley, Edmund Sonuga-Barke, Russell S
only schizophrenia and depression from other diagnoses. In the parallel follow-up analyses of Thomas G Schulze, Anita Thapar. PGC Cross-Disorder Group: Nicholas Craddock, Kenneth S Kendler, Jordan W Smoller (
x- psychopathology.
the 3 independent healthy participant data sets, we found that the common gray matter loss Ayman Fanous, Benjamin Neale, Michael Neale, John I Nurnberger, Roy Perlis, Shaun Purcell, Marcella Rietschel, Susan S
Thomas G Schulze, Anita Thapar. PGC coordinating committee: Patrick F Sullivan (chair), Patrick Bender, Sven Cichon, N
regions formed a tightly interconnected network during tasks and at resting and that lower
Craddock, Mark J Daly, Stephen V Faraone, John Kelsoe, Thomas Lehner, Douglas Levinson, Mick O’Donovan, Pablo Gej
gray matter in this network was associated with poor executive functioning. Jonathan Sebat, Pamela Sklar, Jordan W Smoller. See appendix for PGC Collaborators from Analysis Committee (Mark J D
chair), ADHD Workgroup (Stephen V Faraone, chair), Autism Workgroup (Mark Daly, Bernie Devlin, cochairs), Bipolar D
CONCLUSIONS AND REVELANCE We identified a concordance across psychiatric diagnoses in
terms of integrity of an anterior insula/dorsal anterior cingulate–based network, which may
poldracklab.org
Workgroup (John Kelsoe, Pamela Sklar, cochairs), Major Depressive Disorder Workgroup (Patrick Sullivan, chair), Schizop
Workgroup (Michael O’Donovan, chair).
 J • y

Downloaded from http://schizophreniabulletin.oxfordjournals.org/ by guest on April 7, 2013

205 -

Psychiatric disease is not a steady state

.195 -

185 -

175 -

0 5 10 15 2 0 2 5 30 3 5 4 0 4 5 5 0 5 5 6 0

Fluctuations in daily life function of schizophrenic patients

Weeks
Schizophrenia Bulletin, Vol. 26, No. 3, 2000 Z. Kupper and H. Hoffmann

re 3. Case example (Patrick C.) from the middle-level/slight descent subgroup: Raw scores,
Figure 2. Case example (Marianne B.) from the middle-level/fluctuating subgroup: Raw scores, mean,
n, trend, and variability trend, and variability
NOSIE
255 -
NOSIE
255 -
Functional
level
245 - 245 -

235 -
235 - high
•* \\ f\! medium
225 -

215 -
225 -

215 -
' ^ .Iw-i-^------ low
205 -
205 -
J • y
195 -
.195 -

185 -
185 -

175-
7
i 7
' I ' —I-1 | I r- 1
I ' 175 -
10 15 20 25 30 35 40 45 50 55 60
0 5 10 15 2 0 2 5 30 3 5 4 0 4 5 5 0 5 5 6 0
Weeks
Weeks Weeks Weeks

Figure 3. Case example (Patrick C.) from the middle-level/slight descent subgroup: Raw scores,
686 mean, trend, and variability

NOSIE
255 -

245 -

235 -

225 -

215 -
Kupper & Hoffman, 2000, Schiz. Bull.
poldracklab.org
How does the healthy brain fluctuate over time?

poldracklab.org
The MyConnectome Project
Resting fMRI
Task fMRI
MRI
Diffusion MRI
Structural MRI

Mood

Behavior Lifestyle
Sleep

RNA-seq
PBMCs Exome seq
Blood
Serum Metabolomics

poldracklab.org
Timeline and measurements

81
48
51
13
20
84
58
83
89

poldracklab.org
Resting state fMRI: Surface parcellation
616 cortical parcels + 14 subcortical regions

Default
Visual 1 (Vis 1)
Visual 2 (Vis 2)
Fronto-parietal 1 (FP 1)
Fronto-parietal 2 (FP 2)

Dorsal Attention 1 (DA)

Ventral Attention (VA)
Salience (Sal)
Cingulo-opercular (CO)

Somato-motor (SM)
Medial parietal (MP)
Parieto-occipital (PO)

Similar community structure to previous group studies

(Power et al., 2011) Laumann et al, 2015, Neuron
poldracklab.org
 Within-subject and between subject variability differ drastically
A) Across sessions B) Across subjects
.3 .3
None None
DMN
Vis 1 DMN
Vis 2
Vis .25
.25
FP 1 FP

std z(r)
std z(r)
DA
FP 2
DA VA
VA
Sal Sal
.20 .20
CO
CO
SM

SM SM (lat)
Aud
MP MP
.15 PO .15
PO

Au t)
d
VA

CO

SM M

MP
l
ne

DA
Vis

FP

PO
N
VA
DA2

SM

Sa
MP
1

CO

PO
l

(la
1
N
ne

Sa

S
DM
FP
FP

No
DM
Vis
Vis
No

PO

0.13 0.17 0.15 0.22

mean std z(r) mean std z(r)

Laumann et al, 2015, Neuron

-subject variability in resting state correlations. A) Above, parcel-to-parcel correlation standard deviation across sessions based on the individual subject parcellation an
poldracklab.org
relation standard deviation for each parcel across all of its connections (i.e. the color in each parcel reflects the mean value across the corresponding row in the matrix a
Food/caffeine drastically affects brain connectivity
Tuesday (Uncaffeinated/fasted) Thursday (Caffeinated/fed)
1.0 1.0
None None

DMN 0.8 DMN 0.8

Vis 1 Vis 1
Vis 2 Vis 2
0.6 0.6

FP 1

mean z(r)

mean z(r)
FP 1
0.4 0.4
FP 2 FP 2
DA DA
VA 0.2 VA 0.2
Sal Sal

CO 0 CO 0

-0.2 -0.2
SM SM

MP MP
PO -0.4 PO -0.4

D2A
SaA
2

SM

PO
MP
1

1
ne

l
CO
DA

N
MP
SaA
2

SM

PO
ne

l
CO
1
2
N

V
FP
V

Vis
DM

FP
Vis
FP
DM
Vis
Vis

No
FP
No

Tuesday minus Thursday

0.3
None
DMN
Vis 1 0.2
Vis 2

FP 1 0.1

delta z(r)
FP 2
DA
VA 0
Sal
CO

-0.1
SM
MP
PO -0.2
SaA
DA

MP
2

SM

PO
1
ne

CO
2

Laumann et al, 2015, Neuron

N

V
FP
DM
Vis
Vis

FP
No

poldracklab.org
Caffeine/food affects large-scale network structure

Tuesday (fasted/no caffeine) Thursday (fed/caffeinated)

Fronto-parietal
Default Mode

Cingulo-opercular

Somatomotor

Secondary visual

Provincial hub

Connector hub
Poldrack et al, submitted
poldracklab.org
Reduced rsfMRI dimensionality: within-module connectivity

Compute mean connectivity within each of the 12

resting state networks

mean
within-
network
functional
connectivity
ROIs

mean
between-
ROIs network
functional
connectivity

poldracklab.org
 Within-module connectivity

Between-module connectivity

Poldrack et al, submitted

poldracklab.org
Phenome-wide network analysis

Behavior

Food

WGCNA

Connectivity

Metabolites

ImmPort
genes

Poldrack et al, submitted

why bother?

poldracklab.org
 and disappointment. Refutation and exist, rather than null findings. research findings are false. PLoS Med 2(8): e124.
controversy is seen across the range of As has been shown previously, the

Science in crisis (?)

Copyright: © 2005 John P. A. Ioannidis. This is an
research designs, from clinical trials probability that a research finding open-access article distributed under the terms
and traditional epidemiological studies is indeed true depends on the prior of the Creative Commons Attribution License,
probability of it being true (before which permits unrestricted use, distribution, and
[1–3] to the most modern molecular reproduction in any medium, provided the original
research [4,5]. There is increasing doing the study), the statistical power work is properly cited.
concern that in modern research, false of the study, and the level of statistical

COMMENT
Abbreviation: PPV, positive predictive value
findings may be the majority or even significance [10,11]. Consider a 2 × 2
the vast majority of published research table in which research findings are John P. A. Ioannidis is in the Department of Hygiene
Open
and access, freely
Epidemiology, available
University online
of Ioannina School of
claims [6–8]. However, this should compared against the gold standard
Medicine, Ioannina, Greece, and Institute for Clinical
not be surprising. It can be proven of true relationships in a scientific
Essay Research and Health Policy Studies, Department of
that most claimed research findings field. In a research field both true and Medicine, Tufts-New England Medical Center, Tufts
University School of Medicine, Boston, Massachusetts,
false hypotheses can be made about
2 9 8 | NAT U R E | VO L 4 8 5 | 1 7 M AY 2 0 1 2 Why Most Published Research Findings
are false. Here I will examine the key
AVIAN INFLUENZA Shift expertise
the presence of relationships. Let R
Past
EARTH SYSTEMSbe theclimates HISTORYofOF“true
ratio of the number
United States of America. E-mail: [email protected]

SCIENCE Descartes’ OBITUARYThe

Competing Interests: Wylie
author Vale
has declared that
The
oftheyAre False
to Essay
tracksection
mutations
broademerge
containswhere
interest top.534
give valuable
opinion pieces on topics clues to future
relationships”
warming p.537
a general medical audience.
to “no lost letter tracked usingno competingand
relationships”
Google p.540
among those tested in the field. R
an elusive
interests exist.
hormone p.542
DOI: 10.1371/journal.pmed.0020124
stress

John P. A. Ioannidis

PLoS Medicine | www.plosmedicine.org factors that influence is characteristic

0696this problem and August 2005 of the field and
| Volume can 8 | e124
2 | Issue

S. GSCHMEISSNER/SPL
Summary some corollaries thereof. vary a lot depending on whether the
There is increasing concern that most field targets highly likely relationships
Many landmark findings in preclinical oncology research are not reproducible, incurrent
part because of inadequate
published research findings are
cell
Modeling
PE lines
R SP EC TI and
the
V ES animal
Framework
O N P SY CHfor models.
O LFalse
O G I CA L S orCIE searches
N CE for only one or a few
Positive Findings true relationships among thousands
false. The probability that a research claim
Several methodologists have and millions of hypotheses that may

Raise standards
Puzzlinglyfor
is true may depend on study power and
bias, the number of other studies on the pointed out [9–11] that the high be postulated. Let us also consider,
same question, and, importantly, the ratio PE R SP rate
EC of
TI nonreplication
V ES O N P SY CH (lack
O L O of
G I CA L S CI E N for
CE computational simplicity,
of true to no relationships among the confirmation) of research discoveries circumscribed fields where either there
is only one true relationship (among

High Correlations relationships probed in each scientific is a consequence of the convenient,

field. In this framework, a research finding yet ill-founded strategy of claiming many that can be hypothesized) or
is less likely to be true when the studies conclusive research findings solely on the power is similar to find any of the

preclinical cancerfMRIresearch
conducted in a field are smaller; when the basis of a single study assessed by several existing true relationships. The

in fMRI Studies
Puzzlingly of Emotion,
High Correlations effect sizes are smaller; when there is a
greater number and lesser preselection
of tested relationships; where there is
formal statistical significance, typically
for a p-value less than 0.05. Research
is not most appropriately
and summarized by p-values, but,
represented
pre-study probability of a relationship
being true is R⁄(R + 1). The probability
of a study finding a true relationship
reflects the power 1 − β (one minus

inPersonality, and
of Social
greater flexibility in designs, definitions,

Studies Emotion, outcomes, and analytical modes; when

there is greater financial and other
interest and prejudice; and when more
unfortunately,
notion that
there
medical
is a
research
widespread
articles
the Type II error rate). The probability
of claiming a relationship when none
truly exists reflects the Type I error
It can be proven that 1 rate, α. Assuming that c relationships

Cognition
C. Glenn Begley and Lee M. Ellis proposePersonality,
how and Social
methods, publications and
teams are involved in a scientific field
in chase of statistical significance.
Simulations show that for most study
most claimed research
fi ndings are false.
are being probed in the field, the
expected values of the 2 × 2 table are
given in Table 1. After a research
incentives must change if patients are to benefit.
1 designs and settings,1it is more likely for 2 2 finding2 has been claimed based on
Edward

fields,Massachussetts
Vul,

claimed research fi
Christine
a research claim to be false than true.
Moreover, for many current scientific
1
Institute
ndings may
Cognition
Harris, Piotr
p-values.
Winkielman,
should be interpreted based only on
2
Research findings
of Technology and University of California, San Diego
here as any relationship
&
are
reaching
Harold
defi ned
Pashler
achieving
the post-study
formal statistical significance,
probability that it is true
is the positive predictive value, PPV.

E
often be simply1 accurate measures of the 2 2 2
Edward
prevailingVul,bias. InChristine Harris,
this essay, I discuss the Piotr Winkielman,
formal statistical signifi & cance,
Harold e.g.,Pashler The PPV is also the complementary
fforts over the past decade to trials in oncology have the highest failure
1 implications of these problems for the 2 investigators must reassess their approach to
effective interventions, informative probability of what Wacholder et al.
have called the falseimagingpositive report
Massachussetts Institute
Functional of research.
Technology
magnetic and
resonance University
predictors, of
risk factors,
imaging California,
(fMRI) San Diego
or associations.
Functional magnetic resonance (fMRI) studies of
characterize the genetic alterations rate compared with other therapeutic areas.
conduct and
ABSTRACT— interpretation of
studies of emotion, personality, and“Negative” social cognition researchhavetranslating discovery research into greater
Many landmark findings in preclinical oncology research are not reproducible, in part because of inadequate
is also very
cell lines
useful.personality, and social cognition to
emotion, probability [10]. and animal models.
According the 2
scarcely existed 10
× 2 table, one gets PPV = (1 − β)R⁄(R
in human cancers have led to a better Given the high unmet need in oncology, it clinical success and impact.
“Negative” is actually a misnomer, and

P Raise standards for

drawn much attention in recent years, with high-profile years ago, and yet the field has already achieved a remarkable
IN THE WAKE
understanding of molecular OF HIGH-PROFILE CONTROVERSIES, PSYCHOLOGISTS ublished research findings are the misinterpretation is widespread. − βR + α). A research finding is thus

ARE FACING UPdrivers of this is understandable that barriers to clinical Many factors are responsible for the high
studiesFunctional
frequently magnetic
reportingresonance
extremely high (e.g.,
imaging (fMRI)>.8) cor-
Functional level ofmagnetic
attentionresonance
and prominence. imagingWithin (fMRI)thestudies space of of a few
TO PROBLEMS WITH REPLICATION. However, here we will target
ABSTRACT— sometimes refuted by subsequent
studies relations
of emotion,between brain activation
withpersonality, and socialandrelationships
personality
cognition that measures.
have emotion,
investigators years, it has spawned
personality,
claim several
and social new journals
cognition scarcely (Social Neuroscience,
existed 10
complex set of diseases. Although we in the development may be lower than for other
and We
drawn
evidence,
show attention
disappointment.
much that these
ensuing confusion
incorrelations
Refutationrecent andyears, are exist,
higher
with rather
thanthan
high-profileshould failure rate, notwithstanding the inher-
null years
fi
bendings. Social
ago, and Cognitive
Citation: Ioannidis JPA (2005) Why most published
and Affective
yet theresearch
field has areNeuroscience)
already false.achieved aand is the focus of
remarkable

preclinical cancer research

findings PLoS Med 2(8): e124.

cancer field hoped that this would lead to

BY ED YONG disease areas, and a larger number of drugs
controversy
studies expected is given
frequently seenreporting
across the extremely
rangelimited)
the (evidently As has>.8)
of highreliability
(e.g.,
probability
been shown
of both
that
cor-
a
ently difficult nature of this disease. Cer-
fMRI previously,
level of
research fi
the
substantial
attention
nding
new funding
andCopyright:
prominence. initiatives
Within
© 2005 John P. (National
the space
A. Ioannidis. Institute
This of a of
is an fewMental
research designs,
andbetween
personality from clinical
brainmeasures. trials
activationThe andhigh correlations are all the years,Health, 2007) while receiving
open-access article lavish(Social
distributed attention
under the from the popular
terms
Neuroscience,

F
it has spawned several new journals
more effective drugs, historically, our ability relations
with suboptimal preclinical validation will
Weand more
traditional
show puzzling
that epidemiological
these because
correlations
personality
studiessections
method are higher than
measures.
rarelyshould contain
tainly, the limitations of preclinical tools
is indeed true depends on the prior
be much Socialpress Cognitive
(Hurley,
of the Creative Commons Attribution License,
and 2008)
Affective andNeuroscience)
which permits the trade use,
unrestricted pressand of
is theand
distribution, psychological
focus of
probability of it being true (before
to translate cancer research to clinical suc- enter oncology trials. However, this low suc-
[1–3] to the most modern molecular
expecteddetail about
given the how the
(evidently
research [4,5]. There is increasing
correlations
limited) were obtained.
reliability of both WefMRI such as inadequate cancer-cell-line and
surveyed substantial
doing the study), the statistical power research new funding
communityreproduction
initiatives
(e.g., in any medium, provided
(National
Fiske,
work is properly cited.
2003).Institute the original
Perhaps of Mental
even more
1
cess has been remarkably low . Sadly, clinical C.
authors Glenn
cess rate is not sustainable or acceptable, and
and personality of 55 Begley
measures.
articles
concern that in modern research, false The
that and
high Lee
reported M.
correlations
findings Ellis
areof propose
2
all
this thekind to
of the study, and the level of statistical how
impressive,methods,
mouse models make it difficult for even
Health, 2007) while receiving
however, ispublications
lavish
the number attention
of papers and
from the
from popular
this area that
Abbreviation: PPV, positive predictive value
more puzzling
determine
findings may be abecause
thefew incentives
method
details
majority sections
on how these
or even must
signifi
rarely
table We
change
cance
contain [10,11].
correlations
inusing
which
much if patients
Consider
werepress a 2
(Hurley, ×
have appearedare toinJohn
2 2008) benefit.
and the trade press of the
such prominent journals as Science, Nature, psychological
the vast
detail aboutmajority
computed. how the of published
More correlations
than half research
were obtained.
acknowledged aresearch
surveyed strategy findings
research are
and community
Nature (e.g.,P. A.Fiske,
Neuroscience. Ioannidis 2003).
is in the Department
Perhapsofeven
and Epidemiology, University of Ioannina School of
Hygienemore
claims [6–8]. However, this should compared against the impressive,
gold standard
authors of 55 articles 2 9 correlations
that M A R C Hfor
reported findings 2individual
0 1relationships
of 2 | voxels
this kind V O Land
to 4 8 3 Although
| c N A Tthe
however, UMedicine,
is R E Ioannina,
the | 5and
number 3 Greece,
of1methods
papers from this area
suchthat

E
that computes separate questions usedforinClinical
and Institute research
not be surprising.
fforts over the It can
past bedecade
proven to of true
trials in oncology have inthea scientifi
highest failure investigators must reassess their approach to in
© 2012 Macmillan Publishers Limited. All
that
rights
determine
reports a reserved
few details on how
means of only those voxels exceeding
most claimed research findings
these correlations
field. chosenwere
In a research thresh- have appeared
fieldtherapeuticare quite
both true and
in such
diverse,
Research
a
and
prominent
substantial
Health Policy
journals
number
Studies,
as
of
Department
Science,
widely
Medicine, Tufts-New England Medical Center, Tufts
of
Nature,
cited papers
characterize
computed. the
Moreshow genetic
thanhow alterations
halfthis acknowledged rate
usingcompared with
a strategy other and Nature areas.
Neuroscience. translating discovery research into greater
olds. Here
are false. We I willhave
examine nonindependent false analysis
hypotheses inflates
can be made this field
about have University School
reported of Medicine,
a specific type Boston, Massachusetts,
of empirical finding that
led tothe key
thatincomputes
human cancers
correlations separate
while yielding
a better
correlations Given
for individual
the
reassuring-looking
the high unmet
voxels
presence ofand need
relationships.
scattergrams.
in oncology,
Although Let
appears R the
to
it
questions
bridge
clinical
United
the
Statessuccess
and of America.
methods
and impact.
E-mail:
used [email protected]
in and
suchbrain:
research
understanding of molecular drivers of this is understandable that barriers to clinical Many factors are responsible for theextremely
divide between mind high
reports means of only those voxels exceeding be the
chosen ratio of the
thresh- number of “true Competing Interests: The
of author hascited
declared that in
complex
olds.
cancer
This
The Essay
analysis
setsection
of diseases.
Weinterest
field
of
of broad theshow
hoped how
to athat
implausibly
technique
containsAlthough
general this
this
was used
weonintopics
opinion pieces the
nonindependent
would
high
medical lead to
correlations
audience.
to obtain
analysis
indisease
our survey
the vast majority
areas,inflates
and
sample. larger
are
development may be lower than for other
relationships” to a“no this
quitehigh
relationships”
diverse,
field have
Innumber
a
correlationssubstantial
no failure
reported aently
of drugs specific
number
competingrate,
between
typenature
difficult
poldracklab.org
widely
notwithstanding
measures
interests exist. of individual
of empirical
of this finding
papers
the differences
inher-
disease.that
relating to personality, emotion, and social cognition and mea-Cer-
 td the
runningsummary provided
small studies with low statistical clear
power
California 94305, evidence
. USA.
dence from aArange
1,5
forof subfields
simulationthis.ofTherefore,
within
genetic theassociationthestudies
neuroscience averageshowed literature. We illustrate th

Threats to reproducibility: Low power

simulation of genetic association studies
rthermore,
at a typical dataset the statistical
would generate
sitive result almost 97% of the time , and
showed
Department
University
4

at least onepower
of
literature. We
of Psychology,
Virginia,
false endemic
Charlottesville,
of studies
two efforts cations of this
problem
illustrate
that a typical
inin
positive
thatdataset
neuroscience
result
low statistical
ouralmost and
would power
analysis
97%discuss
of
generate
may
is an
thetime
the impli-
6
at least one false endemic problem in neuro
in
, andfact
two be cations of this for interpre
efforts
to for interpreting the results of individual
6
stimates of the even lower
Virginia than
22904,
replicate promising findings in biomedicine reveal studies.
USA. the 8–31% replicaterangepromisingwe observed.
findings in biomedicine reveal studies.
Wellcome Trust Centre for
5
replication rates of 25% or less . Given that these pub-
7,8

ted with
plication rates small
of 25% or less . Given
7,8
that these pub-

Power failure: why small sample

Human Genetics, University of lishing biases are pervasive across scientific practice, it Low power in the absen
hing biases are pervasive across scientific practice, it Low
Oxford, Oxford, OX3 7BN, UK. power in the absence of other biases
is possible that false positives heavily contaminate the Three main problems cont
Ethical
possible that false positives heavily Schoolimplications.
contaminate the Three
of Physiology
6
Low
and main problems average
contribute power
to producing in
unreliable neuro-
of in studies with low power, even when all other problem may
neuroscience literature as well, and this findings in studies with lo
at least as much,on
if not even size
science
moreBristol, undermines
uroscience literature as well, and this Pharmacology,
nectare based
problem mayUniversity
studies
so, theBristol,
findings
also
mostBS8 research
1TD, has
affect
UK. practices ethical
are
prominent
the
atideal.
least as much,
They are: reliability
implications.
the if not
low even more
probability of In our
so, the of
most research practices are ideal.
the lowjournals . finding true effects; the low
Correspondence to M.R.M. 9,10

eHere,
appropriately neuroscience
analysis of animal model studies, the average sample
ominent journals . 9,10
finding true effects; positive predictive value (PPV;
e-mail: marcus.munafo@ Here, we focus on one major aspect of the problem: see BOX 1 for definitions of
we focus on one major aspect of the problem:
bristol.ac.uk
see BOX 1 for definitions of key statistical terms) when an
ited
w evidence.
statistical size of
power. The relationship 22 study
between animals
doi:10.1038/nrn3475 effect isfor the
lowand
claimed;
power
water
statistical maze
power.
an exaggerated
anda the
Theexperiments
estimate
veracity
relationship
of the mag- between
of the resulting
wasstudy effect is claimed; and an ex
wer and the veracity of the resulting
Katherine finding
S.online
Published is 10 April
Button nitude
, 2013of theP.
John
1,2 effect when
A. Ioannidis true ,effect
3
Claireis discovered.
Mokrysz 1
, Brianfinding
Here, A. Nosek
4 is , nitude of the effect when a
lts are so con-
der-appreciated. only
Low statistical power sufficient
(because
Corrected todiscuss
of 15 April
online we detect
2013 these an effect
under-appreciated.
problems
Jonathan Flint , Emma S. J. Robinson and Marcus R. Munafò
5 6
size
in more detail. of d =power
Low statistical 1.26(because
1
with of we discuss these problems
ds, such as the Abstract | A study with low statistical power has a reduced chance of detecting a true effect,
sNCEthat we have but it NATURE
is less wellREVIEWS | NEUROSCIENCE VOLUME 14 | MAY 2013 | 365
appreciated that low power also reduces the likelihood that a statistically
© 2013 Macmillansignificant
Publishers Limited.
resultAll rights reserved
reflects a true effect. Here, we show that
© 2013 the average
Macmillan statistical
Publishers Limited. Allpower of
rights reserved
16
studies in the neurosciences is very low. The consequences of this include overestimates of
14 size and low reproducibility of results. There are also ethical dimensions
effect 30 to this
12
problem, as unreliable research is inefficient and wasteful. Improving reproducibility
25 in
neuroscience is a key priority and requires attention to well-established but often ignored
search finding 10
methodological principles. 20

%
that the aver- 8
N

It has been claimed and demonstrated that many (and

15
low sample size of studies, small effects or both) nega-
d of neurosci- 6
possibly most) of the conclusions drawn from biomedi- tively affects the likelihood that a nominally statistically
10a true effect. We dis-
8% and ~31%, 4
cal research are probably false1. A central cause for this significant finding actually reflects
important problem is that researchers must publish in cuss the problems that arise when low-powered research
order2to succeed, and publishing is a highly competitive
5
bfields within enterprise, with certain kinds of findings more likely to
designs are pervasive. In general, these problems can be
divided into two categories. The first concerns prob-
0 0
r we observed be published than others. Research that produces novel
results, statistically significant results (that is, typically
lems that are mathematically expected to arise even if
the research conducted is otherwise perfect: in other
10

0
00
–2

–3

–4

–5

–6

–7

–8

–9
uroscience lit-
0–

–1
p < 0.05) and seemingly ‘clean’ results is more likely to be words, when there are no biases that tend to create sta-
11

21

31

41

51

61

71

81
91
1
School of Experimental
published2,3. As a consequence, researchers have strong tistically significant (that is, ‘positive’) results that are
mplications for
Psychology, University of
Bristol, Bristol, BS8 1TU, UK. incentives to engage in research practices Power
that make(%) spurious. The second category concerns problems that
2
School of Social and their findings publishable quickly, even if those prac- reflect biases that tend to co-occur with studies of low
ikelihood that
Community Medicine, tices reduce the likelihood that the findings reflect a true power or that become worse in small, underpowered poldracklab.org
 ofrequired
134 animalsthreshold
would be required of claiming
for the water itsmaze
discovery
balance(that
to strikeis, between using as few animals as po

d
Low power -> unreliable
experiment
science
reaching(assuming
nominalanstatistical of d = 0.49) and such
effect size significance,
68 animals for the radial maze experiment
ble inasexperiments
p < 0.05)and the need to obtain robust, relia
(assuming findings. We argue that it is important to appreciate
anactually
effect sizereflects a true
of d = 0.69); effect95% power,
to achieve
1,12
. Thisthese
probability is called
waste associated with an underpowered study — eve
the PPV
sample
Positive sizes of
Predictive a claimed
would need to
Value discovery.
increase
(PPV): The 220Theand formula
to probability 112, study linking
that achieves
Winner’s the onlyoverestimation
Curse: 80% power still presents
of a2
thatrespectively.
a positive
PPV to What
resultisis:
power particularly
is true striking, however, is possibility
effect sizesthat the foranimals
significanthave results
been sacrificed wi
ed the inefficiency of a continued reliance on small sample out the study detecting the underlying true effect. If
sizes. Despite the apparently large numbers of animals average power in neuroscience animal model studie
PPV =to([1
required – β]acceptable
achieve × R) ⁄ ([1− β] power
statistical × R +inα) these between 20–30%, as we observed in our analysis abo
the ethical implications are clear.
where (1 − β) is the power, β is the type II error, α is the
Low power therefore has an ethical dimension tim
type
100 I error and R is the pre-study odds (that
100
is, theresearch
unreliable oddsis inefficient and wasteful. This app rel
that a probed effect is indeed non-null among to boththehuman
Natureeffects and animal research.
Reviews The principles of
| Neuroscience tes
80 ‘three
80 Rs’ in animal research (reduce, refine and replac dr
being probed). The formula is derived from requireaappropriate
simple experimental design and statist
Post-study probability (%)

an

d
two-by-two
60
table that tabulates the presence
—60 bothandtoo non-
many and too few animals present an ot is

rese rc
as they reduce the value of research outputs. A requi ex
presence of a non-null effect against significant and
ment for sample size and power calculation is includ sta
are
non-significant
40 research findings1. The formula 40
shows
in the Animal Research: Reporting In Vivo Experime

e t ve b s
res
as, that, for studies with a given 80% power pre-study(ARRIVE)odds R,guidelinesthe , but such calculations requir
84
co
20 30% power clear
20 appreciation of the expected magnitude of effe inc
lower the power and the higher 10% power the type I error,
being sought. the ad
nd lower
0 the PPV. And for studies with a givenOf 0 pre-study
course, it is also wasteful to continue datanifc
ur odds 0R and0.2 a given 0.4 type 0.6 I error0.8 (for lection
1.0 example,
once
0 the 20
it is
tra- 40clear that the
60
effect
80
being
100
sought th
d
not exist or is too small to be ofstinterest. an
That is, stud
t t st c er d
ditional p = 0.05Pre-study threshold),
odds R
the lower the power,
are not the when they stop too early, theyint
just wasteful
Figure 5 | The winner’s curse: effect size inflation as
lower the PPV.
Figure 4 | Positive predictive value as a function of the also wasteful when they Naturestop too late.
Reviews
a function of statistical power. The winner’s curse
Planned, sequ
| Neuroscience
no
Nature Reviews | Neuroscience
, pre-study odds of association
For example, suppose
for different
that Button
we
levels
work
of
et al.,
in a tial analyses
2013
scientific
refers are sometimes
field
to the phenomenon used
that studies infind
that large clinical tri
evidence sig
statistical power. The probability that a research finding ofwhen there
an effect oftenisprovide
considerable expenseoforthe
inflated estimates potential
size of hahy
in which
reflects one —
a true effect inalso
five of the
known effects
as the positive we test are that expected
effect.
associated Such to is expected
inflation
with testing when an
participants. effect has
Clinical
poldracklab.orgtrialsinm
predictive value (PPV) — depends on both the pre-study to pass a certain threshold — such as reaching statistical
Threats to reproducibility: Analytic flexibility

Image Distortion Motion Slice Timing

•
Reconstruction Correction Correction Correction

Using standard FSL analysis

options
Spatial Statistical Spatial
FSL/AFNI

•
Normalization Analysis Smoothing

5,376 possible analysis

workflows SPM Statistical
Analysis
Spatial
Smoothing
Spatial
Normalization

Poldrack et al., 2011

poldracklab.org
 Threats to reproducibility: Analytic flexibility
ORIGINAL RESEARCH ARTICLE
published: 11 October 2012
doi: 10.3389/fnins.2012.00149

On the plurality of (methodological) worlds: estimating the

arp
analytic flexibility of fMRI experiments Analytic flexibility in fMRI res

Joshua Carp*
6,912 pipelines
Department of Psychology, University of Michigan, Ann Arbor, MI, USA

Edited by: How likely are published findings in the functional neuroimaging literature to be false?
Satrajit S. Ghosh, Massachusetts
According to a recent mathematical model, the potential for false positives increases with
Institute of Technology, USA
the flexibility of analysis methods. Functional MRI (fMRI) experiments can be analyzed
Reviewed by:
Jonathan E. Peelle, Washington using a large number of commonly used tools, with little consensus on how, when, or
University, USA whether to apply each one. This situation may lead to substantial variability in analysis out-
Eugene Duff, University of Oxford, UK comes. Thus, the present study sought to estimate the flexibility of neuroimaging analysis
*Correspondence: by submitting a single event-related fMRI experiment to a large number of unique analysis
Joshua Carp, Department of
procedures. Ten analysis steps for which multiple strategies appear in the literature were
Psychology, University of Michigan,
530 Church Street, Ann Arbor, MI identified, and two to four strategies were enumerated for each step. Considering all pos-
48109, USA. sible combinations of these strategies yielded 6,912 unique analysis pipelines. Activation
e-mail: [email protected] maps from each pipeline were corrected for multiple comparisons using five thresholding
approaches, yielding 34,560 significance maps. While some outcomes were relatively con-
sistent across pipelines, others showed substantial methods-related variability in activation
strength, location, and extent. Some analysis decisions contributed to this variability more
than others, and different decisions were associated with distinct patterns of variability
across the brain. Qualitative outcomes also varied with analysis parameters: many con-
trasts yielded significant activation under some pipelines but not others. Altogether, these
results reveal considerable flexibility in the analysis of fMRI experiments. This observation,
when combined with mathematical simulations linking analytic flexibility with elevated false
FIGURE 1 | Variation in activation strength across positive rates,pipelines.
analysis suggests that false positive
pipelines. results
Images aremay be more
presented prevalent than
in neurological expected
orientation, with the left
Mean activation denotes the average Z -value forineach the literature.
voxel acrossThis
all risk of inflated false positive
hemisphere rates
displayed on may be Note
the left. mitigated by constraining
that color scales differ across
poldracklab.org
P-hacking: Anything can become significant
Study 2: musical contrast and chronological rejuvenation
…we asked 20 University of Pennsylvania undergraduates to
listen to either “When I’m Sixty-Four” by The Beatles or
“Kalimba.” Then, in an ostensibly unrelated task, they indicated
their birth date (mm/dd/ yyyy) and their father’s age. We used
father’s age to control for variation in baseline age across
participants.
An ANCOVA revealed the predicted effect: According to their birth
dates, people were nearly a year-and-a-half younger after listening to
“When I’m Sixty-Four” (adjusted M = 20.1 years) rather than to
“Kalimba” (adjusted M = 21.5 years), F(1, 17) = 4.92, p = .040.
-Simmons et al., 2011, Psychological Science

poldracklab.org
ogy
P-hacking: Anything can become significant
Table 1. Likelihood of Obtaining a False-Positive Result

Significance level

Researcher degrees of freedom p < .1 p < .05 p < .01

Situation A: two dependent variables (r = .50) 17.8% 9.5% 2.2%
Situation B: addition of 10 more observations 14.5% 7.7% 1.6%
per cell
Situation C: controlling for gender or interaction 21.6% 11.7% 2.7%
of gender with treatment
Situation D: dropping (or not dropping) one of 23.2% 12.6% 2.8%
three conditions
Combine Situations A and B 26.0% 14.4% 3.3%
Combine Situations A, B, and C 50.9% 30.9% 8.4%
Combine Situations A, B, C, and D 81.5% 60.7% 21.5%
Note: The table reports the percentage of 15,000 simulated samples in which at least one of a
set of analyses was significant.
-Simmons Observations
et al., were drawn independently
2011, Psychological Science from a normal distribu-
tion. Baseline is a two-condition design with 20 observations per cell. Results for Situation A were
obtained by conducting three t tests, one on each of two dependent variables and a third on the
average of these two variables. Results for Situation B were obtained by conducting one poldracklab.org
t test after
• “My result isn’t significant, so I need to add more
subjects…”

poldracklab.org
Sample size
4 flexibility
Minimum Sample Size Table 2. Simple Solutio
Publications

Percentage of False-Positive Results

n ≥ 10 n ≥ 20
25 Requirements for autho
22.1% 1. Authors must deci
20 before data collect
16.3% 17.0% 2. Authors must coll
15 14.3% provide a compelli
13.3%
11.5% 12.7% 3. Authors must list
10.4% 4. Authors must repo
10
failed manipulation
5. If observations are
5 the statistical resu
6. If an analysis includ
0 statistical results o
1 5 10 20 Guidelines for reviewers
1. Reviewers should
Number of Additional Per Condition
2. Reviewers should
Observations Before Performing Another t Test
3. Reviewers should
Fig. 1. Likelihood of obtaining a false-positive result when data collection results do not hin
ends upon obtaining significance (p ≤ .05, highlighted by the dotted line). The 4. If justifications of d
figure depicts likelihoods for two minimum sample sizes, as a function of the
ling, reviewers sho
frequency with which significance tests are performed.
exact replication.
-Simmons et al., 2011, Psychological Science
Contradicting this intuition, Figure 1 shows the false-posi-
tive rates from additional simulations for a researcher who has poldracklab.org
pair of observations.
 10. G. Reuter et al., J. Biol. Chem. 278, 35193 (2003). 20. W. L. Hubbell, H. S. Mchaourab, C. Altenbach, M. A. 28. A. L. Davidson, J. Bacteriol. 184, 1225 (2002). RETRACTED 22 DECEMBER 2006; SEE LAST PAGE
11. H. W. van Veen, coproteins,
C. F. Higgins, which
W. N.have theseRes.
Konings, components fused
Lietzow, Structure 4, 779 (1996). R E P O R T S 29. I. L. Urbatsch, G. A. Tyndall, G. Tombline, A. E. Senior,

Threats to reproducibilty: software errors

Structure of MsbA from E. coli: Microbiol. 152, 365 (2001). 21. The collision frequency with paramagnetic reagents J. Biol. Chem. 278, 23171 (2003).
into a single polypeptide, the msbA
12. G. Chang, C. B. Roth, Science 293, 1793 (2001).
gene en-
was deduced from the analysis of the saturation 30. P. Mitchell, Nature 180, 134 (1957). 12. S. J. Mojzsis, T. M. Harrison, R. T. Pidgeon, Nature 409,
within 30 My (30) of Earth_s formation. Our Compston, Geochim. Cos
codes a half transporter
13. G. Chang, J. Mol. Biol. 330, 419 (2003). that contains a single
behavior of the spin label at each site. Power sat- 31. P. Mitchell, Res. Microbiol. 141, 286 (1990). 178 (2001). (1992).
resultsgas, support the M. view that continental crust
A Homolog of the Multidrug 14. C. F. Higgins, K.membrane
15. T. W. Loo, D. M. Clarke,
MsbA J. is
Biol.
spanning
J. Linton, Science
Chem. 271, as
assembled
region
293, 1782
27482
a
fused withuration
(2001).
(1996).
homodimer
16. T. W. Loo, D. M. Clarke, J. Biol. Chem. 276, 31800 (2001).
with
a NBD.
a
curves were obtained under nitrogen
the presence of 20% oxygen, or underwas
total
in the presence of 50 mM NiEDDA. Distinct terpart
at least
nitrogen
patterns
in

that formed
32.
and a component
34. at
F. Rosenberg
33. P. C. Smith
E. F.È4.5
of the
Pettersen
Ga,etbut
et al., EMBO
enriched
et al., Mol. Cellcoun-
al., this
J. Comput.
J. 20, 5615

Chem. 25,
original
13. (2001).
10, 139 (2002).
S. A. Wilde, J. W. Valley, W. H. Peck, C. M. Graham,
Nature 409, 175 (2001).
14.1605 (2004). D. C. Lee, A. N. Halliday, R. T. Pidgeon,
Y. Amelin,
29.

30.
J. D. Vervoort, P. J. Patchett,
Earth Planet. Sci. Lett. 168,
M. Boyet, R. W. Carlson, S
17. M. Seigneuret, molecular mass ofJ.129.2Biol. kD. Hydropathy anal- accessibility are expected for
Resistance ATP Binding Cassette 278, 30115 (2003).
18. D. R. Stenham et
A. Garnier-Suillerot,

al., FASEB
with the NBD J. 17, located
2287 (2003).
Chem.
ysis indicates six membrane spanningbrane
on the cytoplasmic
of oxygen
regions
transmem-
crust was
segments, whereas exposure to the aqueous
side results in high NiEDDAbyaccessibility.
environments
largely
35. We thank D. Piston, A. Beth, C. Cobb, H. Koteiche,
recycled
the onset of the work Archean
back
was (G4
into the mantle
R. Nakamoto for critically reading the manuscript.
Ga). by a discovery grant
supported
Natureand399, 252 (1999).
This R. Rubatto, J. Metamorphic Geol. 21,
15. J. Hermann,
from
833 (2003).
31.
32.
33.
G. F. Davies, Geochim. Cosmo
M. Boyet et al., Earth Planet
G. Caro, B. Bourdon, J. L. B

(ABC) Transporters 19. Starting with a cysteine-less MsbA background

of the cell
the transmembrane
alanines, we constructed
membrane
where the two native cysteines were replaced with (17).
domainmu-
116 single-cysteine
The primary
Spin-label mobility was estimated from the inverse

is to recognize
role
22. J. Dong, and
of
of the central line width. References and Notes
H. S. Mchaourab, unpublished observations.
Vanderbilt University.

Supporting
1. J. Veizer, S. L. Jansen, Online
J. Geol. Material
87, 341 (1979).
16. Materials and methods are available as supporting
material on Science Online.
17. U. Soderlund, P. J. Patchett, J. D. Vervoort, C. E. Isachsen,
34.
423, 428 (2003).
We thank M. McCulloch, P.
T. Ireland, and S. Mussett
tants. Only fourtransport
mutationssubstrates
resulted inacross the lipid 23.
either weak G. J. Poelarends,
bilayer. The W. N. Konings, J. Biol.2.Chem. 277, S. www.sciencemag.org/cgi/content/full/308/5724/1023/
S. R. Taylor, M. McLennan, The Continental Crust: Earth Planet. Sci. Lett. 219, 311 (2004). tions to this project. The w
Geoffrey Chang* and Christopher B. Roth or no expression (51, 53, 282) or slow aggregation 42891 (2002). Its CompositionDC1 and Evolution (Blackwell Scientific, 18. J. Blichert-Toft, F. Albarède, Earth Planet. Sci. Lett. Australian National Univ
ABC, which is the
after reaction with the spin label (78). MsbA hallmark of the MDR-ABC Oxford, 1985). Materials and Methods
24. T. W. Loo, M. C. Bartlett, D. M. Clarke, Biochemistry 148, 243 (1997). search Council grant DP0
Multidrug resistance (MDR) is a serious medical problem and presents a major mutants were transporter
spin labeled after family Ni and is located
affinity chro- in the
43, NBD,
12081 (2004). 3. S. A. Bowring, Figs. S1 and S2
I. Williams, Contrib. Mineral. Petrol. 19. F. Albarède et al., Geochim. Cosmochim. Acta, in press. French Institut National
challenge to the treatment of disease and the development of novel thera- matography, purified
couples by size-exclusion
the energy of chromatog-
ATP hydrolysis 25. A. to
H. sub- 134, 3 (1999). Table S1
Buchaklian, A. L. Funk, C. S. Klug, Biochemistry 20. P. J. Patchett, J. D. Vervoort, U. Soderlund, V. J. M. and the Programme Natio
raphy (SEC) and then reconstituted in unilamellar 43, 8600 (2004). 4. M. T. McCulloch, References
V. C. and Notes Lithos 30, 237
Bennett, Salters, Earth Planet. Sci. Lett. 222, 29 (2004). to J.B.T. and F.A., and by
peutics. ABC transporters that are associated with multidrug resistance (MDR- strate translocation. Although
vesicles. Details of the structural and functional the NBD struc-
26. C. L. Reyes, G. Chang, Science 308, 1028(1993). (2005). 21. J. Blichert-Toft, unpublished data. NAG5-13497 to S.J.M.
ABC transporters) translocate hydrophobic drugs and lipids from the inner to analysis of thetures of the
mutants are histidine transporter 27.
in the supporting (HisP), the
W. T. Doerrler, H. S. Gibbons, C. R. Raetz,
5. J. D.J. Vervoort,
Biol. 21Blichert-Toft,
J. October 2004; accepted
Geochim. 22 February 2005
Cosmochim. 22. P. W. O. Hoskin, U. Schaltegger, Rev. Mineral. Geochem.
53, 27 (2003). Supporting Online Material
the outer leaflet of the cell membrane. To better elucidate the structural basis maltose
material available transporter
on Science Online. (MalK), the DNA Chem.repair
279, 45102 (2004). 10.1126/science.1106592
Acta 63, 533 (1999).
23. J. D. Vervoort, P. J. Patchett, Geochim. Cosmochim. www.sciencemag.org/cgi/conte
6. C. G. Chase, P. J. Patchett, Earth Planet. Sci. Lett. 91,
for the “flip-flop” mechanism of substrate movement across the lipid bilayer, enzyme (Rad50), and the branched-chain ami- 66 (1988). Acta 60, 3717 (1996). Materials and Methods
we have determined the structure of the lipid flippase MsbA from Escherichia no acid transporter from Methanococcus jann- 24. D. reveal
G. Pearson, G. M. Nowell, J. Petrol. 45, 439 (2004). Figs. S1 and S2
Structure of the ABC Transporter 7. J. Blichert-Toft,(EM)N. T. of other
Arndt, MDR
Earth Planet.ABC transporters,
Sci. Lett.

Downloaded from www.sciencemag.org on April 4, 2015

25. S. J. G. Galer, S. L. Goldstein, Geochim. Cosmochim. Tables S1 to S3
coli by x-ray crystallography to a resolution of 4.5 angstroms. MsbA is organized aschii (MJ1267) have been determined, the 171, 439 (1999).
that large conformational changes are Acta possible
55, 227 (1991). References
8. J. Blichert-Toft, N. T. Arndt, G. Gruau, Chem. Geol.
as a homodimer with each subunit containing six transmembrane !-helices and structural basis for substrate translocation
MsbA in Complex with ADPIVanadate 207, 261 (2004). in both the TMDs and NBDs (19–23). 26. E. B. Watson, T. M. Harrison, Science 308, 841
a nucleotide-binding domain. The asymmetric distribution of charged residues through the cell membrane is not clear (18–21). 9. R. L. Armstrong, Philos.Despite Trans.attempts
R. Soc. Londonto model (2005).
Ser. A the structural 25 July 2005; accepted 2 Nove
301, 443 (1981). 27. D. Trail, S. J. Mojzsis, T. M. Harrison, Geochim. Cosmochim. Published online 17 November
lining a central chamber suggests a general mechanism for the translocation of The structure of MsbA establishes the changes of MsbA and other MDR ABC Actatrans-
68, A743 (2004). 10.1126/science.1117926
and Lipopolysaccharide
10. R. L. Armstrong, Aust. J. Earth Sci. 38, 613 (1991).
substrate by MsbA and other MDR-ABC transporters. The structure of MsbA can general architecture of the MDR-ABC 11. W. Compston, R. porters, a detailed
T. Pidgeon, view
Nature 321, 766of(1986).
conformational rear-P. D. Kinny, I. S. Williams, D. O. Froude, W.
28. R. Maas, Include this information when
serve as a model for the MDR-ABC transporters that confer multidrug resistance transporter family, and facilitates our un- rangements during ATP hydrolysis and substrate
to cancer cells and infectious microorganisms. derstanding
Christopher of theL.fundamental
Reyes and Geoffrey flipping Chang* translocation has remained elusive (24). WhatRETRACTED 22 DECEMBER 2006; SEE LAST PAGE
mechanism that moves hydrophobic sub- exchanges two or more p
increasing incidence of multidrug re- strates, including amino acids, peptides, strates from
Select members of thethe inner totriphosphate
adenosine the outer mem- (ATP)–binding cassette (ABC) X-ray Structure of the EmrE
are the conformational
during the catalytic within
R E Pchanges
cycle?30 What
O R T S of the TMDs
residues
My (30) of Earth_s formation. Our
are ecule through a Bhydrop
12. S. J. Mojzsis, T. M. Harrison, R. T. Pidgeon, Nature 409, Compston
ance is a significant health problem that ions, sugars, toxins, lipids, and drugstransporter and brane leaflet. The protein sequence
family couple ATP binding and hydrolysis to substrate efflux and of Eco- involved in substrate binding and release? And 178 (2001). pathway (10, 11). (1992).
profoundly impacted the treatment of are implicated in a number of seriousconfer hu- multidrug
msbA is resistance.
36 and 32% Weidentical
have determined to the NHthe 2- x-ray structure of MsbA Multidrug Transporter in Complex
what specific role does
results support the view that continental crust
was at nucleotide
least a component binding andenriched coun-
of the
13. S. A. Wilde, J. W. Valley, W. H. Peck, C. M. Graham,
Nature 409, 175 (2001).
The general model 29. J. D. Vervoo
fo
Earth Plane
ctious diseases and cancer. The World man diseases, including cystic fibrosisinand complex terminal and COOH-terminal
with magnesium, adenosinehalves diphosphate,of hu- RETRACTED
and inorganic 22vanadate
DECEMBER 2006; hydrolysis SEEplay LAST duringPAGE
terpart thethatcatalytic
formed atcycle? È4.5 Ga,To but this original 14. Y. Amelin, D. C. Lee, A. N. Halliday, EmrE and other30.seconda
R. T. Pidgeon, M. Boyet,

lth Organization has recently reported several disorders of the immune system (MgIADPIV man MDR1, respectively R E P O R T S (Fig. 1) (22).
i) and the rough-chemotype lipopolysaccharide, Ra LPS. The with a Substrate
address these questions, crust was largely recycled back into the mantle
we describe the struc-
by the onset of the Archean (G4 Ga).
Nature 399, 252 (1999).
15. J. Hermann, R. Rubatto, J. Metamorphic
833 (2003).
alternating
model,
Geol. 21,
the EmrE
31. G. F. Davies
access
32. M.
33. G.
mecha
Boyet e
Caro, B
transpo
multidrug-resistant bacteria can ac- (5–7 ). structureHuman supports MDR3,a model which
10. involving
G. Reuteriset aal., phosphatidyl-
aJ. Biol.
rigid-body
Chem. 278, 35193 torque (2003).of the two
20. W. trans-
L. Hubbell, H. S. Mchaourab, C.ture of MsbA
Altenbach, M. A. from 28. A. L.Salmonella
Davidson, J. Bacteriol. typhimurium
184, 1225 (2002). 16. Materials and methods are available as supporting 423, 428 (
nt for up to 60% of all hospital-acquired MsbA is a member of the MDR-ABC choline flippase 11. H. W. van Veen, C. F. Higgins, W. N. Konings, Res.
and is 73% identical in Lietzow, Structure 4, 779 (1996). 29. I. L. Urbatsch, G. A. Tyndall, G. Tombline, A. E. Senior, material on * conformations, 34.inward-f
frequency Owen Pornillos,reagents Yen-Ju Chen, 278, Andy P. Chen, Geoffrey Chang
in complex with adenosine
References 5¶-diphosphate
and Notes Science Online. We thank
membrane domains during Microbiol. ATP hydrolysis
152, 365 (2001).and suggests a mechanism 21. The collision by with paramagnetic J. Biol. Chem. 23171 (2003). 17. U. Soderlund, P. J. Patchett, J. D. Vervoort, C. E. Isachsen, T. Ireland,
ctions globally (1). Multidrug resis- transporter group by sequence homologywhich and theprotein sequence12.toG.human MDR1, is 293,31% was deduced from the analysis and of theinorganic
saturation vanadate
1. J. Veizer, S. L. Jansen, J. Geol. 2þ, 87, 341 (1979). facing, with the drug-bind
2. S. (ADPIV ), McLennan,
Mg(1957). andContinental Crust:
Chang, C. B. Roth, Science 1793 (2001). 30. P. Mitchell, Nature 180, 134
nucleotide-binding domain communicates
13. G. Chang, J. Mol. Biol. 330, 419 (2003).
with the transmembrane behavior of the spin label at each site. Power sat-
R. Taylor, S. iM.
31. P. Mitchell, Res. Microbiol.
The
141, 286 (1990).
Earth Planet. Sci. Lett. 219, 311 (2004). tions to th
Scientific, of18. J. Blichert-Toft, F. Albarède, Earththe cytoplasm
Planet. Sci. Lett. or periplas
Australian
e in the treatment of cancer is respon- is more closely related to the mammalian domain. identical
We propose in protein
a lipid 14. C.sequence
F. Higgins, K. J.to
‘‘flip-flop’’ Eco-msbA
mechanism
Linton, Science 293, in1782
which(2001).the sugar curves were EmrE
urationgroups obtained is a prototype
under nitrogen gas, in of 32.
rough-chemotype the Small
M.(Ra)
F. Rosenberg Multidrug
Its Composition
et al., EMBO Resistance
lipopolysaccharide
Oxford, 1985).
and Evolution (Blackwell family
J. 20, 5615 (2001). efflux
148, 243
trans-
(1997). search Cou
e for tens of thousands of deaths per P-glycoproteins than any other bacterial are (23). Theinsimilarity
ABCsequestered in protein sequence and
15. T. W. Loo, D. M. Clarke, J. Biol. Chem. 271, 27482 (1996). the presence of 20%portersoxygen, orand
underactively
(Ra nitrogen andexpels
LPS). The 33. P.positively
C.
structure Smith et al.,charged
provides Mol. Cell 10, hydrophobic
139
evidence (2002). drugs across the
19. F. Albarède inner conversion
et al., Geochim. Cosmochim. Acta, in press.between theIns
French
the chamber
16. T. W. Loo, D.while
M. Clarke, the hydrophobic
J. Biol. Chem. 276, 31800 (2001).tails are in thedragged
presence of 50 mM NiEDDA. Distinct patterns
3. S.
34. E. F. Pettersen
A. Bowring,
et al., J.
I. Williams,
Comput.
Contrib.
Chem. 25,
Mineral.
1605
Petrol.
(2004). 20. P. J. Patchett, J. D. Vervoort, U. Soderlund, V. J. M. and the Pr
function membrane of Escherichia 35.coli.
134, 3 (1999).
Here, we A.report the x-raya Lithoscrystal
and 237 structure, at 3.7 promoted by drug and/or
. Multidrug resistance can be conferred transporter (8, 9). Although LmrA fromthrough Lac- the lipid between
bilayer. 17.MsbA and human
M. Seigneuret, MDR1/J. Biol. Chem.
A. Garnier-Suillerot, of oxygen accessibility forfor
are expected antransmem-
intermediate Weafter M. ATP
4.thank D. Piston,
T. hydrolysis
McCulloch, Beth, and
V. C.C.Cobb, H.
Bennett, Koteiche, 30, Salters, Earth Planet. Sci. Lett. 222, 29 (2004). to J.B.T. a
278, 30115 (2003). brane segments, whereas exposure to the aqueous
angstrommolecular resolution, of for R. Nakamoto
one for
conformational critically reading the
state ofmanuscript.
the This
EmrE transporter
21. J. Blichert-Toft,in unpublished data. we describe the x-ray NAG5-134cr
a number of transporters that pump tococcus lactis is functionally more similar to MDR3 suggests 18.a D.common R. Stenham et al., evolutionary
FASEB J. 17, 2287 (2003). environments results in high NiEDDA basis
accessibility. work
(1993).
coupling
5. J.was supportedATP by hydrolysis
a discovery grantCosmochim.
from 22. P. W. O. Hoskin, U. Schaltegger, Rev. Mineral. Geochem.
complex with theainverse
translocation
D. Vervoort,
substrate,
J. Blichert-Toft, Geochim. conformation Supporting of the On E
gs out of cells. Certain multidrug resis- the P-glycoproteins, MsbA is evenMultidrug more con- origin
resistance is anand, therefore,
alarming 19.
and they
Starting
rapidly withmay have
a cysteine-less
MsbA com- MsbA
is background
an essential Spin-label ABC
bacterial mobility trans-
was estimatedwith
from amphipathic Vanderbilt
substrate Acta 533 (1999).tetraphenylphosphonium. 53,
University.
63,transport. Two EmrE
27 (2003).
where the two native cysteines were replaced with of the central line width.
polypeptides form a homodimeric 6. C. G. Chase, transporter
P. J. Patchett, Earth that Planet.binds substrate
Sci. Lett. 91, 23. J. D. Vervoort, P. J. Patchett, Geochim.
at the di- complex Cosmochim. with www.sciencem
the drug
e transporters such as the human P- served by sequence homologygrowing (10, 11). obstaclemon in the mechanisms
treatment of by which
infectious
alanines, they
we constructedporter catalyze
116that transports
single-cysteine mu- lipid22. J. A
Dong,and lipopoly-unpublished Crystals
H. S. Mchaourab, observations. ofSupporting
MsbA 66Online in Material
complex with Acta 60, 3717 (1996). Materials and M
oprotein, MDR1, can transport a di-
e class of amphipathic drug molecules
Geoffrey Chang
MDR-ABCRETRACTED transporters have
to act as “hydrophobic vacuum cleaners”
22 DECEMBER
been proposed
diseases,
malaria, and be- cancer
the flipping
ture(1).
LASTofPAGE
2006;immunodeficiency
human SEE
of Drug-resistant
virus
Eco-msbA determined
tants. Only four mutations resulted in either weak
substrates. (HIV),
bacterial
after
The crystal
or no expression
reaction with
saccharide
(51, 53, 282)
to
and 4.5
the that
struc-
spinÅ has
(LPS) to the outer
or slow aggregation
in been
label
23. G. J. Poelarends, W. merization
membrane (7–10)
42891 (2002).
shown24.toT. have
(78). MsbA W. Loo, overlapping
N. Konings, J. Biol.
brane and
interface.
MgIADPIV Chem. 277, The
adopt
i andDC1
slightly
detergent-solubilized
M. C. Bartlett, D. M. Clarke, Biochemistry
two
Ra 7.LPS
different
Materials
(1988).
J.subunits were
Blichert-Toft,
171, 439 (1999).
protein
8. and Methods folds,
incubated
J. Blichert-Toft,
have
www.sciencemag.org/cgi/content/full/308/5724/1023/
grown
N. opposite
T. Arndt,
forming
N. T. Arndt,
using
Earth Planet.

withG. Ra an
orientations
Sci. Lett.
asymmetric
Gruau, Chem. Geol.
inD.the
24.
25. S. J. G.
antiparallel
mem-
G. Pearson,
Galer,
Acta 55, 227 (1991).
S. L. Goldstein,
was
G. M. Nowell, J. Petrol. 45, determined
Geochim.
lous
439 (2004).
Cosmochim.
dispersion
to S1
Figs.
Tables
3.7and) S2
S1 to S3
methods
References

andcauseresolution
gonorrhea,provides
mutants were spin labeled after Ni affinity chro-
a framework
43, 12081 (2004). dimer. This unusual architecture Figs. S1 and 207, S2likely
261 (2004). confers unidirectionality to26.transport
E. B. Watson, byT. M. Harrison, analog
Science 308, of 841 TPP and seleno
There is evidence that some of these cause of their ability to removestrains lipidsthat byfor deci- 2þ,
pneumonia, chol-purifiedsubstrate
matography, specificity
size-exclusion chromatog-with the25. A. H. Buchaklian, A. L. Funk, C. S.LPS
multidrug-resistant purified from
Klug, Biochemistry S.S1typhimurium.
Table 9. R. L. Armstrong, Philos. ATP,Trans. Mg R. Soc. London Ser. A (2005). 25 July 2005; a
g transporters may act as phospholipid drugs from the inner membrane leaflet era, and(12). phering
tuberculosis are P-glycoproteins
widespread raphy and(SEC) and
dif- and then
suggests
ABC (MDRaand
reconstituted inABC)
unilamellar
transporter 43, 8600
LmrA (2004).
and creating
with an boiled
and asymmetric sodium substrate
References and 443
orthovanadate
301, Notes translocation
(1981). were added pathway. On 27. theD. Trail,
basis of T. M. Harrison, Geochim. Cosmochim.proteins
S. J. Mojzsis,
substituted (Fig.
Published onlin
RESEARCH ARTICLE vesicles. Details of the structural functional 26. C. L. Reyes, G. Chang, Science
available 308, 1028
structural (2005). data, we 10. propose
R. L. Armstrong, a model
Aust. J. EarthforSci. the
38, 613 proton-dependent
(1991). Acta 68, A743
drug (2004). SeMet-labeled prot
10.1126/scienc
pases and it has been proposed that MsbA transports lipid A, a majorficult componentto treat (2). general mechanism
In humans, a similar for the
analysis oftransport
drug the mutantshuman ofareP-glycoprotein
sub-
in the supporting(P-gp) (11).
27. W. MsbAH.adeno-
T. Doerrler, S. Gibbons, C.toR. favor
Raetz, J.the
Biol.transition
21 October
11. W. state
2004; conformation
accepted
Compston, R. T.22 February
Pidgeon, be-
2005321, 766 (1986).
Nature 28. R. Maas, P. D. Kinny, I. S. Williams, D. O. Froude, W.
study were
Include this inf
produced in
coproteins, material
which have available on Science fused
these components Online.
(ATPase)Chem. efflux mechanism
279, 45102 (2004). of EmrE. 10.1126/science.1106592
tidrug transporters may function as of the bacterial outer cell membrane, effluxand is
mechanism strateis aacross
major the lipid
reason bilayer.
for the sine triphosphatase hydrolysis is stim- fore crystallization (25). Mass spectrometry
g flippases, translocating drugs from the Structure of MsbA from E. coli:
the only bacterial ABC transporter failurethat is
of several Structure
chemotherapeutics
into a single polypeptide, the msbA gene en-
codes determination.
intransporter
a half the treat-that contains Membrane
ulated by LPS and lipid A and A
a single also
major showsobstacle ontowashedeffectivecrystals
treatment indicated of bac- the presence
classes ofoftransporters: ATP-binding cassette
because SeMet-EmrE di
vivo. Briefly, EmrE exchanges wastw
r to the outer leaflet of the lipid bilayer. essential for cell viability (13). Loss
A Homolog of the Multidrug ment of cancers.protein
of MsbA Foundx-ray crystallography
ubiquitously in bothregion of transportersStructure of the ABC Transporter
vanadate-inhibited activity (12). LPS makes Ra LPS,
(EM) of other
nucleotide, and
MDR ABC
vanadate. X-ray Structure of the EmrE
The
transporters,
struc-
reveal

Downloaded from www.sciencemag.org on April 4, 2015

membrane spanning fused with a NBD. eculeinthroug
MsbA is assembled as a homodimer with
terial infections is the emergence of
that strains
large that
conformational (ABC)changes proteins
are directly
possible couple drug efflux to the T7 promoter E.
an effort to understand the structural from the cell membrane or a mutation bacteria that and ion
and humans, ABC channels
transporterspresents havenewupchallenges the aoutertotal leaflet of the outer membrane in ture wasantibiotics.determined by single-wavelength pathway (10
MsbA in Complex with ADPIVanadate
in bothOf the TMDs and adenosine
NBDs (19–23).
s of multidrug resistance, we have de- Resistance ATP Binding Cassette
disrupts transport results in a lethal implicated owing
beenaccumu- the
molecular mass of 129.2 kD. Hydropathy anal-
in bothto antibiotic
ysisdisorder andsix
indicates caused
cancer
membrane by detergent
Gram-negative
spanning and bacteria and potently
regions
are resistant to available
ularactivates
concern areanomalous
multidrug-resistantdispersion Despite (SAD),
strains
partic-
thatandtothe
attempts Multidrug Transporter in Complex
whereas electron
model the
5¶-triphosphate (ATP) hydrolysis,
structural transporters use the energy
secondary
mented with nucleotid
polymerase, and
The
EmrEappro
gene
and o
mined the crystal structure of the multi- lation of lipid A in the cytoplasmic leaflet and
drug resistance the represent
inherent with
key thetargets
NBD located
movement for of on the
the cytoplasmic
transmembrane
the mammalian side innate immune cause system in density maps changes
were of MsbA and
improved by other usingMDR non- ABC trans-
g resistance transporter homolog, (ABC) Transporters
(14, 15). Several bacterial homologs of msbA development of agents
!-helices. the to of the cell membrane (17). The primary role of
We
reverse have, therefore,
multidrug adoptedand a
response and Lipopolysaccharide
to bacterial infections; it can
common
cause
diseases
crystallographic
gonorrhea, and hospital-acquired
such as tuberculosis,
symmetry
staphylococ- averaging with a Substrate
derived
porters, a detailed view of conformational rear-
to a
from
res-
proton or cation electrochemical (14). Experimental
As data are very
mapsa
alternating
well
transmembrane domain is to recognize rangements during ATPgradients hydrolysis and across the lipid bilayer.
substrate model, the
A, from Escherichia coli (Eco-msbA). thatGeoffrey
includeChang*the flippase ValA from
and Christopher resistance
B. RothFran- (3, 4).strategy
Severaloftransport
MDRrapidly ABC exploring
substrates across thecrystallization
efflux septic
lipid bilayer.shockThe (13–15). ABC transporters cal Chang are
infections
*
olution
(1). Multidrug ) (see arises,
of 4.2resistance
translocation tableOwen
has S1)Pornillos,
remained (26).
EmrE
elusive The is(24).
Yen-Jua proton-dependent
WhatChen, Andy P.secondary Chen, Geoffrey trans- Chang that * in vitro– conformatio
and in v
ABC, whichoverexpressing,
is the lipids
hallmark of the
Christopher
MDR-ABC
L. Reyes and Geoffrey facing, with
msbA gene product belongs to a super- cisella novicida and LmrA, have been pumps have beenspace
report- shown bytocloning,
extrude both and puri-composed of two transmembrane
minimally in part, throughasymmetric the action unit revealed
of are
integral two dimers
mem-
the conformational porterof MsbA
changes from
of the TMDs
Escherichia coli and is a prototype of proteins adopt the a similar
Multidrug resistance (MDR) is a serious medical problem and presents a major transporter family and is located in the NBD, EmrE is acycle?prototype ofresidues
the Small Multidrug Resistance family of efflux trans- cytoplas
ily of transporters that contain an aden- challenge ed in treatment
to the over 30 ofdivergent
disease andprokaryotic and
the development species
drug fying morecouples
molecules,
of novel thera- which than 20
suggests the full-length
energy of ATPbacterial
a common
Select hydrolysis totrans-
members domains of the (TMDs)
sub- adenosine that encode
triphosphate substrate
(ATP)–binding speci-cassette with (ABC) clear
brane proteins called multidrug transporters electronduring density
the corresponding
catalytic
porters and
the
actively
What
Small to
expels
a
Multidrug are
positively
Resistance (SMR) family
charged hydrophobic drugs across the inner
shows that cell-free
conversion mb
transporter family couple ATP binding and hydrolysis to substrate efflux and involved in substrate binding and release? And
e triphosphate (ATP) binding cassette (16).
peutics. ABC transport
transporters that are associated with multidrug resistance portersfor
mechanism
(MDR- andstratetheirtranslocation.
amphipathichomologs com- derived
Although the from
ficity
NBD and sev-
struc- a pair of nucleotide-binding (1, domains
2). Each of nucleotide
these transporters and Racan
what
LPS. The TMDs
actively
specific membrane
role does
(3,
of in SMRs
4). each coli.
Escherichia
nucleotide binding
represent
Here,
and we the smallest
report the transporters
x-ray crystal structure,alternative
at 3.7 to traditional
promoted byl
ABC transporters) translocate hydrophobic drugs andpounds lipids from the inner to tures of (5,the 6). confertransporter
histidine multidrug resistance.
(HisP), the We have determined the x-ray structure of MsbA
C), which is also called a nucleotide- The across eral
the cellMDR-ABC
membrane transporter families (NBDs) and 12 conserved
with structural features. dimer of exhibit
drugsaand 30- toxictorque relative to nature;
the mo-
of overall organization of MsbA
elucidateisthe
consis- expel a wide variety com-
angstrom in each polypeptide has only of the 105 EmrEto pression in systems. we describe
the outer leaflet the cell membrane. To better structural basis maltose transporter in complex
(MalK), the with
DNA magnesium,
repair adenosine diphosphate, and inorganic vanadate hydrolysis play duringresolution,
the catalytic of cycle?
one conformational
To state transporter
Comparison of the x-ray structures of MsbA lecular two-fold axis and complex
an with
extensive a translocation
inter- substrate, tetraphenylphosphonium. Two EmrE conformatio
ding domain (NBD) (3, 4 ). ABC trans- tent with
for the “flip-flop” most bacterial
mechanism MDR-ABC
of substrate movement across transport- bacterial species
the lipid bilayer, (24). (MgIADPIV
enzyme (Rad50), Ourtheexpectation
and branched-chain was
ami-
i) and the rough-chemotype lipopolysaccharide, Ra LPS. The
pounds from the cell. There are two broad 120
address these questions, we describe the struc- amino acid residues and four transmem- Consistent with bioch
complex wi
we have determined digitation of the helices polypeptides
(Fig. 1, A form
and a
B). homodimeric
A transporter that binds substrate at the di-
ers translocate a wide variety of sub- ers and itstheamino
structure of the
acid lipid flippase
sequence MsbA from Escherichia
isDepartment
remarkably that one orno more
acid transporter
of from Methanococcus
these
structure natural and
supports the a vitamin
jann-
variants model B
involving ABC
a importer,
rigid-body torqueBtuCD,
of the two trans- ture of MsbA from brane
Salmonella helices,
typhimurium and forms homo- or hetero- ing that EmrE is primarily
was determi
coli by x-ray crystallography to a resolution of 4.5 angstroms. MsbA isoforganizedMolecular Biology, The Scripps
(MJ1267)Research
12 merization interface.5¶-diphosphate
The two subunits have opposite orientations in the mem-
aschii have been determined,
membrane domains the
during ATP hydrolysis and suggests a mechanism by model in complex with adenosine
similar
as a homodimer withtoeach
several
subunitmammalian P-glycoproteins
Institute, 10550
containing six transmembrane !-helices would
North
and Torrey be Pines
more optimal
Road
structural CB105,
basis forfor La protein
substrate
suggests expres-
translocation
that differences in substrate specific-
Department
chemical
of Molecular Biology, Theand
with
Scripps
good geometry
brane
Research and oligomers
adoptwasslightly
built(3). EmrEfolds,
different
2þ, and
is wellforming documented
an asymmetric and binds drugslous
to antiparallel with a2
dispers
which the nucleotide-binding domain communicates with the transmembrane inorganic vanadate (ADPIV ), Mg
a involved in multidrug resistance. All known
nucleotide-binding
rtment of Molecular Biology, MB-9, The Scripps domain. The asymmetric Jolla,
distribution CA
of 92137,
charged sion, purification, anddomain.
USA.
residues through the cell membranecrystalis not ities
clear are
formation.
(18–21). a consequence of structurally Institute,diver-
10550 with
North R
Torrey of 28% and
cryst Pines rough-chemotype
Road, CB-105, R dimer. of 33%.
This function
unusual
free La (Ra) lipopolysaccharide
i as a homooligomer (5–9) and confers
architecture likely confers unidirectionality to (9,
transport 15),
by the asymmetric
analog of TPu
lining a central chamber suggests a general mechanism for the translocation of The structure of MsbA
Weestablishes
propose athelipid ‘‘flip-flop’’ mechanism in which the sugar groups substituted p
arch Institute, La Jolla, CA 92037, USA. ABC transporters are composed of
*Tofourwhom mod- Each
correspondence full-length
should MDR-ABC
be addressed. gent
transporter
are sequestered TMDs was (16–18). These structures,
in the chamber while the hydrophobic tails are dragged
Jolla, CA along
92037, USA. In this structure, (Ra each
LPS). creating
Thedimer an
structure resistance
asymmetric
contains providestwo to positively
substrate
evidence charged
translocation hydrophobic
pathway. On the basiscrystal of is composed of tw
substrate by MsbA and other MDR-ABC transporters. The structure of MsbA can general architecture of the MDR-ABC available structural data,such we propose a model forethidium, the proton-dependent drug SeMet-l
cloned andtransporter recombinantly and expressed
with thoseinun- a derived from electron microscopy bound LPS molecules located at antibiotics,
the protein- as tetracycline, and and one molecule of TP
E- as a ules,
modelincluding two transporters
membranethat spanning regionsresistance
E-mail: [email protected] through the lipid our bilayer. *To whom correspondence should be addressed. for an intermediate after ATP hydrolysis and a
whom correspondence should be addressed.serve for the MDR-ABC confer multidrug family, facilitates efflux mechanism of EmrE. study were
[email protected] to cancer and
cells two NBDs. microorganisms.
and infectious Unlike the mammalian P-gly- BL21 strain of E. coli
derstanding of the (25). Althoughflipping
fundamental func- E-mail: [email protected] molecular basis for coupling tetraphenylphosphonium
ATP hydrolysis (TPP) (3, 4). EmrE imally functional becauseunitSeM o
mechanism Multidrug resistance
that moves is an alarming
hydrophobic sub- and rapidly MsbA is an essential bacterial ABC trans- with A
amphipathic substrate
major obstacle transport.
to effective treatment of bac- classes of transporters: ATP-binding cassette vivo. Briefl
The increasing incidence of multidrug re- 1028
strates, including amino acids, peptides, strates from
growing 13 MAY
the inner to the
obstacle 2005
outer
in the VOL
mem-
treatment 308 SCIENCE
of infectious www.sciencemag.org
porter that transports lipid A and lipopoly- Crystals of MsbA
terial infections in complex
is the emergence with that
of strains (ABC) proteins directly couple drug efflux to the T7 pro
sistance is a significant health problem
www.sciencemag.org that
SCIENCE ions,
VOLsugars,
293 toxins, lipids, and drugs
7 SEPTEMBER 2001 and brane leaflet. The human
diseases, sequence of Eco- 1793
protein immunodeficiency virus (HIV), 1950(LPS) to the outer membrane (7–10)
saccharide MgIADPIV 23i and
DECEMBER
are resistant Raavailable
to
2005
LPS were grownVOL
antibiotics.
310 SCIENCE
using
Of partic-
www.sciencemag.org
adenosine 5¶-triphosphate (ATP) hydrolysis, mented wit
has profoundly impacted the treatment of are implicated in a number of serious hu- msbA is malaria,
36 and 32%and identical
cancer (1). the NH2-
to Drug-resistant bacterial and that has been shown to have overlapping detergent-solubilized
ular concern are protein incubated with
multidrug-resistant Ra that
strains whereas secondary transporters use the energy polymerase
infectious diseases and cancer. The World man diseases, including cystic fibrosis and terminal strains
and COOH-terminal halves ofpneumonia,
hu- LPS purified from S. typhimurium. Mg2þ,
that cause gonorrhea, chol- substrate specificity with the multidrug-resistant cause common diseases such ATP,
as tuberculosis, derived from proton or cation electrochemical (14). Experi
Health Organization has recently reported several disorders of the immune system man MDR1, respectively (Fig. 1) (22). and boiled sodium
that multidrug-resistant bacteria can ac- (5–7 ).
era, and tuberculosis are widespread and dif-
Human MDR3, which is a phosphatidyl-
ABC (MDR ABC) transporter LmrA and with gonorrhea, andorthovanadate were
hospital-acquired added
staphylococ- gradients across the lipid bilayer. As data are
count for up to 60% of all hospital-acquired
infections globally (1). Multidrug resis-
tance in the treatment of cancer is respon-
MsbA is a member of the MDR-ABC
transporter group by sequence homology and
is more closely related to the mammalian
ficult to treat (2). In humans, a similar drug
choline flippase and is 73% identical in
efflux mechanism
protein sequence
identical failure
is a major
to human MDR1,
of several
in protein
is 31%
chemotherapeutics
sequence
reason for the
to Eco-msbA in the treat-
human P-glycoprotein (P-gp) (11). MsbA adeno-
sine triphosphatase (ATPase) hydrolysis is stim-
ulated by LPS and lipid A and also shows
to favor
fore in
on washed
the transition
cal infections
crystallization
brane proteins
(1). state
part, through(25).
conformation
Multidrug
Mass of
the action
crystals indicated
resistancebe-
spectrometry
called multidrug
arises,
integral mem-
the presence of
transporters
EmrE is a proton-dependent secondary trans-
porter from Escherichia coli and is a prototype of
the Small Multidrug Resistance (SMR) family
poldracklab.org that in vitr
proteins ado
shows that
 LETTERS

Downloaded from www.sciencemag.org o

ctures (1, 2, 5) were incorrect in both the hand of the struc- caused and, in particular, subsequent research efforts that were unpro-

Threats to reproducibilty: software errors

e topology. Thus, our biological interpretations based on
ed models for MsbA are invalid.
ouse data reduction program introduced a change in sign for
edited by EttawhichKavanagh
ductive as a result of our original findings.
GEOFFREY CHANG, CHRISTOPHER B. ROTH,
CHRISTOPHER L. REYES, OWEN PORNILLOS,
differences. This program, was not part of a conven- YEN-JU CHEN, ANDY P. CHEN
processing package, converted the anomalous pairs (I+ and Department of Molecular Biology, The Scripps Research Institute, La Jolla, CA 92037, USA.
nd F+), thereby introducing a sign change. As the diffrac-
References
ollected for each set of MsbA crystals and for the EmrE
Retraction
re processed with the same program, the structures reported
6) had the wrong hand.
1.
2.
3.
G. Chang, C. B. Roth, Science 293, 1793 (2001).
C. L. Reyes, G. Chang, Science 308, 1028 (2005).
tron density for the connecting loop regions. Un
the multicopy refinement procedure still allowe
O. Pornillos, Y.-J. Chen, A. P. Chen, G. Chang, Science 310, 1950 (2005).
4. R. J. Dawson, K. P. Locher, Nature 443, 180 (2006).
or in the topology of the original MsbA structure was a con- 5. G. Chang, J. Mol. Biol. 330, 419 (2003). able refinement values for the wrong structures
f the low resolution of the data as well as breaks in the elec- 6. C. Ma, G. Chang, Proc. Natl. Acad. Sci. U.S.A. 101, 2852 (2004).
WE WISH TO RETRACT OUR RESEARCH ARTICLE “STRUCTURE OF The Protein Data Bank (PDB) files 1JSQ
MsbA from E. coli: A homolog of the multidrug resistance ATP bind- MsbA and 1S7B and 2F2M for EmrE have bee
ulture ing in cassette (ABC) operations that reveal little, if any, negative industry governed by regulations with a rational
transporters” and both
impact on the environment of our
or local Reports
ecosys- basis in“Structure
the ecology of of the oceans ofand obsolete
the eco- PDB entries. The MsbA and Em
e Zones the ABC transporter tems MsbA in complex
(2, 3). Naylor criticizes the with
NationalADP•vanadate
nomic realities of theand marketplace. recalculated from the original data using the pro
Offshore Aquaculture Act of 2005 because CLIFFORD A. GOUDEY
RIAL BY lipopolysaccharide”
ROSAMOND NAYLOR, it and lacks “X-ray structure ofstandards.
specific environmental the EmrEMassachusetts
multidrug trans- alous differences, and the new Ca coordinate
Institute of Technology, Cambridge, MA
porter
quaculture in complex
legislation” (8 Sept., with a substrate”
Yet, she recommends (1–3).California’s recent 02139, USA. will be deposited.
uggests that the motivation for Sustainable Oceans Act as a legislative
The recently reported structure of Sav1866 (4) indicated
uaculture into the open ocean is model, although it is similarly silent, leaving
References that our
and Notes We very sincerely regret the confusion th
1. The SeaStead project a decade ago, four miles off
MsbAnear
ne fish farming structures
the shore (1, 2, 5)
those were
details incorrectininresponse
to rule-making both the to hand of the(seestruc-
Massachusetts caused and, in particular, subsequent research e
www.nmfs.noaa.gov/mb/sk/
by state regulations.” Although the best available science. saltonstallken/enhancement.htm) and the recent
ture and the
e regulations may exist in a few
topology. Thus, our biological interpretations
Naylor criticizes the use of fishmeal as
basedConsortium
Offshore Aquaculture on experimental
ductive cageas a result of our original findings.
operation 22 miles off Mississippi (see www.masgc.
he largerthese
schemeinverted models
this is irrele- for MsbA
an aquaculture are invalid.
ingredient, ignoring the fact org/oac/). GEOFFREY CHANG
e offshore aquaculture
An in-houseprojects
datathat industrial fisheries
reduction program are introduced
well managed a change 2. See www.lib.noaa.gov/docaqua/reports_noaaresearch/
in sign for CHRISTOPHER L. R
nder way, none are occurring in and would occur with or without aquacul- hooarrprept.htm/.
anomalous
clusive Economic Zone differences.
(EEZ); ture’s This
demand. program, which
Naylor ignores the was
highernot part of a conven-
3. See www.blackpearlsinc.com/PDF/hoarpi.pdf. YEN
4. See www.salmonoftheamericas.com/env_food.html.
y are happening
tionalindata stateprocessing
waters. efficiency
package,of using fishmeal tothe
converted feed fish
anomalous pairs
5. D. Pauly, (I+ and
V. Christensen, Nature 374, 255 (2002).
Department of Molecular Biology, The Scripps Research Instit
orically, only two aquaculture compared with its use in land-based live-
ave everI-) to (F-inand
occurred F+),stock
federal thereby introducing
operations a signischange.
(4). Also ignored the IN HER AsPROVOCATIVE
the diffrac- EDITORIAL “OFFSHORE
References
tion data collectedinefficiency for eachofset of small
using MsbA crystals
pelagic fish in and for the
aquaculture EmrE (8 Sept.,
legislation” p. 1363),
f Naylor’s stated concern over the natural setting to feed predator fish (5). R. Naylor raises valid points regardingChang,1. G. regu- C. B. Roth, Science 293, 1793 (2001).
uaculturecrystals
is based onwere processed
historical with the
Researchers andsame program,
entrepreneurs the structures
currently lation of oceanic reported aquaculture,2.since C. L. itReyes,
is G. Chang, Science 308, 1028 (2005).
in (1–3,
with near-shore fish5, 6) had
farms. This the wrongthehand.
developing technologies needed for offshore sure to grow in the future because 3. O.of Pornillos,
dwin- Y.-J. Chen, A. P. Chen, G. Chang, Science 310,
f years of more relevant offshore aquaculture share a vision of a well-managed dling global fishery supplies. This 4. growth
R. J. Dawson,
is K. P. Locher, Nature 443, 180 (2006).
The error in the topology of the original MsbA structure was a con- 5. G. Chang, J. Mol. Biol. 330, 419 (2003).
sequence of the low resolution
www.sciencemag.org of theVOL
SCIENCE data314as well as breaks
22 DECEMBER 2006in the elec- 6. C. Ma, G. 1875Chang, Proc. Natl. Acad. Sci. U.S.A. 101, 2852 (2

Published by AAAS

Aquaculture in operations that reveal little, if any, negative

impact on the environment or local ecosys-
industry governed by re
basis in the ecology of
poldracklab.org
Small errors can have big effects
# 23-class classification problem

skf=StratifiedKFold(labels,8)

if trainsvm:
pred=N.zeros(len(labels))
for train,test in skf:
clf=LinearSVC()
clf.fit(data[train],labels[train]) data[:,train]
pred[test]=clf.predict(data[test]) data[:,test]

Results:
Results:
53% accuracy
93% accuracy

http://www.russpoldrack.org/2013/02/anatomy-of-coding-error.html
poldracklab.org
Solutions for reproducible research

• Statistical power

• Openness and transparency

• Planning and documentation

• Publication of positive and negative results

• Focus on generalization rather than statistical

significance

poldracklab.org
 Center for Reproducible
Stanford | Neuroscience

Analyzing for reproducibility

Imaging High-performance computing

Metadata
data

Dataset
Workflow 1

Workflow 2
Training
OpenfMRI
Workflow n

Test
Test out-of-sample
reproducibility
Conclusions

• fMRI has come a long way in 20 years, but we have a

long way to go

• There are many threats to research quality and

reproducibility that arise from daily research
practices in neuroimaging

• “The first principle is that you must not fool yourself

and you are the easiest person to fool”
- R. Feynman

poldracklab.org
Acknowledgments
Jeanette Mumford
Sarah Helfinstein
Tom Schonberg
Craig Fox
Koji Jimura
Sanmi Koyejo
Chris Gorgolewski
Tyler Davis
Tal Yarkoni
Jessica Cohen
Robert Bilder
Eliza Congdon James S. McDonnell
Eydie London Foundation
Tyrone Cannon
Nelson Freimer
Stephen J. Hanson
Yaroslav Halchenko The Poldrack Lab @ Stanford

http://reproducibility.stanford.edu

Data sets and code will be made available at www.openfmri.org

poldracklab.org