ORIGINAL ARTICLE

Cognitive Improvement After Treatment

With Second-Generation Antipsychotic Medications
in First-Episode Schizophrenia
Is It a Practice Effect?
Terry E. Goldberg, PhD; Robert S. Goldman, PhD; Katherine E. Burdick, PhD; Anil K. Malhotra, MD; Todd Lencz, PhD;
Raman C. Patel, MD; Margaret G. Woerner, PhD; Nina R. Schooler, PhD; John M. Kane, MD; Delbert G. Robinson, MD

Context: Cognitive impairment in schizophrenia is fre- Results: No differential drug effects were observed.
quent, involves multiple domains, and is enduring. Nu- Of 16 cognitive measures, 9 demonstrated improve-
merous recent clinical trials have suggested that second- ment over time and only 2 demonstrated greater rates
generation antipsychotic medications significantly of change than those observed in the healthy control
enhance cognition in schizophrenia. However, none of group undergoing repeated assessment. The composite
these studies included healthy controls undergoing re- effect size for cognitive change was 0.33 in the healthy
peated testing to assess the possibility that improve- control group (attributed to practice) and 0.36 in the
ments might reflect simple practice effects. patients with first-episode schizophrenia. Improve-
ments in cognition in the first-episode schizophrenia
Objective: To report the results on cognition of a ran-
group could not be accounted for by medication dose,
domized comparison of 2 widely prescribed second- demographic variables, or intellectual level.
generation antipsychotic medications, olanzapine and ris-
peridone, in patients with first-episode schizophrenia and
Conclusions: The cognitive improvements observed
a healthy control group.
in the trial were consistent in magnitude with practice
Design: Randomized clinical trial.
effects observed in healthy controls, suggesting that
some of the improvements in cognition in the first-
Setting: Hospital-based research units. episode schizophrenia group may have been due to
practice effects (ie, exposure, familiarity, and/or
Patients: A total of 104 participants with first-episode procedural learning). Our results also indicated that
schizophrenia and 84 healthy controls. differential medication effects on cognition were
small. We believe that these findings have important
Main Outcome Measures: Cognitive assessment of implications for drug discovery and the design of reg-
all study participants occurred at baseline, 6 weeks later, istration trials that attempt to demonstrate cognitive
and 16 weeks later. Neurocognitive tests included mea- enhancement.
sures of working memory and attention, speed, motor
function, episodic memory, and executive function. Arch Gen Psychiatry. 2007;64(10):1115-1122

C
OGNITIVE IMPAIRMENT IN ers associated with chronicity (patient role,
schizophrenia is fre- institutionalization, interactions with ag-
quent, involves multiple ing, or disease processes) are minimized.
domains of information Naturalistic studies of patients with FES
processing, and may be a demonstrate that they have substantial
core feature of the disorder.1-4 Thus, neu- neurocognitive impairments of −1.0 to −2.0
rocognition has come to be viewed as a key SDs below average across a wide range of
target in clinical trials.5 Patients with first- domains, including working memory, at-
episode schizophrenia (FES) may be an es- tention, processing speed, and episodic
pecially important group for such studies memory.6-8 Longitudinal improvements are
Author Affiliations: Division of because they have demonstrable plasticity often considered modest after treatment
Psychiatry Research, Zucker in symptomatic response to antipsychot- in patients taking antipsychotic agents who
Hillside Hospital/Albert
Einstein College of Medicine,
ics, can be tested while drug naive, and do are followed up for several years.9-12 Sev-
Glen Oaks, New York. not have long histories of multiple antipsy- eral recent meta-analyses13,14 have sug-
Dr Goldman currently works chotic drug treatment that may confound gested that second-generation antipsy-
for Pfizer, New York, New York. results. Furthermore, potential confound- chotics (SGAs) improve cognition. Large

(REPRINTED) ARCH GEN PSYCHIATRY/ VOL 64 (NO. 10), OCT 2007 WWW.ARCHGENPSYCHIATRY.COM
1115

©2007 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ on 06/01/2021
 DESIGN
Table 1. Demographic Characteristics of the Samples
The trial design has been presented in detail elsewhere.22 Pa-
Olanzapine Risperidone Healthy tients with FES, schizoaffective disorder, or schizophreniform
Group Group Controls disorder were assessed at baseline and randomly assigned to treat-
Characteristic (n=51) (n=53) (n = 84) ment with olanzapine (2.5-20 mg/d) (n=51) or risperidone (1-6
Age, mean ± SD, y a 23.9±4.6 23.8±5.4 27.9 ± 7.0 mg/d) (n=54) for 16 weeks. Psychopathologic and cognitive as-
Male, No. (%) a 38 (75) 37 (70) 34 (41) sessments were performed by masked (blinded) assessors. The
CGI score for severity illness 5.57±0.58 5.55±0.64 NA mean±SD modal doses were 12.6±6.0 mg/d for olanzapine and
at study entry, mean±SD 3.7±1.9 mg/d for risperidone. Patients with FES and the HC group
WRAT-3 reading standard 88.6±15.1 89.3±14.6 103.23 ± 9.11 received cognitive assessments at baseline (when most patients
score, mean ± SD a,b with FES were drug free) and after 6 and 16 weeks.
Education level (grade), 11.8±1.9 12.0±2.6 14.5 ± 2.0
mean ± SD a,b
COGNITIVE TESTS
Abbreviations: CGI, Clinical Global Impressions scale; NA, not applicable;
WRAT-3, Wide Range Achievement Test 3. The cognitive tests listed in Table 2 were administered to all
a Results of all analysis of variance and ␹2 tests were significant (P ⬍.001);
study participants. They included measures of processing speed,
the olanzapine and risperidone groups differed significantly from the healthy episodic memory, working memory, executive function, and
control group by post hoc analysis.
b Sample sizes are 35 for the olanzapine group, 43 for the risperidone motor speed and dexterity. In the FES group at baseline, 97 to
group, and 51 for the healthy controls. 101 of patients received all tests with the exception of the Wis-
consin Card Sorting Test (WCST) (n=73), the California Ver-
bal Learning Test (CVLT) (n=18), and the Continuous Per-
formance Test identical pairs (CPT-IP) and Delayed Match to
industry-sponsored controlled trials that examined ris- Sample Test (DMS) (not administered); at 6 weeks, 75 to 79
peridone or olanzapine in patients with FES found sig- patients had received all tests with the exception of the CPT-IP
nificant improvement from baseline with the SGAs; effect and DMS (n=49); and at 16 weeks, 70 to 72 patients had re-
sizes ranged from 0.35 to 0.54 on composite measures ceived all tests with the exception of the CPT-IP and DMS
of cognition.15-18 Critically, these studies did not in- (n=49). In the HC sample at baseline, 79 to 84 individuals had
clude control groups, raising the possibility that improve- received all tests with the exception of the CPT-IP and DMS
ments were due to practice effects because patients were (not administered); at 6 weeks, 59 to 61 individuals had re-
tested on multiple occasions. Additionally, the propor- ceived all tests with the exception of the CPT-IP and DMS
tion of non–drug-naive patients at baseline in these trials (n=41); and at 16 weeks, 54 to 55 individuals had received all
tests with the exception of the CPT-IP and DMS (n=41).
was rather high, raising the possibility that effects were
due to medication withdrawal and/or switching from a
drug that has adverse effects on cognition. PSYCHOPATHOLOGIC RATINGS
Herein, we report the results of a randomized com-
parison of 2 of the most widely prescribed SGAs in the The Schedule for Affective Disorders and Schizophrenia–
Change Version plus psychosis and disorganization items was
United States, olanzapine and risperidone (thereby in- used to rate severity of hallucinations and delusions (com-
creasing the generalizability of the result), in patients with bined into a positive symptom dimension) as well as disorga-
FES. Our randomized clinical trial addressed the issues nization in speech (understandability) and bizarre behavior
raised by prior work: (1) inclusion of a healthy control (combined into a disorganized dimension).26 The Schedule for
group to assess practice effects, (2) inclusion of a high the Assessment of Negative Symptoms (Hillside version) was
percentage of patients who were drug naive at baseline, used to rate negative symptoms.27
and (3) federal sponsorship.19,20
STATISTICAL ANALYSES
METHODS
Repeated-measures analyses of variance that used mixed mod-
els to minimize the effects of missing data (Proc Mixed; SAS28)
STUDY PARTICIPANTS and in which covariance patterns were unstructured exam-
ined longitudinal changes and between-group effects in cog-
Demographic information about the olanzapine, risperidone, and
nitive measures. Factors in the models were group, time, and
healthy control groups is given in Table 1. Of the 104 patients
group⫻time interactions. Significance was set at P⬍.003 af-
with FES, 80 were never exposed to medication, 14 had less than
ter Bonferroni correction for multiple comparisons (16 tests)
1 week of antipsychotic exposure, and 10 had more than 1 week
in the initial set of mixed-model repeated measures.
of antipsychotic exposure. Seventy-six patients had been diag-
Our approach to the interpretation of statistical results
nosed as having schizophrenia, 10 as having schizoaffective dis-
follows:
order, and 18 as having schizophreniform disorder as deter-
mined by the Structured Clinical Interview for DSM-IV.21 All 1. A medication type (olanzapine or risperidone)⫻time in-
patients were actively psychotic when they entered the study. teraction would suggest a differential medication effect.
Eighty-four healthy adults recruited from the community by ad- 2. Main effects of treatment week (time) would indicate im-
vertisement or word of mouth served as controls (HCs). All HCs provement due to medication or other causes (ie, practice
underwent the Structured Clinical Interview for DSM-IV and did effects).
not have any diagnosable Axis I disorders. Exclusion criteria for 3. To disambiguate the possibilities in item 2, we would then
both groups included medical conditions known to affect the cen- compare performance of the patient groups with that of an HC
tral nervous system and neurologic conditions or receiving drugs group for whom data from multiple assessments were also col-
known to affect cognition. lected. Group (patients with FES and HCs) ⫻ time interac-

(REPRINTED) ARCH GEN PSYCHIATRY/ VOL 64 (NO. 10), OCT 2007 WWW.ARCHGENPSYCHIATRY.COM
1116

©2007 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ on 06/01/2021
 Table 2. Cognitive Tests Used in the Study and Their Domain

Test and Reference Domain Dependent Measure

MMSE23 Mental status Total score
Verbal fluency23 Speed Words produced in 1 minute
Trail making A and B23 Speed Time to complete trails
WAIS-R digit symbol (scaled score)23 Speed Coded symbols in 2 minutes
WCST23 Executive function Percentage of perseverative errors; categories attained
CPT-IP24 Executive/attention D⬘ (for all stimulus sets)
CVLT23 Episodic memory Words recalled in trials 1-5; recognition errors
WMS-R logical memory23 Episodic memory Story elements recalled
WMS-R visual reproduction23 Episodic memory Design elements recalled
Judgment line orientation23 Spatial processing Lines accurately matched
DMS25 Working memory Memory for designs after short delays
WMS-R digit span23 Working memory Digit sequences recalled
Finger rapping23 Motor Taps in 10 seconds
Grooved pegboard23 Motor Time to place pegs

Abbreviations: CPT-IP, Continuous Performance Test identical pairs subtest; CVLT, California Verbal Learning Test; DMS, Delayed Match to Sample Test;
MMSE, Mini-Mental State Examination; WAIS-R, Wechsler Adult Intelligence Test–Revised; WCST, Wisconsin Card Sorting Test; WMS-R, Wechsler Memory
Scale–Revised.

Table 3. Effects of Risperidone and Olanzapine in Patients With First-Episode Schizophrenia

Treatment
Weekⴛ Medication
Treatment Week Medication Group Group

Variable a F P Value F P Value F P Value

Semantic fluency (2/1/2, 101) 2.98 .06 1.12 .29 2.15 .12
Trail making A and B (2/1/2, 102) 26.20 ⬍.001 0.35 .55 1.71 .49
WCST % perseveration (2/1/2, 71; 2/1/2, 96) 9.28 ⬍.001 0.38 .54 0.94 .39
WCST loss of set 3.03 .05 0.44 .51 0.38 .68
CVLT trials 1-5 (2/1/2, 82) 13.70 ⬍.001 0.06 .80 1.64 .20
CVLT recognition errors 6.80 .002 0.13 .71 0.17 .84
Finger tapping (2/1/2, 100) 1.81 .17 1.25 .27 0.59 .56
Grooved peg (2/1/2, 102) 2.63 .08 0.01 .92 2.82 .06
Line orientation (2/1/2, 99) 7.91 ⬍.001 0.29 .59 0.32 .79
MMSE (2/1/2, 100) 37.41 ⬍.001 0.07 .78 2.79 .07
Digit span (2/1/2, 100) 4.80 .01 0.05 .83 1.98 .14
Digit symbol (2/1/2, 99) 12.55 ⬍.001 0 .97 1.82 .17
WMS-R logical memory (stories) (2/1/2, 100) 41.16 ⬍.001 0.52 .47 1.64 .20
WMS-R visual reproduction (designs) (2/1/2, 101) 25.03 ⬍.001 0.65 .42 0.21 .81
CPT-IP (2/1/2, 58) 1.44 .24 1.31 .25 1.0 .32
DMS (2/1/2, 58) 0.02 .94 0.05 .90 0.23 .85
Severity of illness (2/1/2, 101) 116.25 ⬍.001 0 .95 0.09 .92

Abbreviations: CPT-IP, Continuous Performance Test identical pairs subtest; CVLT, California Verbal Learning Test; MMSE, Mini-Mental State Examination;
WCST, Wisconsin Card Sorting Test; WMS-R, Wechsler Memory Scale–Revised.
a Information in parentheses is time/group/group ⫻ time, denominator df.

tions that favored steeper improvement in the FES group would modal antipsychotic dose served as the independent measure and
be viewed as evidence of a drug effect, reflecting cognitive en- cognitive change scores served as dependent measures. For mul-
hancement. A main effect for time in the absence of such an tiple regressions, we relaxed significance of the equation to P⬍.05
interaction could be viewed as representing practice effects. and set significance for entry of predictors at P⬍.10.
In a set of secondary analyses, we attempted to determine if
cognitive change could be attributed to causes other than drugs RESULTS
or practice using multiple regression that involved a variant of
stepwise selection (MAXR in the Proc Reg module of SAS28). Thus, In a preliminary analysis, we sought to determine if base-
we sought to determine if baseline state variables or baseline to
line differences between the medication groups were pres-
week 16 change measures (positive, negative, or disorganized
symptoms) predicted cognitive change (ie, if changes could be ent. Study participants assigned to olanzapine and ris-
due to pseudospecificity and simply reflect antipsychotic effects peridone did not differ on any cognitive measure at study
on symptoms). To ascertain if systematic dose effects on cogni- entry (Table 3), as indicated by both nonsignificant medi-
tive change scores were present, we also constructed a series of cation group effects and post hoc t test between-group
linear regressions within each of the medication groups in which effects at baseline.

(REPRINTED) ARCH GEN PSYCHIATRY/ VOL 64 (NO. 10), OCT 2007 WWW.ARCHGENPSYCHIATRY.COM
1117

©2007 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ on 06/01/2021
 Table 4. Healthy Controls Compared With Patients With First-Episode Schizophrenia on Cognitive Measures
That Demonstrate Change

Treatment Week Group Treatment Week ⴛGroup

Variable a F P Value F P Value F P Value

Trail making (2/1/2, 183) 43.53 ⬍.001 85.36 ⬍.001 17.30 ⬍.001
WCST % perseveration (2/1/2, 2/1/2, 175) 14.76 ⬍.001 58.33 ⬍.001 4.69 .01
CVLT trials 1-5 (2/1/2, 164) 34.20 ⬍.001 159.93 ⬍.001 3.61 .03
CVLT recognition errors (2/1/2, 164) 9.36 ⬍.001 42.86 ⬍.001 0.20 .82
Line orientation (2/1/2, 182) 9.96 ⬍.001 31.97 ⬍.001 2.27 .11
MMSE (2/1/2, 184) 35.8 ⬍.001 90.99 ⬍.001 18.02 ⬍.001
Digit symbol (2/1/2, 183) 19.74 ⬍.001 68.85 ⬍.001 1.35 .22
Logical memory (2/1/2, 184) 80.10 ⬍.001 189.10 ⬍.001 0.30 .74
Visual reproduction (2/1/2, 185) 18.87 ⬍.001 50.66 ⬍.001 16.58 ⬍.001

Abbreviations: CVLT, California Verbal Learning Test; MMSE, Mini-Mental State Examination; WCST, Wisconsin Card Sorting Test.
a Information in parentheses is time/group/group ⫻time, denominator df.

pine and risperidone groups into a single FES group (given

Olanzapine group
the dearth of differential drug effects). We examined those
Risperidone group 9 variables that had previously demonstrated improve-
HCs ment over time. Significant interactions (P⬍.006 for all)
35 were revealed for 3 measures (Table 4). Rate of im-
provement was greater in the schizophrenia group than
30
the HC group for memory for visual designs and trail mak-
25
ing, suggesting that drug effects were larger than prac-
tice effects. For the Mini-Mental State Examination
20 (MMSE), a clear ceiling effect in the HC group artifac-
Score

tually produced an interaction. For the WCST and CVLT

15
trials 1 through 5, improvement was greater in the HC
10 group, but the P values of the interaction were .01 and
.03, respectively, which we suggest are at trend levels.
5 For the remaining 5 variables only a main effect of treat-
0
ment week was found, suggesting that improvement was
0 6 16 no greater than changes mediated by practice effect.
Time, wk The Figure illustrates the respective slope of improve-
ment of the FES groups and the HC group for logical
Figure. Performance on logical memory (for stories) for the patients with memory. The magnitude of improvement was often simi-
first-episode schizophrenia (FES) treated with risperidone and olanzapine
and the healthy controls (HCs) at baseline, 6 weeks, and 16 weeks. The lar among the groups, and the 2 FES groups demon-
graph is representative of most tests because the difference between the strated nearly identical patterns of change for most mea-
2 FES groups is small, the rate of improvement between the FES and HC sures. All cognitive performances in patients with FES,
groups is similar, and the difference between the FES and HC groups
persists throughout the study. Error bars indicate SEMs.
irrespective of whether or not they demonstrated im-
provement, were significantly below those of the con-
trol group even after premorbid intellectual ability, as mea-
PRIMARY ANALYSES sured by the Wide Range Achievement Test 3, served as
a covariate in the mixed-model analyses (FES group vs
We turned to the possibility that olanzapine and risperi- HC group main effects: P ⬍.001 for all entries except 1).
done had differential impact on cognitive improvement, The composite effect size (Cohen d) in the FES group
as would be indicated by medication group⫻ treatment (from baseline through week 16) for the 16 measures ini-
week (time) interaction. No such interactions were ob- tially examined was 0.36. However, for the HC group the
served for any variable (Table 3). Cohen d was similar (0.33). The effect size inspection
We then sought to determine if performance on cog- (Table 5) indicated that improvements in cognition were
nitive measures improved irrespective of medication type larger in treatment weeks 0 to 6 than 6 to 16 in both
(ie, if a treatment week main effect were present). We groups. (For the sake of context, we also included effect
observed such effects for most variables (Table 3). Only sizes of key positive, disorganized, and negative symp-
fluency, digit span, CPT-IP, DMS, and the motor tasks tom dimensions in Table 5.) Performance of the groups
did not demonstrate change over time. on each test at each time point is given in eTable 1
Given the aforementioned results, we sought to de- (available at: http://terrygoldberg.net/Documents
termine if improvements over time were equivalent to /msfe_supplement_tables.doc).
or greater than practice effects in HCs who were also tested We sought to determine if otherwise equivalent change
on 3 occasions. In these analyses, we collapsed the olanza- scores in the FES and HC groups were an artifact of dif-

(REPRINTED) ARCH GEN PSYCHIATRY/ VOL 64 (NO. 10), OCT 2007 WWW.ARCHGENPSYCHIATRY.COM
1118

©2007 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ on 06/01/2021
 Table 5. Effect Sizes in Patients With First-Episode Schizophrenia (Combined) and Healthy Controls Within Each Treatment Period

Patients With Schizophrenia Healthy Controls

Variable Weeks 0-6 Weeks 6-16 Weeks 0-16 Weeks 0-6 Weeks 6-16 Weeks 0-16
Semantic fluency −0.21 0.06 −0.13 0.04 0.01 0.06
Trail making 0.43 0.41 0.70 0.24 0.30 0.56
WCST % perseveration 0.37 0.06 0.36 0.42 −0.01 0.42
WCST set −0.23 0.15 −0.06 −0.13 0.04 −0.05
CVLT trials 1-5 0.40 0.48 0.53 0.64 0.18 0.71
CVLT recognition 0.37 0.05 0.42 0.42 0.11 0.44
Line orientation 0.39 −0.01 0.34 0.17 0.05 0.22
MMSE 0.70 0.16 0.79 0.01 0.39 0.44
Logical memory 0.74 0.32 0.84 0.79 0.29 1.00
Visual reproduction 0.64 0.07 0.66 0.12 −0.11 −0.01
DMS NA 0.08 NA NA 0.00 NA
Digit span 0.24 0.17 0.35 0.42 0.07 0.45
Digit symbol 0.43 0.07 0.43 0.17 −0.10 0.62
CPT-IP NA 0.15 NA NA 0.04 NA
Tapping 0.12 0.13 0.26 0.13 0.20 0.29
Pegboard 0.20 0.01 0.14 0.51 0.16 0.67
SANS 0.38 -0.30 0.10 NA NA NA
Hallucinations 1.30 0.11 1.40 NA NA NA
Delusions 0.96 0.58 1.47 NA NA NA
Severity of illness 1.21 0.53 1.42 NA NA NA
Disorganization 0.67 0.26 0.84 NA NA NA

Abbreviations: CPT-IP, Continuous Performance Test identical pairs subtest; CVLT, California Verbal Learning Test; DMS, Delayed Match to Sample Test;
MMSE, Mini-Mental State Examination; NA, not applicable; SANS, Schedule for the Assessment of Negative Symptoms; WCST, Wisconsin Card Sorting Test.

ferences in premorbid intellectual function (as mea- COMMENT

sured on the WRAT reading subtest1,2) so that degree of
change was masked in the FES group. We performed a
series of linear regressions in which change scores for those Before discussing key results, it is important to appreci-
variables for which a significant time effect was present ate that the sample of patients with FES studied herein
(Table 4) served as dependent measures and the WRAT appears to be representative, both in terms of demon-
reading score and 2 demographic variables, sex and age, strating rather widespread cognitive impairments at base-
on which the HC group differed from the FES group, line and in experiencing large reductions in positive and
served as independent predictors. No independent vari- disorganized symptoms with antipsychotic treatment.22
able entered significantly in any of these equations, sug- In our study, we observed highly significant improve-
gesting that results were not due to preexisting intellec- ments in cognitive performance. These improvements oc-
tual or demographic differences that affected the slope. curred in tests of speed of processing that required psy-
chomotor function (trail making or digit symbol),
SECONDARY ANALYSES consistently in verbal and visual tests of episodic memory,
and in executive functions related to set shifting.
We next examined the association of clinical symptoms Our results indicate that differential medication ef-
at baseline and clinical symptom improvement (from base- fects on cognition were small. Although these results sug-
line to 16 weeks) with cognitive changes in the FES group. gest that the choice of initial SGA should not be based
We restricted our analyses to variables for which we had on presumptive cognitive advantages for one or the other
data at all 3 assessment points. Symptoms proved to be drug early in schizophrenia, we recognize that many other
rather weak and inconsistent predictors of cognitive factors can weigh in the selection of one drug or an-
change, irrespective of whether the cognitive measure other (eg, antipsychotic efficacy). These results are con-
demonstrated changes of large magnitude or small mag- sistent with several recent reports of direct comparison
nitude at the group level. No R2 exceeded 0.08 for any of olanzapine and risperidone that did not observe dif-
independent measure or set of measures, as can be seen ferential effects in long-term samples.29,30 Practice ef-
in eTable 2 (available at: http://terrygoldberg.net fects were not examined in these studies.
/Documents/msfe_supplement_tables.doc). To the best of our knowledge, this is the first con-
Last, the mean modal dose of SGAs was not predic- trolled clinical trial that includes multiple assessments of
tive of change in the cognitive variables used in the prior an HC group to (1) assess the presence of practice effects
multiple regressions (ie, WCST, visual reproduction, trail and their magnitude in an HC group and (2) determine if
making, CVLT measures, memory for stories, MMSE, digit drug effects were greater than practice effects in the group
symbol, or line orientation). P values were between .11 of patients with FES. In 2 instances the performance gains
and .97. All R2 values were below 0.09. Samples for all these of patients with FES exceeded the practice effects in HCs:
analyses were generally between 31 and 39 patients. episodic memory for visual designs and trail-making at-

(REPRINTED) ARCH GEN PSYCHIATRY/ VOL 64 (NO. 10), OCT 2007 WWW.ARCHGENPSYCHIATRY.COM
1119

©2007 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ on 06/01/2021
 tention and speed (MMSE was not considered, since a ceil- On the face of it, practice effects may be advantageous
ing effect was present in healthy controls). Because the ef- clinically. Many activities in daily life rely on practice or rep-
fects could not be attributed to high covariation with etition for optimizing performance. However, little evidence
positive or negative symptoms, these effects might be con- indicates that such types of improvement will generalize
sidered to represent valid cognitive enhancement. How- to other tasks33 because a practice effect is paradigm spe-
ever, most variables did not demonstrate rates of improve- cific (eg, familiarity with testing instructions and demands)
ment above and beyond practice effects: verbal episodic or item specific (eg, words on a list). In the present context,
memory, visual spatial processing, card sorting and set shift- practice effects may not reflect change in the compromised
ing, or digit symbol coding speed. Several other variables neurobiological function of schizophrenia that would then
did not demonstrate any change (eg, verbal fluency, digit effect improvement in broad domains of cognition. Further-
span, CPT-IP, or DMS). It is sobering to note that the com- more, practice effects will not compensate for baseline dif-
posite effect size in the FES group of 0.36 would be con- ferences, since patients who start lower than controls also
sidered moderate and could be attributed to treatment; only end lower than controls (who are also practicing) despite
when it is compared with the effect size in the HC group improvement (as can be seen in the Figure). Several lines
(0.33) does it become clear that the magnitude of the effect of evidence suggest that patients with schizophrenia are ca-
is in keeping with a practice-related phenomenon. Addi- pable of demonstrating a practice effect, including intact
tionally, effect sizes for cognition in the group of patients consolidation. Thus, patients do not demonstrate markedly
with FES were larger in the 0- to 6-week period than in accelerated rates of forgetting; they retain what they can en-
the 6- to 16-week period, which is consistent with the pat- code.34,35 To the extent that some of the possible practice
tern of practice effects in the HC group. effects observed in patients with FES may have been me-
Other naturalistic studies10-12 of cohorts of patients with diatedbyprocedurallearning,thistypeoflearningis thought
FES have included HCs who underwent serial cognitive to be relatively intact in schizophrenia.36 Additionally, prac-
testing in parallel designs. Although these studies nec- tice effects might engage subtly different neural systems than
essarily did not control medication regimens (and there- those involved in initial task performance.37
fore included first-generation antipsychotics and SGAs) The magnitude of practice effects on verbal memory and
and had variable numbers of drug-naive patients at base- speed measures observed in the HC group is similar to that
line testing, as well as smaller sample sizes than the pres- reported in the literature.38-40 Several tests used in our study
ent study, cognitive results were strikingly comparable demonstrated minimal practice effects and had several com-
to those reported herein. Broadly, these studies found that monalities: simple directions, large numbers of trials, and
patients with FES continued to demonstrate cognitive im- a restricted set of stimuli, resulting in little distinctiveness
pairments of −1 to −2 SDs during 2- to 5-year periods among individual instances. These tests were associated with
compared with HCs. Thus, even when patients with FES minimal improvement in both the FES and HC groups.
made gains, so did HCs (presumably on the basis of prac- Several studies in patients with FES have discerned dif-
tice effects), who thereby maintained their advantage. ferences in cognitive improvement between an SGA and a
Our results were also broadly consistent with a meta- first-generation comparator. Beyond possible sources of bias
analysis of SGA effects on cognition14; we found that ver- in design, drug dose, or analysis in these trials,19 we note
bal fluency and measures of simple working memory dem- that (1) the effect sizes in the SGA groups in this study were
onstrated smaller effect sizes than tests of either learning in keeping with the practice effects discerned in HCs and
or psychomotor speed. However, in our study some of (2) first-generation drugs might retard a practice effect.41-43
the larger effects in verbal learning and speed of process- Thus, we would suggest that these results do not indicate
ing (eg, digit symbol coding) appeared to be consistent cognitive enhancement beyond that found in practice ef-
with practice effect gains, not cognitive enhancement per fects. Whether SGAs permit or attenuate the development
se.31 The large Clinical Antipsychotic Trials of Interven- of practice effects is an intriguing question but one that can-
tion Effectiveness study,32 which involved a somewhat not be resolved fully by this study, since practice effects in
complicated switching design, has found smaller effects the drug-free state in the FES sample cannot be assessed be-
of SGAs. Thus, our study may have implications for the cause of ethical issues. Similarly, it might be argued that if
reinterpretation of several previous trials in which cog- 1 patient group demonstrated larger cognitive changes than
nition improved because the magnitudes of improve- another after receiving an adjunctive medication, this would
ment found in prior trials may not be greater than the indicate prima facie that cognitive enhancement had oc-
practice effect demonstrated by the HC group in our study. curred. However, under certain circumstances an HC group
We considered other explanations besides practice ef- might still be necessary because of the possibility that the
fects for the cognitive improvements observed in our pa- base antipsychotic might suppress a practice effect, whereas
tients with FES. Improvements in symptoms could not ac- the adjunctive exerts a compensatory effect. Because im-
count for cognitive change, although our use of statistical provement would be no greater than that seen in an HC
control methods was admittedly not optimal.5 Similarly, group, it might be necessary to provide a context or met-
demographic variables and premorbid level of intellec- ric for rationally calibrating magnitude of improvement.
tual function did not predict slope. Moreover, the rela- Our approach to pseudospecificity was admittedly im-
tionship of the magnitude of cognitive change scores and perfect, and it is possible that general improvement in
medication dose was close to nil in patients. Although these symptom status, organization, and general test-taking be-
results further strengthen the case that change was due to havior accounted for some of the change. Such a result
practice and not induced by medication, demographic, or would not be inconsistent with our general notion that
state variables, they do not prove it. SGAs may not directly enhance cognition to the degree

(REPRINTED) ARCH GEN PSYCHIATRY/ VOL 64 (NO. 10), OCT 2007 WWW.ARCHGENPSYCHIATRY.COM
1120

©2007 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ on 06/01/2021
 previously thought. Furthermore, the possibility that halo- Our findings may also have implications for drug dis-
peridol at relatively high doses suppressed practice ef- covery and regulatory approval of new antipsychotic medi-
fects in some earlier studies43 that compared cognitive cations, including inclusion of an HC control group if cog-
change induced by first-generation antipsychotics and nitive change is being measured. In some circumstances,
SGAs is consistent with our argument that changes we our findings may have implications for the design of trials
observed may be in part due to practice effects, given the that use cognitive enhancing adjunctive agents, such as cases
relatively low doses of SGAs used in this study (ie, doses in which it becomes important to calibrate the magnitude
unlikely to interfere with a practice effect). of change to some external metric (ie, a practice effect in
Minimizing practice effects may not be easy. Use of a an HC group). We recognize that our study cannot fully
different design to reduce practice effects, such as a cross- disambiguate contributions to cognitive change due to prac-
over with counterbalancing or serial testing during a tice effects, pseudospecificity, or drug-induced cognitive
lead-in period, are not without pitfalls.44 Alternate test enhancement. Nevertheless, we hope that our findings in-
forms may attenuate but not eliminate practice ef- crease awareness of practice effects as a potential source
fects.45,46 It might also be possible to develop tasks based of cognitive change in clinical trials and that our findings
on the criteria we outlined herein (eg, multiple trials, re- can be used heuristically in the development of study de-
stricted stimulus set, and high interference) that are rela- signs and tests that are relatively insensitive to practice-
tively resistant to practice effects. related changes. Such advances may be important for im-
This study has several limitations. We could not as- proving methods involved in the assessment of cognitive
sess practice effects in drug-free individuals because of change in clinical trials.
concerns about human subjects. Thus, our results should
be viewed as inferential. Although we have tried to pro- Submitted for Publication: November 1, 2006; final re-
vide compelling circumstantial evidence in favor of our vision received March 2, 2007; accepted March 19, 2007.
account, it nevertheless remains circumstantial. Since we Correspondence: Terry Goldberg, PhD, Zucker Hillside
cannot say with absolute certainty what proportion of gain Hospital, 7559 263rd St, Glen Oaks, NY 11004 (tgoldber
is associated with practice effects, it is probably fair to @nshs.edu).
say that practice effects should at least be considered when Financial Disclosure: Dr Goldman currently works for
interpreting cognitive improvement in clinical trials. Pfizer. At the time the study was designed and initiated
Also, the generalizability of our study to patients with and the data were collected, he was at Zucker Hillside
more chronic conditions is unclear. We do not think that Hospital. Dr Kane is a consultant for Abbott, BMS, Pfizer,
our results extend to recent and select cognitive reha- Janssen, and Lilly and lectures for BMS and Janssen. Dr
bilitation programs that use drills, reinforcement, and Schooler serves on advisory boards for Janssen amd BMS
meta-cognitive instruction, since they have sometimes and has received unrestricted educational grants from As-
resulted in generalizable improvements and vocational traZeneca, BMS, Eli Lilly, Janssen, and Pfizer. Dr Rob-
successes.47,48 Occult ceiling and floor effects may have inson lectures for Janssen and has received funding for
been present in some of our data. Future studies might an investigator-initiated grant from Lilly.
use novel item response theory to minimize such prob- Funding/Support: The study was supported by grants
lems. Of course, item response theory–based analyses MH60004, NIDA K23 DA015541, MH41960 (The Zucker
would in no way preclude findings of improvement in Hillside Center for Intervention Research in Schizophre-
true scores due to exposure, test-taking strategy, and ef- nia), and RR018535 (Feinstein Institute for Medical Re-
ficiency of response, as we believe might have occurred search General Clinical Research Center) from the Na-
in the present study. Last, the WRAT reading standard tional Institutes of Health.
scores (a putative measure of premorbid intellectual func- Additional Information: The eTables are available at
tion) were not well matched between the HC and FES http://terrygoldberg.net/Documents/msfe_supplement
groups. We carefully addressed this in a series of mul- _tables.doc.
tiple regressions in which reading level, age, and sex served Additional Contributions: The following individuals
as independent measures and cognitive change scores as helped with various facets of this study: Denise Coscia,
dependent measures. We found no evidence that read- MA, Adrianna Franco, MA, Handan Gunduz-Bruce, MD,
ing level, sex, and age were significant predictors of the Faith Gunning-Dixon, PhD, Ali Khadivi, PhD, Beth Lorell,
magnitude of cognitive gain, irrespective of group. Re- LCSW, Joanne McCormack, LCSW, Alan Mendelowitz,
sults from such regressions are generally considered more MD, Rachel Miller, LCSW, Barbara Napolitano, MS, Gail
transparent than analysis of covariance statistics.49 Reiter, MA, Serge Sevy, MD, and Jose Soto-Perello, MD.
We wish to emphasize that individual patients may
also benefit cognitively from SGAs. Determining the clini-
REFERENCES
cal and genetic characteristics of these individuals may
be important for future research. 1. Goldberg TE, Ragland DR, Gold J, Bigelow LB, Torrey EF, Weinberger DR. Neu-
We did not have baseline data for several measures ropsychological assessment of monozygotic twins discordant for schizophrenia.
(eg, CPT-IP and DMS). This lack of data prevents us from Arch Gen Psychiatry. 1990;47(11):1066-1072.
commenting on change from drug-naive to treated sta- 2. Weickert TW, Egan MF, Weinberger DR, Goldberg TE. Preserved and compro-
tus in the FES groups for these tests. However, we were mised intellect in schizophrenia: neurocognitive implications. Arch Gen Psychiatry.
2000;57(9):907-913.
struck by the lack of change from first exposure to sec- 3. Green MF. What are the functional consequences of neurocognitive deficits in
ond exposure in the HC group, suggesting that the tests schizophrenia? Am J Psychiatry. 1996;153(3):321-330.
themselves may not be prone to practice effects. 4. Bowie CR, Reichenberg A, Patterson TL, Heaton RK, Harvey PD. Determinants of real

(REPRINTED) ARCH GEN PSYCHIATRY/ VOL 64 (NO. 10), OCT 2007 WWW.ARCHGENPSYCHIATRY.COM
1121

©2007 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ on 06/01/2021
 world functional performance in schizophrenia subjects: correlations with cognition, to-sample test in first-episode schizophrenia. Arch Gen Psychiatry. 2003;60
functional capacity, and symptoms. Am J Psychiatry. 2006;163(3):418-425. (3):238-243.
5. Buchanan RW, Davis M, Goff D, Green MF, Keefe RS, Leon AC, Nuechterlain KH, 26. Spitzer RL, Endicott J. Schedule for Affective Disorders and Schizophrenia-
Laughren T, Levin R, Stover E, Fenton W, Marder SR. A summary of the FDA- Change Version. 3rd ed. New York: Biometrics Research Department, New York
NIMH-MATRICS workshop on clinical trial design for neurocognitive drugs for State Psychiatric Institute; 1978.
schizophrenia. Schizophr Bull. 2005;31(1):5-19. 27. Andreasen NC. Negative symptoms in schizophrenia: definition and reliability.
6. Saykin AJ, Shtasel DL, Gur RE, Kester DB, Mozley LH, Stafiniak P, Gur RC. Arch Gen Psychiatry. 1982;39(7):784-788.
Neuropsychological deficits in antipsychotic naı̈ve patients with first-episode 28. SAS Institute Inc. SAS/STAT User’s Guide, Version 9.1. Vol 3. Cary, NC: SAS
schizophrenia. Arch Gen Psychiatry. 1994;51(2):124-131. Institute Inc; 1999.
7. Mohamed S, Paulsen JS, O’Leary D, Arndt S, Andreasen N. Generalized cogni- 29. Bilder RM, Goldman RS, Volavka J, Czobor P, Hoptman M, Sheitman B, Linden-
tive deficits in schizophrenia: a study of first-episode patients. Arch Gen Psychiatry. maer JP, Citrome L, McEvoy J, Kunz M, Chakos M, Cooper TB, Horowitz TL, Lie-
1999;56(8):749-754. berman JA. Neurocognitive effects of clozapine, olanzapine, risperidone, and halo-
8. Bilder RM, Goldman RS, Robinson D, Reiter G, Bell L, Bates J, Pappadopulos E, peridol in patients with chronic schizophrenia or schizoaffective disorder. Am J
Willson DF, Alvir JM, Woerner MG, Geisler S, Kane JM, Lieberman JA. Neu- Psychiatry. 2002;159(6):1018-1028.
ropsychology of first-episode schizophrenia: initial characterization and clinical 30. Keefe RS, Young CA, Rock SL, Purdon SE, Gold JM, Breier A; HGGN Study Group.
correlates. Am J Psychiatry. 2000;157(4):549-559. One-year double-blind study of the neurocognitive efficacy of olanzapine, ris-
9. Gold S, Arndt S, Nopoulos P, O’Leary DS, Andreasen NC. Longitudinal study of peridone, and haloperidol in schizophrenia. Schizophr Res. 2006;81(1):1-15.
cognitive function in first-episode and recent onset schizophrenia. Am J Psychiatry. 31. Hawkins KA, Wexler BE. California Verbal Learning Test practice effects in a schizo-
1999;156(9):1342-1348. phrenia sample. Schizophr Res. 1999;39(1):73-78.
10. Hill SK, Shuepbach D, Herbener ES, Keshavan MS, Sweeney JA. Pretreatment 32. Keefe RS, Bilder RM, Davis SM, Harvey PD, Palmer BW, Gold JM, Meltzer HY,
and longitudinal studies of neuropsychological deficits in antipsychotic-naı̈ve pa- Green MF, Capuano G, Stroup TS, McEvoy JP, Swartz MS, Rosenheck RA, Per-
tients with schizophrenia. Schizophr Res. 2004;68(1):49-63. kins DO, Davis CE, Hsiao JK, Lieberman JA; the CATIE Investigators and the
11. Hoff AL, Sakuma M, Wieneke M, Horon R, Kushner M, DeLisi LE. Longitudinal Neurocognitive Working Group. Neurocognitive effects of antipsychotic medica-
neuropsychological follow-up study of patients with first-episode schizophrenia. tions in patients with chronic schizophrenia in the CATIE trial. Arch Gen
Am J Psychiatry. 1999;156(9):1336-1341. Psychiatry. 2007;64(6):633-647.
12. Albus M, Hubmann W, Mohr F, Hecht S, Hinterberger-Weber P, Seitz NN, Kuchen- 33. Logan GD. Toward an instance theory of automatization. Psychol Rev. 2002;109
hoff H. Neurocognitive functioning in patients with first-episode schizophrenia: (2):376-400.
results of a prospective 5-year follow-up study. Eur Arch Psychiatry Clin Neurosci. 34. Heaton R, Paulsen JS, McAdams L, Braff D, Jeste DV. Neuropsychological defi-
2006;256(7):442-451. cits in schizophrenics: relationship to age, chronicity, and dementia. Arch Gen
13. Keefe RS, Silva SG, Perkins DO, Lieberman JA. The effects of atypical antipsy- Psychiatry. 1994;51(6):469-476.
chotic drugs on neurocognitive impairment in schizophrenia: a review and 35. Gold JM, Rehkemper G, Binks SW, Carpenter C, Fleming K, Goldberg TE, Wein-
meta-analysis. Schizophr Bull. 1999;25(2):201-222. berger DR. Learning and forgetting in schizophrenia. J Abnorm Psychol. 2000;
14. Woodward ND, Purdon SE, Meltzer HY, Zald DH. A meta-analysis of neuropsy- 109(3):534-538.
chological change to clozapine, olanzapine, quetiapine, and risperidone in 36. Weickert TW, Terrazas A, Bigelow LB, Malley JD, Hyde T, Egan MF, Weinberger DR,
schizophrenia. Int J Neuropsychopharmacol. 2005;8(3):457-472. Goldberg TE. Habit and skill learning in schizophrenia: evidence of normal striatal pro-
15. Keefe RS, Seidman LJ, Christensen BK, Hamer RM, Sharma T, Sitskoorn MM, cessing with abnormal cortical input. Learn Mem. 2002;9(6):430-442.
Rock SL, Woolson S, Tohen M, Tollefson GD, Sanger TM, Lieberman JA; HGDH 37. Kelly AM, Garavan H. Human functional neuroimaging of brain changes associ-
Research Group. Long-term neurocognitive effects of olanzapine or low-dose ated with practice. Cereb Cortex. 2005;15(8):1089-1102.
haloperidol in first-episode psychosis. Biol Psychiatry. 2006;59(2):97-105. 38. Theisen ME, Rapport LJ, Axelrod BN, Brines DB. Effects of practice in repeated
16. Harvey PD, Rabinowitz J, Eerdekens M, Davidson M. Treatment of cognitive im- administrations of the Wechsler Memory Scale Revised in normal adults.
pairment in early psychosis: a comparison of risperidone and haloperidol in a Assessment. 1998;5(1):85-92.
large long-term trial. Am J Psychiatry. 2005;162(10):1888-1895. 39. Woods SP, Delis DC, Scott JC, Kramer JH, Holdnack JA. The California Verbal
17. Keefe RSE, Hongbin G, Sweeney JA, Perkins DO, McEvoy JP, Hamer RM, Lie- Learning Test—second edition: test-retest reliability, practice effects, and reli-
berman JA. The effects of olanzapine, quetiapine, and risperidone on neurocog- able change indices for the standard and alternate forms. Arch Clin Neuropsychol.
nitive function in first-episode psychosis: a double-blind, 52-week comparison. 2006;21(5):413-420.
Poster presented at the Society of Biological Psychiatry’s 61st Annual Conven- 40. Hinton-Bayre A, Geffen G. Comparability, reliability, and practice effects on al-
tion and Scientific Program; May 18-20, 2006; Toronto, Ontario, Canada. ternate forms of the Digit Symbol Substitution and Symbol Digit Modalities tests.
18. Keefe RS, Seidman RJ, Christensen BK, Hamer RM, Sharma T, Sitskoorn MM, Le- Psychol Assess. 2005;17(2):237-241.
wine RR, Yurgelun-Todd DA, Gur RC, Tohen M, Tollefson GD, Sanger TM, Lieber- 41. Harvey PD, Moriarty PJ, Serper MR, Schnur E, Lieber D. Practice-related im-
man JA. Comparative effect of atypical and conventional antipsychotic drugs on neu- provement in information processing with novel antipsychotic treatment. Schizophr
rocognition in first-episode psychosis: a randomized, double-blind trial of olanzapine Res. 2000;46(2-3):139-148.
versus low doses of haloperidol Am J Psychiatry. 2004;161(6):985-995. 42. Harvey PD, Palmer BW, Heaton RK, Mohamed S, Kennedy J, Brickman A. Sta-
19. Heres S, Davis J, Maino K, Jetzinger E, Kissling W, Leucht S. Why olanzapine bility of cognitive performance in older patients with schizophrenia: an 8-week
beats risperidone, risperidone beats quetiapine, and quetiapine beats olanza- test-retest study. Am J Psychiatry. 2005;162(1):110-117.
pine: an exploratory analysis of head-to-head comparison studies of second- 43. Carpenter WT, Gold JT. Another view of therapy for cognition in schizophrenia.
generation antipsychotics. Am J Psychiatry. 2006;163(2):185-194. Biol Psychiatry. 2002;51(12):969-971.
20. Ridker PM, Torres J. Reported outcomes in major cardiovascular clinical trials 44. Weickert TW, Goldberg TE, Marenco S, Bigelow LB, Egan MF, Weinberger DR.
funded by for-profit and not-for-profit organizations: 2000-2005. JAMA. 2006; Comparison of cognitive performances in a placebo period and an atypical neu-
295(19):2270-2274. roleptic treatment period in schizophrenia. Neuropsychopharmacology. 2003;
21. First MB, Spitzer RL, Gibbon M, Williams JBW. Structured Clinical Interview for 28(8):1491-1500.
Axis I DSM-IV Disorders-Patient Edition. New York: Biometrics Research De- 45. Uchiyama CL, D’Elia LF, Dellinge AM, Becker JT, Selnes OA, Wesch JE, Chen BB,
partment, New York State Psychiatric Institute; 1998. Satz P, van Gorp W, Miller EN. Alternate forms of the Auditory-Verbal Learning
22. Robinson DG, Woerner MG, Napolitano B, Patel RC, Sevy SM, Gunduz-Bruce H, Test: issues of test comparability, longitudinal reliability, and moderating variables.
Soto-Perello JM, Mendelowitz A, Khadivi A, Miller R, McCormack J, Lorell BS, Arch Clin Neuropsychol. 1995;10(2):133-145.
Lesser ML, Schooler NR, Kane JM. Randomized comparison of olanzapine ver- 46. Beglinger LJ, Gaydos B, Tangphao-Daniels O, Duff K, Kareken DA, Crawford J,
sus risperidone for the treatment of first-episode schizophrenia: 4-month outcomes. Fastenau PS, Siemers ER. Practice effects and the use of alternate forms in se-
Am J Psychiatry. 2006;163(12):2096-2102. rial neuropsychological testing. Arch Clin Neuropsychol. 2005;20(4):517-529.
23. Lezak MD. Neuropsychological Assessment. 3rd ed. New York, NY: Oxford Uni- 47. Bellack AS, Weinhardt LS, Gold JM, Gearon JS. Generalization of training ef-
versity Press; 1995. fects in schizophrenia. Schizophr Res. 2001;48(2-3):255-262.
24. Cornblatt BA, Risch NJ, Faris G, Friedman D, Erlenmeyer-Kimling L. The Con- 48. McGurk SR, Mueser KT, Pascaris A. Cognitive training and supported employ-
tinuous Performance Test, identical pairs version (CPT-IP) 1: new findings about ment for persons with severe mental illness: one-year results from a random-
sustained attention in normal families. Psychiatry Res. 1988;26(2):223-228. ized controlled trial. Schizophr Bull. 2005;31(4):898-909.
25. Lencz T, Bilder RM, Turkel E, Goldman RS, Robinson D, Kane JM, Lieberman JA. 49. Cohen J, Cohen P. Applied Multiple Regression/Correlation Analysis for the Be-
Impairments in perceptual competency and maintenance on a visual delayed match- havioral Sciences. Hillsdale, NJ: Lawrence Erlbaum; 1983.

(REPRINTED) ARCH GEN PSYCHIATRY/ VOL 64 (NO. 10), OCT 2007 WWW.ARCHGENPSYCHIATRY.COM
1122

©2007 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ on 06/01/2021