PSYCHOLOGICAL SCIENCE

Research Report
THE VALIDITY OF THE RORSCHACH AND THE MINNESOTA
MULTIPHASIC PERSONALITY INVENTORY:
Results From Meta-Analyses
Howard N. Garb,1,2 Colleen M. Florio,3 and William M. Grove4
1
Pittsburgh VA Health Care System; 2Department of Psychiatry, University of Pittsburgh; 3Gynecologic Oncology Group, Roswell
Park Cancer Institute, Buffalo, New York; and 4Department of Psychology, University of Minnesota

Abstract—Results from meta-analyses have been widely cited to dures were not used to aggregate estimates across studies (Atkinson,
defend the validity of the Rorschach. However, the meta-analyses have 1986). We describe problems with a fourth meta-analysis (Parker et
been flawed. For example, one meta-analysis included results that al., 1988) in greater detail.
were obtained by calculating correlations but not results that were For their meta-analysis, Parker et al. (1988) included Rorschach
obtained by conducting t tests or analyses of variance. When we re- and MMPI studies that were published between 1970 and 1981 in the
analyzed the data from the most widely cited meta-analysis (Parker, Journal of Personality Assessment and the Journal of Clinical Psy-
Hanson, & Hunsley, 1988), we found that for confirmatory studies chology. They placed results into an “unknown-validity” category if
(also called convergent-validity studies), the Minnesota Multiphasic they “lacked a theoretical or empirical rationale” (p. 370) and into a
Personality Inventory (MMPI) explained 23% to 30% of the variance, “convergent-validity” category if the authors of an article described a
whereas the Rorschach explained only 8% to 13% of the variance. theoretical reason for expecting a statistically significant result or if
These results indicate that the Rorschach is not as valid as the MMPI. the intent of a study was to replicate a finding that was statistically sig-
nificant in a previous study. Results for the unknown-validity catego-
ry, reported as correlation coefficient equivalents of the average z
The use of the Rorschach Inkblot Method continues to be contro- scores, were .24 for the MMPI and .07 for the Rorschach. Results for
versial (e.g., Exner, 1996; Wood, Nezworski, & Stejskal, 1996a, the convergent-validity category were .46 for the MMPI and .41 for
1996b). However, results from several meta-analyses seem to support the Rorschach.
the Rorschach (Atkinson, 1986; Atkinson, Quarrington, Alp, & Cyr, A problem with the meta-analysis conducted by Parker et al.
1986; Parker, 1983; Parker, Hanson, & Hunsley, 1988). For example, (1988) is that results that were obtained by calculating correlations
based on the results for one of the meta-analyses, Parker et al. (1988) were not pooled with results that were obtained by conducting t tests
concluded that or F tests. Apparently, Parker et al. did not pool the correlation results
with results obtained by conducting other statistical analyses because
the convergent-validity estimates for the Rorschach and MMPI [Minnesota they believe that correlational analyses are more powerful. Indicative
Multiphasic Personality Inventory] were not significantly different. . . . It
of this belief is the following statement made by Parker et al. (1988,
appears that both the MMPI and Rorschach can be considered to have adequate
psychometric properties if used for the purposes for which they were designed
p. 372): “Statistics should be rank ordered from the most to the least
and validated. (abstract, p. 367) powerful as follows: correlations, ANOVAs [analyses of variance],
and t tests.” However, textbooks on meta-analysis do not recommend
The results from the meta-analyses, especially the results from the that meta-analyses be based on correlation results alone. Instead, they
meta-analysis conducted by Parker et al. (1988), have been widely recommend that F ratios, t tests, and chi-square values be included in
cited to defend the validity of the Rorschach (e.g., Beutler & Davison, analyses (e.g., Cooper & Hedges, 1994). If an independent variable is
1995; Ganellen, 1996; Masling, 1997; Shontz & Green, 1992; Weiner, categorical (e.g., if the levels of the independent variable are normal
1996, 1997). It does seem unfair to argue that the Rorschach should subject and schizophrenic patient) and a dependent variable is contin-
not be used in clinical and forensic practice if meta-analyses have sup- uous (e.g., a continuous Rorschach score), then the use of a t test or
ported its use, especially in view of the fact that the meta-analysis ANOVA is appropriate.
conducted by Parker et al. (1988) was published in a highly prestigious Parker et al. (1988) did present results on convergent validity for
journal (Psychological Bulletin) and has been reprinted as an illustra- studies that used correlational analysis, ANOVA, or t tests, but instead
tive example of meta-analysis in a book on methodology (Kazdin, of pooling the data, they presented the results separately for the three
1992). different groups. However, unless the data are pooled, one cannot sat-
Unfortunately, the meta-analyses that have been cited to support isfactorally test for statistical significance or make estimates of level
the Rorschach have been flawed. For example, in one meta-analysis, of validity.
results on validity were not analyzed separately from results on relia- Other problems with the meta-analyses conducted by Parker et al.
bility (Parker, 1983); in another, meta-analysis estimates were not (1988) can also be described. First, assessment instruments sometimes
made of the magnitude of effects (Atkinson et al., 1986); and in a shared methodological variance with the construct measures. For
third, meta-analysis effect sizes were estimated but statistical proce- example, in one study (Last & Weiss, 1976), a Rorschach measure of
ego strength was evaluated by comparing it with another Rorschach
measure of ego strength. In this study, a large effect size may have been
Address correspondence to Howard N. Garb, Behavioral Health (116A-H), obtained because the two measures share method variance, not because
VA Medical Center, 7180 Highland Dr., Pittsburgh, PA 15206-1297; e-mail: either measure is a good measure of ego strength. Second, and even
[email protected]. more damaging, in many of the studies coded by Parker et al., a small

402 Copyright © 1998 American Psychological Society VOL. 9, NO. 5, SEPTEMBER 1998
 PSYCHOLOGICAL SCIENCE

Howard N. Garb, Colleen M. Florio, and William M. Grove

effect size meant that a test was valid. For example, in one study based our results on 18 studies for the Rorschach and 66 studies for
(Hersen & Greaves, 1971), a small effect size indicated that Rorschach the MMPI.
productivity is not related to verbal reinforcement. Similarly, in many In a second meta-analysis, studies were excluded if a Rorschach
studies (e.g., Griffin, Finch, Edwards, & Kendall, 1976; Newmark, score was used as a construct measure in a Rorschach study, a self-
Newmark, & Cook, 1975), an abbreviated form of the MMPI was high- report personality measure was used in an MMPI study, or a small
ly correlated with the standard form of the MMPI, but the effect size effect indicated that validity was good. Using these exclusion rules, we
was estimated to be zero (apparently because statistical tests between eliminated the results from 4 Rorschach studies and 30 MMPI studies.
the short form and regular form were typically nonsignificant). Thus, the results were based on 14 Rorschach studies and 36 MMPI
studies. These results also indicate that the MMPI is more valid than
the Rorschach. Correlation coefficient equivalents of the average
REANALYSIS OF THE PARKER ET AL. (1988) DATA z scores were .33 for the Rorschach and .55 for the MMPI. The differ-
ence between the two correlation coefficients was statistically signifi-
Method cant, z = 8.25, p < .001. Corresponding r2 values were .11 and .30 for
the Rorschach and the MMPI.
The terms “unknown validity” and “convergent validity” are mis- We conducted a third meta-analysis after eliminating studies that
leading. For example, it seems odd to report a validity coefficient for used chi-square analyses. Chi-square analyses were used in four of the
an unknown-validity category (if validity really were unknown, one Rorschach confirmatory studies, but they were not used in any of the
would not be able to report a validity coefficient). Instead of referring MMPI studies. When chi-square analyses were conducted, the depen-
to unknown-validity studies and convergent-validity studies, we refer dent variables were continuous, but they were turned into dichotomous
here to exploratory studies and confirmatory studies. We coded a study variables by the investigators. They may have been turned into
as being exploratory if Parker et al. had coded it as being of unknown dichotomous variables because by looking at the data while they set a
validity, and we coded a study as being confirmatory if Parker et al. cutoff score, the research investigators could capitalize on chance.
had coded it as measuring convergent validity. When the effect-size estimates based on the chi-square analyses were
In a meta-analysis, an effect-size estimate is obtained for each excluded, the correlation coefficient equivalents of the average z
study, and statistical procedures are used to aggregate the estimates scores were .36 for the Rorschach and .55 for the MMPI. The differ-
across studies. We used the effect-size estimates that Parker et al. had ence between the two correlation coefficients was statistically signifi-
calculated. They calculated r2 values for studies that used correlation- cant, z = 5.52, p < .001. Corresponding r2 values were .13 and .30 for
al or chi-square analyses and ω2 values for studies that used t tests and the Rorschach and the MMPI.
ANOVAs.
Multiple findings from a single study may be nonindependent. To Exploratory studies
control for nonindependence, we included only a single estimate of
Though Parker et al. emphasized the importance of the results from
effect size from each study. As did Parker et al., we used the median
the confirmatory studies, for the sake of completeness, we also reana-
value in our meta-analysis when more than one estimate of effect size
lyzed the results from the exploratory studies. For 24 studies on the
could be calculated. If there was an even number of effect sizes that
Rorschach and 138 studies on the MMPI, correlation coefficient
could be calculated, we took the average between the two middle val-
equivalents of the average z scores were .11 for the Rorschach and .11
ues (weighted by the sample sizes for the two values). Parker et al. had
for the MMPI. The difference between the two correlation coefficients
randomly selected the higher or lower value at the median.
was not statistically significant, z = 0, p > .05. Corresponding r2 val-
Having obtained our median estimates of effect size, we used the
ues were .01 and .01 for the Rorschach and the MMPI, respectively.
same statistical procedure that Parker et al. used to aggregate the
As we did for the confirmatory studies, we conducted a second
results. After taking the square root of every median estimate of effect
meta-analysis to (a) control for shared methodological variance
size, we transformed all of these values to be unbiased estimators of
between the assessment instruments and the criteria and (b) ensure
the population correlation coefficient by using Equation 3 from
that effect size was not negatively related to validity. These exclusion
Hedges and Olkin’s (1985, p. 225) book. All of these values were then
rules eliminated the results from 9 Rorschach studies and 63 MMPI
transformed into Fisher z scores to normalize the distribution. Weight-
studies. Thus, the results were based on 15 Rorschach studies and 75
ed mean Fisher z scores were calculated, and these scores were trans-
MMPI studies. Correlation coefficient equivalents of the average
formed into correlation coefficient equivalents.
z scores were .18 for the Rorschach and .11 for the MMPI. The differ-
ence between the two correlation coefficients was statistically signifi-
cant, z = –2.08, p < .05. Corresponding r2 values were .03 and .01 for
Results
the Rorschach and the MMPI, respectively.
Confirmatory studies
The results for the confirmatory data indicate that the MMPI is Rorschach scoring systems
more valid than the Rorschach. Correlation coefficient equivalents of We also compared different Rorschach scoring systems. Studies
the average z scores were .29 for the Rorschach and .48 for the MMPI. were included only if (a) the criterion measure was not a Rorschach
The difference between the two correlation coefficients was statisti- score and (b) validity was positively related to effect size. Exner’s
cally significant, z = 8.04, p < .001. Corresponding r2 values were .08 Comprehensive System (Exner, 1974, 1978) was used in 2 of the 14
and .23 for the Rorschach and the MMPI, respectively. Although Park- confirmatory-validity studies and 3 of the 15 exploratory studies, and
er et al. based their confirmatory- (convergent-) validity results on Klopfer’s scoring system (Klopfer, Ainsworth, Klopfer, & Holt, 1954;
only 5 studies for the Rorschach and 31 studies for the MMPI, we Klopfer & Davidson, 1962) was used in 3 of the 14 confirmatory-

VOL. 9, NO. 5, SEPTEMBER 1998 403

Rorschach and Minnesota Multiphasic Personality Inventory

validity studies and 3 of the 15 exploratory studies. No other scoring other studies on the Rorschach (e.g., that the incremental validity of
systems were used as frequently. In several of the studies, investigators the Rorschach is poor; Archer & Krishnamurthy, 1997; Garb, 1984,
created their own rules for scoring Rorschach protocols. 1998), we recommend that less emphasis be placed on training in the
The results indicate that validity did not differ across Rorschach use of the Rorschach. In fact, psychology graduate students may ben-
scoring systems. For confirmatory studies alone, the results from a efit more from a class on judgment and decision making and the use
Kruskal-Wallis one-way ANOVA were not statistically significant of structured interviews than a class on the Rorschach.
when Exner’s Comprehensive System was compared with all other
scoring systems, χ2(1, N = 14) = 0.21, p > .05, or when comparisons
were made among the Comprehensive System, Klopfer’s scoring sys- Acknowledgments—We express our appreciation to Kevin C.H. Parker
for giving us a list of the studies that were assigned to the convergent-
tem, and the remaining scoring systems, χ2(2, N = 14) = 0.21, p > .05. validity and unknown-validity categories and a list of the effect-size esti-
Results were also not statistically significant when the confirmatory mates for each of the studies.
and exploratory studies were combined. This was true when Exner’s
Comprehensive System was compared with all other scoring systems,
χ2(1, N = 29) = 0.16, p > .05, and when comparisons were made REFERENCES
among Exner’s Comprehensive System, Klopfer’s scoring system, and
the remaining scoring systems, χ2(2, N = 29) = 0.45, p > .05. Archer, R.P., & Krishnamurthy, R. (1997). MMPI-A and Rorschach indices related to
depression and conduct disorder: An evaluation of the incremental validity hypoth-
esis. Journal of Personality Assessment, 69, 517–533.
Atkinson, L. (1986). The comparative validities of the Rorschach and MMPI: A meta-
Type of statistic analysis. Canadian Psychology, 27, 238–247.
We also analyzed data by type of statistic (r, F, t, and χ2). For this Atkinson, L., Quarrington, B., Alp, I., & Cyr, J. (1986). Rorschach validity: An empirical
meta-analysis, effect-size estimates for the Rorschach and the MMPI approach to the literature. Journal of Clinical Psychology, 42, 360–362.
Beutler, L., & Davison, E. (1995). What standards should we use? In S. Hayes, V. Follette,
from confirmatory and exploratory studies were all pooled together. R. Dawes, & K. Grady (Eds.), Scientific standards of psychological practice: Issues
As before, studies were excluded if there was shared methodological and recommendations (pp. 11–24). Reno, NV: Context Press.
variance between the assessment instruments and the criteria or if Cooper, H., & Hedges, L. (Eds.). (1994). The handbook of research synthesis. New York:
Sage.
effect size was negatively related to validity. When a Kruskal-Wallis Exner, J.E., Jr. (1974). The Rorschach: A comprehensive system. Vol. 1. New York: Wiley.
one-way ANOVA was conducted, the main effect for type of statistic Exner, J.E., Jr. (1978). The Rorschach: A comprehensive system. Vol. 2. Current research
was not statistically significant, χ2(3, N = 140) = 6.06, p >.10. and advanced interpretation. New York: Wiley.
Exner, J.E., Jr. (1996). A comment on “The Comprehensive System for the Rorschach: A
critical examination.” Psychological Science, 7, 11–13.
Ganellen, R. (1996). Exploring MMPI-Rorschach relationships. Journal of Personality
DISCUSSION Assessment, 67, 529–542.
Garb, H.N. (1984). The incremental validity of information used in personality assess-
ment. Clinical Psychology Review, 4, 641–655.
The results of the confirmatory studies indicate that validity is bet- Garb, H.N. (1998). Studying the clinician: Judgment research and psychological assess-
ter for the MMPI than for the Rorschach. The MMPI explained 23% ment. Washington, DC: American Psychological Association.
to 30% of the variance, whereas the Rorschach explained only 8% to Griffin, J.L., Jr., Finch, A.J., Jr., Edwards, G.L., & Kendall, P.C. (1976). MMPI/MIDI-Mult
correspondence with parents of emotionally disturbed children. Journal of Clinical
13% of the variance. Psychology, 32, 54–56.
In the exploratory studies, both the Rorschach and the MMPI Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Aca-
explained only a negligible amount of the variance (3% and 1% for the demic Press.
Hersen, M., & Greaves, S. (1971). Rorschach productivity as related to verbal reinforce-
Rorschach and the MMPI, respectively). However, there was no theo- ment. Journal of Personality Assessment, 45, 436–441.
retical or empirical reason to believe that the Rorschach or MMPI Kazdin, A. (Ed.). (1992). Methodological issues and strategies in clinical research. Wash-
would do well. For example, in many of these studies, two or three of ington, DC: American Psychological Association.
Klopfer, B., Ainsworth, M.D., Klopfer, W., & Holt, H.A. (1954). Developments in the
the MMPI validity and clinical scales were correlated with a criterion Rorschach technique. New York: Harcourt, Brace, & World.
measure. If two or three of the scales are strongly related to the crite- Klopfer, B., & Davidson, H.H. (1962). The Rorschach technique: An introductory manu-
rion but the other scales are unrelated to the criterion, then the median al. New York: Harcourt, Brace, & World.
Last, U., & Weiss, A. (1976). Evaluation of ego strength based on certain Rorschach vari-
effect size will be zero. Thus, median effect sizes of zero were often ables. Journal of Personality Assessment, 40, 57–66.
assigned to the MMPI even though the MMPI could be used to make Masling, J.M. (1997). On the nature and utility of projective tests and objective tests. Jour-
nal of Personality Assessment, 69, 257–270.
accurate judgments for the task (accurate judgments could be made by Newmark, C.S., Newmark, L., & Cook, L. (1975). The MMPI-168 with psychiatric
attending to only two or three of the scales). patients. Journal of Clinical Psychology, 31, 61–64.
One limitation of the Rorschach studies should be noted. Exner’s Parker, K.C.H. (1983). A meta-analysis of the reliability and validity of the Rorschach.
Journal of Personality Assessment, 47, 227–231.
Comprehensive System was used in relatively few of these studies, Parker, K.C.H., Hanson, R., & Hunsley, J. (1988). MMPI, Rorschach, and WAIS: A meta-
and results may be more promising when Exner’s system is used. analytic comparison of reliability, stability, and validity. Psychological Bulletin,
However, in our reanalysis of the Parker et al. data, results were not 103, 367–373.
Shontz, F., & Green, P. (1992). Trends in research on the Rorschach: Review and recom-
more positive when Exner’s system was used. mendations. Applied & Preventive Psychology, 1, 149–156.
The results also indicate that estimates of effect size did not differ Weiner, I.B. (1996). Some observations on the validity of the Rorschach Inkblot Method.
significantly depending on whether correlations, t tests, ANOVAs, or Psychological Assessment, 8, 206–213.
Weiner, I.B. (1997). Current status of the Rorschach Inkblot Method. Journal of Person-
chi-square analyses were used to analyze data in the primary articles. ality Assessment, 68, 5–19.
This finding undermines the claim by Parker et al. (1988) that correla- Wood, J.M., Nezworski, M.T., & Stejskal, W.J. (1996a). The Comprehensive System for
the Rorschach: A critical examination. Psychological Science, 7, 3–10.
tional statistics are somehow more powerful than the other analyses.
Wood, J.M., Nezworski, M.T., & Stejskal, W.J. (1996b). Thinking critically about the
Questions can be raised about the use of the Rorschach in clinical Comprehensive System for the Rorschach: A reply to Exner. Psychological Science,
and forensic practice. Our reanalysis of the Parker et al. data indicates 7, 14–17.
that the Rorschach is not as valid as the MMPI. Given findings from (RECEIVED 7/21/97; REVISION ACCEPTED 1/5/98)

404 VOL. 9, NO. 5, SEPTEMBER 1998