Using the Case Survey Method to Analyze Policy Studies

Author(s): Robert K. Yin and Karen A. Heald

Source: Administrative Science Quarterly , Sep., 1975, Vol. 20, No. 3 (Sep., 1975), pp.
371-381
Published by: Sage Publications, Inc. on behalf of the Johnson Graduate School of
Management, Cornell University

Stable URL: https://www.jstor.org/stable/2391997

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

Sage Publications, Inc. and Johnson Graduate School of Management, Cornell University are
collaborating with JSTOR to digitize, preserve and extend access to Administrative Science
Quarterly

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
Using the Case Survey A common feature of most policy literatures is that the bulk
of the empirical evidence is embodied in case studies. This
Method to Analyze Policy
presents a problem for subsequent analysis. Although each
Studies
case study may provide rich insights into a specific situa-
tion, it is difficultto generalize aboutthe studies as a whole.
Robert K. Yin and The following article describes one means of dealing with
Karen A. Heald the problem: analyzing the content of case studies by using
a closed-ended questionnaire. The resulting case survey
method allows an analyst to aggregate the case study
experiences and to assess the quality of each case study in a
reliable and replicable manner.1

Evaluation research often involves the assessment of existing

policy studies. The, assessments may serve at least two pur-
poses. First, investigators may be interested in determining
what the research says before establishing a framework for
testing new hypotheses. Second, policymakers may want to
know what has already been learned before considering alterna-
tive public programs. Because of the rapid expansion of public
service programs during the 1 960s and an increase in policy
research, there are now many published and unpublished
studies worthy of secondary analysis: manpower and training
studies, drug treatment and health care research, studies of
educational achievement, research on urban innovations and
the application of new technology to social problems, and
numerous investigations focusing on public organizations and
organizational behavior.

In general, three approaches may be used to review this

research and evaluate the more significant findings.2 Thefirstis
a propositional method, which works best when previous
studies have been organized along a similar experimental
paradigm, so that the reviewer's main task is to compare the
investigators' final propositions or conclusions. The second is a
cluster method, which is most effective when previous studies
have produced large amounts of original quantitative data that
can be aggregated and reanalyzed. The third is a case survey
method. This works best when the studies consist of a
heterogeneous collection of case studies. The reviewer's main
task then is to aggregate the characteristics, but not necessarily
the conclusions, of these cases.

The propositional method typically is used in traditional reviews

of laboratory research. In psychology, for instance, the experi-
mental paradigm is so consistently applied that an analysis of
the propositions or conclusions presented by previous studies
can be fruitfully carried out. The cluster method has been made
popular recently by the increased availability of various sources
of survey data that deal with similar policy issues, so that a
clustering and then reanalysis of the results of several surveys
can be conducted (Light and Smith,, 1971; Hyman, 1972).
The study reported in this article was sup- The case survey method, in contrast, is only in its formative
ported by National Science Foundation
Contract Number C817. The authors are
stage. Yet, for reviewing policy studies, it may be the most
indebted to William Lucas for his advice on appropriate of the three methods since the public policy litera-
the application of the case survey method. ture, to the extent that it is empirical, is heavily dominated by
2
case studies. These case studies often do not follow a similar
These alternative methods and a com-
research paradigm, so the propositional method might result in
prehensive description of the case survey
method are found in Lucas (1974). many false leads; to weigh one investigator's conclusions

September 1975, volume 20 371/Administrative Science Quarterly

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 against those of another, when both investigators have used
entirely different research logics to arrive at their conclusions,
would simply be foolhardy. Neither do the case studies provide
the rich sources of quantitative evidence needed to justify using
the cluster method. In fact, the case survey method was
devised specifically to deal with the problem of reviewing policy
studies on such disparate topics as citizen participation and
urban decentralization.3 Some precedent for the method may
be found in the use of ethnographic case materials for cross-
cultural research (Whiting, 1968). The Human Relations Area
Files, for instance, provide materials on over 200 societies that
can serve as case studies from which investigators may aggre-
gate lessons about human society in general. The use of
content analysis for communications research also provides
some parallels, although content analysis is primarily concerned
with the relation between the manifest and latent content of
messages and not so much with the aggregation of the
messages (Holsti, 1968).

The case survey method is mainly concerned with the analysis

of qualitative evidence in a reliable manner. The method enables
the reviewer to note the various experiences found in each
policy studyand then to aggregatethefrequency of occurrence
of these experiences. The frequencies form the basis for simple
statements of association and nonassociation of different types
of experiences. In this manner, the case survey method gives
the reviewer a chance to survey different case studies. Until
recently, the main shortcoming of case studies was that the
insights from the studies could not be aggregated in any sense.
The case survey method thus carries the classic case study
method one major step forward; it enables aggregate reviews
of individual case studies to be undertaken with scientific rigor.
To illustrate the use of the case survey method, data from a
recently completed study of urban decentralization are reported
here. The study covered five areas of public service and
included such experiences as police innovations, school decen-
tralization, neighborhood health centers, community develop-
ment corporations, and neighborhood facilities such as little city
halls. The purpose of the study was to assess the impact of
decentralization innovations, as reflected in published and un-
3
published studies and evaluation reports. The study examined
The first and relatively cruder application of
the effectiveness of seven decentralization strategies-
the case survey method was made in Yin et community relations, physical redeployment, grievance
aL. (1973). In that study, the citizen participa- mechanisms, administrative decentralization, employment of
tion literature was reviewed. The study did
not call for a rigorous assessment of the neighborhood residents, development of new neighborhood
literature, however, but rather for the iden- institutions, and political decentralization-used in producing
tification of factors associated with the
five results-improved flow of information, improved service
exercise of citizen power. Thus, a more
recent study on urban decentralization (Yin official attitudes, improved client attitudes, improved services,
and Yates, 1975) may be considered the and increased client control. The main findings of these decen-
first fully developed application of the case
survey method.
tralization experiences have been reported elsewhere (Yin and
4 Yates, 1975). The present article describes the use of the case
Reader-analysts are scientific observers. survey method, indicates the lessons that can be learned about
Their role is similar to that of the policy studies by employing the method, and suggests how the
participant-observer, as recently described
byAlbert Reiss, Jr. (1971). Sincethe obser-
method may be applicable to other public policy topics.
vations become the source of data for the
study, the reader-analyst or participant- BASIC TECHNIQUES OF THE CASE SURVEY
observer is both the experimenter and the
subject in the study. The prime virtue of The case survey calls for a reader-analyst4 to answer the same
reader-analysts is that they are trained ob-
servers and can make more difficult judg-
set of questions for each case study. The questions are
ments than can ordinary respondents. closed-ended, so that the answers can be aggregated for

372/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 Case Survey Method

further analysis. In the decentralization study, the questions

covered the relevant characteristics of decentralization, divided
into four sections: (1) the nature of the case study itself-for
example, the author's background, the source of financial
support for the study, and the research design and methods
used; (2) the background characteristics for the innovation,
such as the size of city, the source of financial support for the
innovation, and the degree of community conflict; (3) the
characteristics of the decentralization innovation-for instance,
type of citizen participation, type of services included, and type
of decentralization strategy followed; and (4) the five results of
the decentralization innovation.

In addition to aggregating various characteristics of individual

case studies, the case survey can address three major
methodological problems usually overlooked by most traditional
reviews of research: establishment of the reliability of the
analysis; differentiation of weak and strong responses on the
part of the reader-analyst; and use of explicit criteria for
excluding some studies from the review.

Reliability

The measurement of reliability, and thus the establishment of

replicability, is a minimum step for developing any scientific
method. In this sense, reviewing the literature has always been
more of an art than a science and, except in rare instances, there
has been no attempt to assess the reliability of the method of
review. The capability of the case survey in this regard is very
straightforward: given a fixed set of closed-ended questions,
the reliability of the reader-analyst's responses can be mea-
sured by having more than one analyst respond to each
question for a single case study. The amount of interanalyst
agreement is then the measure of reliability.5

Weak and Strong Responses

A second common problem faced by those reviewing research

literatures is that some judgments are easier to make than
others. Certain characteristics of a case study may be so well
described that reader-analysts feel confident of their response
to a given question; other characteristics may only be poorly
described, perhaps requiring the reader-analyst to draw an
inference in order to respond to the question. For research
literatures of a highly diverse nature, reviewers would not want
to set their standards of confidence so high that only the most
well documented characteristics would be enumerated. At the
same time, reviewers would not want to set their standards of
confidence so low that well-documented characteristics would
not be distinguished from poorly documented ones.

The case survey attempts to deal with this problem simply by

allowing the reader-analysts to indicate, for each question
answered, their level of confidence. Such a procedure is well
5 known in traditional psychological research, in which observers
This measure of reliability does not address give levels of confidence, for instance, along with theirjudg-
the issue of the accuracy of the original
case study. Other than examining the case
ments of such perceptual phenomenon as the loudness of a tone.
for its research quality, only a replication of Levels of confidence have not been used as frequently in
the actual field experience would provide a
traditional survey research, however, since the respondent,
way of measuring the relation between
events as they occurred and as they were
usually the head of a household, may have neither the training
reported in the case study. nor the time to provide such a rating. In the case survey,

373/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 however, respondents are not simply members of the public at
large. They can learn rules for distinguishing among levels of
confidence, at least to the degree that they can indicate
whether they are sure or not sure of each answer. A not-sure
response is made every time a reader-analyst cannot cite the
specific text or phrase in a case study that contained the answer
to a given question.

Explicit Rejection Criteria

Exhaustive searches of a given literature will inevitably uncover

some studies that will not be used. In some instances, the
reviewers choose to ignore studies that are only marginally
relevant to their topic. In other instances, they may ignore
studies on the basis of poor quality. In virtually every traditional
review of the literature, even those using the propositional and
cluster approaches, reviewers have failed to make their rejec-
tion criteria explicit. This is a serious shortcoming in any
situation. In those situations in which a reviewerpresents only a
small handful of the available studies, noting that the remainder
were not of acceptable quality, the lack of explicit rejection
criteria is an unacceptable flaw.

In the case survey, all case studies that have been found are
reviewed by a reader-analyst who responds to a complete set of
questions for each case. Among the questions are several that
have specifically been designed to serve as exclusion criteria.
After all the case studies have been analyzed, the final caseload
may then be divided into those that have met the exclusion
criteria and those that have failed. I n this way, not only are the
exclusion criteria explicit, but subsequent analysis that com-
pares excluded with included studies is possible. In short, the
actual effects of the exclusion procedure may also be
examined.

APPLICATION TO THE DECENTRALIZATION LITERATURE

The case survey, as applied to the decentralization literature,
involved the uniform application of a 11 8-question checklist to
the case studies of decentralization. In the initial search for case
studies, an attemptwas made to includeall studies in published
or unpublished sources dated 1960 or later, but not including
doctoral dissertations. A case study was defined as any descrip-
tion of a site-specific organizational change in an urban area; in
total, 269 such case studies were found. For each of the 269
case studies, a reader-analyst responded to the entire array of
questions on the checklist. These checklist data then served as
the basic body of evidence about the decentralization litera-
ture.6

Checklist Reliability

The reliability of the overall checklist was tested by having two

or more reader-analysts answer separate checklists for 14 of
the case studies. Table 1 shows that the average amount of
agreement between two reader-analysts was 82.4 percent for
answers with a sure level of confidence and 60.8 percent for
answers with a not-sure level of confidence. The percentage of
6
agreement for the sure answers may be considered moderately
The decentralization literature also includes high, since most of the questions on the checklist involved
noncase materials. Such materials were
collected and used as background for inter-
multiple response categories and hence the level of agreement
preting the results of the case survey. expected through random guessing was well below 50 percent.

374/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 Case Survey Method

Table 1

Agreement between Two Reader-Analysts for 14 Case Studies in Reliabil-

ity Analysis

Level of Confidence % of Answers in Agreement

(% of All Answers)" Mean Standard Deviation
Sure (55.2) 82.4 8.8
Not sure (36.8) 60.2 16.9
No information (8.0)
Total (1 00.0)

Sure is where both reader-analysts were sure; not sure is where one or both
reader-analysts indicated not sure; no information is where one or both reader-
analysts answered no information to the substantive question.

The lower reliability of the not-sure answers, however, pointed

to the need for a separate analysis of all answers according to
the two different levels of confidence. The subsequent analysis
revealed two patterns. First, the percentage of sure responses
forall questions was relatively high (55.2 percent-seeTable 1),
making less critical the analysis of the separate answers
according to level of confidence. The lowest levels of con-
fidence were found in three of the five questions having to
do with the results of the decentralization innovations (see
Table 2). Second, where the level of confidence was low, the
distribution of answers according to sure and not-sure re-
sponses was quite similar. Table 3 gives the distribution of
responses for the three questions from Table 2 that had the
lowest levels of confidence. These two patterns meant that, for
most of the questions in the decentralization study, separate
analysis of the sure and not-sure categories was not necessary.
Future applications of the case survey method, however, may
require such a dual analysis.

Criteria for Rejecting Case Studies

Several questions on the checklist were intended to serve as

criteria for excluding case studies from further analysis. The
explicit nature of this rejection process and the ability to assess
its effect are two of the main strengths of the case survey
method and therefore deserve further discussion. The criteria
were meant to fall under one of two categories: internal and
external validity. These categories derive from traditional con-
cerns for research design, typically as applied to laboratory

Table 2

Level of Confidence for Five Key Decentralization Results (N = 215)-

Level of Confidence for All Answers

Sure Not Sure
Topic Covered by Question Number % Number %

1. Flowof information 154 71.6 61 28.4

2. Attitudes of agency officials 84 39.1 131 60.9
3. Attitudes of clients 85 39.5 130 60.5
4. Changes in services 140 65.1 75 34.9
5. Changes in client control
over services 111 51.6 104 48.4

See Table 4 for an explanation of the 54 case studies that have been excluded.

375/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 Table 3

Distribution of Answers for Questions with High versus Low Level of

Confidence (N = 215)-

Question 1: As a result of the innovation, the attitudes of service officials

toward the service or clients appear to have:

Level of Confidence

% Sure % Not Sure

(N = 84) (N = 131)
a. Improved 13.1 12.2
b. Deteriorated 6.0 6.1
c. Remained unchanged or
no information 80.9 81.7

Total (N = 215) 100.0 100.0

Question 2: As a result of the innovation, the attitudes of clients toward the

service or officials appear to have:
Level of Confidence

% Sure % Not Sure

(N = 85) (N = 130)
a. Improved 27.0 23.1
b. Deteriorated 10.6 5.4
c. Remained unchanged or
no information 62.4 71.5

Total(N = 215) 100.0 100.0

Question 3: The innovation resulted in increased client influence over

services to the extent that:
Level of Confidence

% Sure % Not Sure

(N= 111) (N= 104)
a. Clients implemented some of
their own ideas in
service delivery 26.1 18.3
b. All other 73.9 81.7

Total (N = 215) 100.0 100.0

See Table 4 for an explanation of the 54 case studies that have been excluded.

studies (Campbell, 1957; Campbell and Stanley, 1966;

Suchman, 1967; and Campbell, 1969). Internal validity raises
the question of whether a study's research design is adequate
to support the study's conclusions. A poor research design may
lead an investigator to mistake a spurious effect, such as
regression to the mean, for an effect attributable to a change in
the independent variable. Donald Campbell (1969) has listed
nine such spurious effects, which he calls threats to internal
7 validity.7 External validity, on the other hand, raises the question
These threats are history, maturation, ef- of whether a study's conclusions can be generalized to other
fects of a pretest on the posttest, in- situations. In a laboratory study, the typical problem is to be able
strumentation, regression lo the mean, to generalize from a population that has developed a unique
self-selection of subjects, subject mortali-
ty, interaction between subject selection exposure history-for example, a pretest that may sensitize a
and maturation, and measurement instabil- respondent and thus bias subsequent behavior-to the general
ity.
population, which has had no such history. Campbell (1969) has
8

These threats are the effect of testing, the

described six threats to external validity.8
interaction between subject selection and
Of these two categories, the decentralization study failed to
treatment conditions, the reactive effects
of experimental arrangements-for exam- develop any usable criteria for external validity. Several ques-
ple, the "Hawthorne" effect, inferences tions in the checklist were considered, but the reader-analysts
based on multiple treatments, irrelevant
responsiveness of measures, and irrelev-
found no adequate rationale for deciding when the conditions
ant replicability of treatments. for a specific case study could be said to be generalizable to

376/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 Case Survey Method

other situations. Any decentralization effort involves a specific

community, with a specific set of leaders and history, at a
certain period in time. Under these conditions, which are vastly
different from the laboratory situation, the rules for establishing
external validity are not clear. There is no satisfactory way of
knowing how to generalize from community to community or
from one time period to another (Weiss and Rein, 1970).

Two questions were used to assess internal validity.9 The

questions were concerned with the nature of the research
measures used in the case study and with the study's research
design. Because of the highly nonexperimental nature of the
decentralization literature, only a very weak criterion was set, in
which studies having either no explicitly cited measures or
observations or no specific experimental group focus were
rejected from the final analysis. Table4 shows the responses to
the two relevant questions. A total of 54 of the 269 cases fell
into this category, and thus there were 215 case studies in the
final caseload. Because of the relatively weak criterion, the final
caseload included many studies that would not otherwise have
been acceptable under strict experimental procedures. These
studies basically made no attempt to establish control groups or
to provide pre- and postmeasures. The studies followed a
research design that Campbell (1969) has described as preex-

Table 4

Checklist Questions Used for Excluding Case Studies

Question 1: The type of measures used in the case study were:

Cases
Number %

a. Operational measures' 69 25.7

b. A mixture of operational measures
other measures 24 8.9
c. No operational measures,
measures or observations that were
used informally 127 47.2
d. No explicitly cited measures or observations 49 18.2
Total 269 100.0

Question 2: The type of research designs used in the case study were:

Cases
Number %

a. Experimental and comparison groups, with

pre- and postobservations 9 3.3
b. Experimental and comparison groups, but
with only a-single observation period 24 8.9
c. An experimental group with pre- and
postobservations 19 7.1
d. An experimental group, with
observation period 209 77.7
e. No specific experimental group or
observation periodt 8 3.0
Total 269 100.0

Described in suffic
tion.

t
9
Excluded from subsequent analysis. Three cases fell into both these alterna-
Similar criteria were also used in another
tives; therefore the final number of cases excluded was 54.
recent review of evaluation research
(Bernstein, Rieker, and Freeman, 1974).

377/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 perimental, that is, carrying none of the weight of a true or even
a quasi-experimental design. As Table 4 also indicates, if only
those cases with acceptable research designs were used, there
would have been no more than 52 case studies under review.
The main point about the case survey method, however, is that
investigators can select their rejection criteria explicitly and can
select rigorous or loose criteria, depending upon the nature of
the investigation.
The problem of internal validity is serious enough to warrant
further discussion. At the heart of the matter is ultimately the
applicability of the experimental paradigm to social innovations
on the urban scene (Weiss and Rein, 1970; Campbell, 1970;
Harrar and Bawden, 1972; Wholey, 1972; and Cook and Scioli,
1972). There are many reasons why the paradigm may be
unrealistic and why some other logic than the experimental
framework may be needed to assess social innovations.

First, the paradigm may be unrealistic because an acceptable

research design is expensive and time-consuming yet seldom
arrives at an unequivocal conclusion. On the one hand, the
research can be so expensive and take such a long time as to
become a significant portion of the total innovative effort. On
the other hand, even in laboratory experiments, a single study is
rarely sufficient to establish an unequivocal fact; various find-
ings must be replicated under different conditions and in
different laboratories before a scientific fact is produced. Sec-
ond, organizational or community changes, and not mere
changes in the behavior of individual people, may be the main
focus of a social innovation. To this extent, an experimental
design that is based on the testing of individuals cannot
necessarily provide adequate guidelines for measuring complex
organizational and social effects. Unfortunately, there are few
measures of such a phenomenon as organizational or commu-
nity power, and no existing measures have achieved such status
as to be accepted by all parties as definitive measures. Third, it is
the nature of many social innovations that the actual treatment,
or program innovation, is tailored to suit the needs of the
specific community. Thus, even where the critical results are
agreed upon and measurable, conclusions are still difficult to
make because the innovative program or independent variables
are not easy to specify.

The most relevant occasion for applying the experimental

paradigm may occur in national programs, in which a series of
parallel innovations is attempted at different local sites and
where some single result has been clearly established as
primary and as having an acceptable measure. These conditions
may exist, for instance, in a compensatory education program or
a work-related manpower program. For urban innovations,
however, where different cities undertake theirown versions of
organizational and community change, there is a great need to
develop some alternative evaluation frameworks. Unfortu-
nately, existing discussions of the alternatives have not pro-
duced detailed treatments of any of the alternatives (Weiss and
Rein, 1970; Guttentag, 1971; and Mitroff and Blankenship,
1973).

Comparison of Cases of Different Research Quality

The use of these criteria for assessing the quality of case

studies may be examined in relation to other case study

378/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 Case Survey Method

characteristics. For this purpose, the original 269 cases may be

divided into three categories (refer to Table 4): (1) lower quality
cases, which consisted of the 54 cases originally excluded on
the basis of either no explicitly cited measures or no specific
experimental group focus; (2) medium quality cases, which
consisted of 127 cases that had no operational measures but
had some informal measures or observations; and (3) higher
quality cases, which consisted of the remaining 88 cases and
which used operational measures and a more appropriate
research design.

There were some case characteristics for which the variation in

research quality made little difference and others for which the
variation produced strong differences. The major pattern that
emerged was for studies of higher quality to produce only slight
variations in the frequency with which each of the five service
areas (police, education, health, economic development, and
multiservice programs), size of city, or major decentralization
strategies were studied. Table 5, for instance, shows the
relatively small differences in city size. Some differences were
found, however, in other case study characteristics: studies of
higher quality tended to be conducted more by authors with
academic affiliations and less by authors employed by indepen-
dent research organizations; to be supported more by federal
agencies and less by private sources; and to be more f requently
judged by their authors as reflecting a successful innovation.
Table 5

Population of City in Case Study, by Quality of Case Study (N = 269)-

Quality of Case Study

Population Higher Medium Lower

of City Number % Number % Number %

More than 500,000 60 59.3 90 70.9 32 68.2

100,000-500,000 19 29.6 25 19.7 16 21.6
Less than 100,000 5 7.4 5 3.9 4 5.7
County or township 2 1.9 5 3.9 1 2.3
No information 2 1.9 2 1.6 1 2.3

Total 88 100.0 127 100.0 54 100.0

X2 = 4.03, df = 6, p = n.s. (excludes no information cases).

Most important, the differences in research quality produced

consistent differences in the assessment of decentralization
results: higher quality studies are associated with higher rates
of success. This is especially true in the assessment of changes
in client attitudes. The higher quality studies more often found
such attitudes changed in a positive direction as a result of the
decentralization innovation (see Table 6).

In general, this pattern suggests that the effects of rejecting

lower quality cases does not change the scope of the study in
terms of the services covered, the size of the city studied, or the
strategies studied. It does, however, produce a higher rate of
success and hence a slightly more positive interpretation of the
decentralization literature.10This pattern thus reveals the poten-
10 tial significance in any literature review of the effects of
No systematic analysis was made of the excluding studies. The pattern, however, also provides an
relation between the quality of the study
and the interactions among the checklist
especially important finding about the decentralization litera-
questions. ture: contrary to popular belief, higher quality studies are

379/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 Table 6

Assessed Change in Client Attitudes as a Result of Decentralization (N =

269)-

Quality of Case Study

Higher Medium Lower

Client Attitudes Number % Number % Number %

Improved 33 37.5 20 15.7 4 7.4

Remained unchanged 29 33.0 67 52.8 26 48.1
Deteriorated 4 4.5 12 9.4 9 16.7
No information 22 25.0 28 22.0 1 5 27.8

Total 88 100.0 127 100.0 54 100.0

X2 = 27.98, df = 4,p( .001 (excludes no information cases).

associated with more successf ul cases of decentralization.

Whether this is because better researchers seek out successful
innovations, fail to report about innovations that turn out to be
unsuccessful, or because one of the by-products of a success-
ful innovation is the ability to stand up to more stringent
evaluative efforts, the fact remains that decentralization results
are more positive when research quality is higher.

FUTURE USES AND LIMITATIONS

This discussion has shown some of the results of applying the

case survey method to one policy topic, urban decentralization.
The method offers an excellent opportunity, however, to assess
other public policy literatures where case studies are also
important, whether these involve lessons from specific federal
programs orfrom significant organizational topics such as those
covered by the case studies of the Inter-University Case
Program.11 If the opportunity arises, the case survey method
can also be extended to the field to compare, for instance, the
characteristics of written studies with the results of new
interviews of on-site officials. In a highly subjective and con-
troversial policy area such as urban renewal, the results of
interviews of service providers, consumers or consumer rep-
resentatives, and the original research team itself can all be
compared with the existing body of written case studies.

While the potential uses of the case survey method are

considerable, its limitations should also be noted. The most
important is that the results of the survey are of no better quality
than the quality of the original case studies. Although the case
survey can be used to assess that quality explicitly, the substan-
tive conclusions about what the literature says are still limited by
the level of that quality. Second, the case survey method, in its
focus on aggregating general lessons, may not give sufficient
attention to the unique factors of an individual case. The tradeoff
here is similar to the tradeoff in behavioral research between
experimental and clinical research. Only the latter may provide a
full appreciation of the individual case; the former, however,
must be relied upon more heavily if the goal is to create
generalizations about groups of individuals. Third, the case
survey method may be more appropriate where the primary
concern is with assessment and not necessarily with the
11
discovery of process. Inquiries about process can be fruitfully
Greenberg et al. (1974) have begun to
codify the studies from the Inter-University carried out only where the existing case studies have focused
Case Program. on process and where the key to understanding the process is

380/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms
 Case Survey Method

relatively simple, as in the case of noting the concurrence or

sequence of several events. In spite of these limitations, the
case survey method appears to be an important addition to the
repertoire of research tools available to the policy analyst.

Robert K. Yin is a senior research psychologist at The Rand

Corporation in Washington, D.C., and an assistant professor
of urban studies and planning at the Massachusetts Insti-
tute of Technology. Karen A. Heald is a research associate at
The Rand Corporation in Washington, D.C.

REFERENCES

Bernstein, Ilene, Patricia Rieker Guttentag, Marcia natural social phenomena." In

and Howard Freeman 1971 "Modelsandmethodsinevalu- Herbert Costner (ed.), Sociolog-
1974 A Review of Evaluation Re- ation research." Journal of ical Methodology, 1971: 3-33.
search. New York: Russell Theory and Social Behavior, 10: San Francisco, Calif.: Jossey-
Sage. 75-95. Bass.

Campbell, Donald T. Harrar, William S., and D. Lee Baw- Suchman, Edward A.
1957 "Factors relevant to the validity den 1967 Evaluative Research. New
of experiments in social set- 1972 "The use of experimentation in York: Russell Sage.
tings." Psychological Bulletin, policy formulation and evalua-
Weiss, Robert S., and Martin Rein
54: 297-312. tion." Urban Affairs Quarterly,
1970 "The evaluation of broad-aim
1969 "Reforms as experiments." 7: 419-430.
programs: experimental de-
American Psychologist, 24: Holsti, Ole R. sign, its difficulties, and an al-
409-429. 1968 "Content analysis." In G. ternative." Administrative Sci-
1970 "Considering the case against Lindzey and E. Aronson (eds.), ence Quarterly, 1 5: 97-109.
experimental evaluations of so- The Handbook of Social
cial innovations." Administra- Whiting, John W. M.
Psychology: 596-692. Reading,
tive Science Quarterly, 1 5: 1968 "Methods and problems in
Mass.: Addison-Wesley.
110-113.
cross-cultural research." In G.
Hyman, Herbert Lindzey and E. Aronson (eds.),
Campbell, Donald T., and Julian C. 1972 Secondary Analysis of Sample The Handbook of Social
Stanley Surveys. New York: Wiley. Psychology: 693-728. Read-
1966 Experimental and Quasi- ing, Mass.: Addison-Wesley.
Light, Richard, and Paul Smith
Experimental Designs for Re-
1971 "Accumulating evidence." Wholey, Joseph
search. Chicago: Rand-
Harvard Educational Review, 1972 "Whatcanweactuallygetfrom
McNally.
41: 429-471. program evaluation?" Policy
Cook, Thomas J., and Frank P. Sci- Sciences, 3: 361-369.
Lucas, William
oli, Jr.
1974 The Case Survey Method: Ag- Yin, Robert K., William A. Lucas,
1972 "Research strategy for analyz-
ing the impact of public policy."
gregating Case Experience. Peter L. Szanton, and J. Andrew
Administrative Science Quar- Santa Monica, Calif.: Rand, Spindler
terly, 17: 328-339. R-I 51 5-RC. 1973 Citizen Organizations: I ncreas-
ing Client Control over Ser-
Greenberg, George D., Jeffrey Mil- Mitroff, Ian I., and L. Vaughan Blan-
vices. Santa Monica, Calif.:
kenship
ler, Lawrence Mohr, and Bruce Vla- Rand, R-1 196-HEW
deck 1973 "On the methodology of the
wholistic experiment." Yin, Robert K., and Douglas Yates
1974 Case Study Aggregation and
Policy Theory. Unpublished Technological Forecasting and 1975 Street-Level Governments:
Social Change, 4: 339-353. Assessing Decentralization and
manuscript, Department of
Political Science, University of Urban Services. Lexington,
Reiss, Albert, Jr.
Michigan, Ann Arbor. Mass.: D. C. Heath.
1971 "Systematic observation of

381/ASQ

This content downloaded from

139.153.79.195 on Fri, 01 Dec 2023 22:01:58 +00:00
All use subject to https://about.jstor.org/terms