\

DOCONIVT IMMIX
164 066 CR 019 010
-t

MITBOR DonabSdian, Auntie

TITLI Needed Research in the Assessment and Monitoring or
the Quality of Medical Care. NCHSR Reseatth ItePollt
Series. . -

--INSTITUTION National Center for Health Services Research and Oft

Detelopment (DHEN/PHS), Rockville, Md.

REPORT NO DHEN-PHS-78-3219
PUB DATE Jul 78
'MOTE 46p.
--kVAILA-8141-F-RO, National Center for Health Services Researcit, office
of Scientific and Technics': Infor 0.°11, 3700.-
East-Neat Highway,- R m 7$4, Hyatt vine, Maryland
20782- f-

EDRS,..PRICE 817-$0.83 HC-$2.06:Plus Postage.

DESCRIPTORS .
Bibliographies; -Evaluation CriteriavNeasurement
Techniques; *Medics). Care Evaluation; Medical
Services; Models; Program Evaluation;-*4militr.
'Control; Research-Methodology; *Research Needs;
andards; State of the Art Reviews '

S.

ABSTRACT
In this paper the author assesses 'the
state-of-the-art on quality assessment and monitoring or medical cmre
and makes recommendations for needed research in this area. Following'
a brief iutebduction, the content is presented in two sections. The
first, providing a frame'of reference, covets definitions; quality
assessment and program evaluation; relationship of quality.aad
quantity; relationship of quality and cost; strategies of care;
structure, process, and outcome; monitoring versus research; and the
uses,,of outcoges. The second section presents.a catalog of needed
research on assessing and monitoring the qualityNof medical care. The
research areas covered are as follows: basic explorations and studies
of what constitutes quality, description of prevalent patterns and
'sttategies of care, the epidemiology of quality, the relationshiP Of
structure to process or outcome, development of basic tools for
assessment, specification and testing'of system-design elements,
comparative studies of quality using different approaches, further
develqpment of promising current approaches, integrative measures o
qualitle appliehtions,to special areas, consumer perspectives and th-e
consumeWs role, quality assessment and monitoring, as a social
process, and effectiveness and the factors that influence it. An
extensive bibliography is attached. (EM)

..

i s-
**********4************************************************************
* Reproductions supplied by EDRS are the best that an be made *
from the original document. 4
* .

***********************4**************************** ******************
4
 RESEARCH REPORT
SERIES .

Needed
Research
in the ,
f
Assessment
and
"Monitoring
of the Quality.
of Medical
Care ts,

Avedis Donabedian, M.D., M.P.H.

Department of
Medical Care 'Organization.
School of Public Health
The University of Michigan,
Ann Arbor

A
July 1978

r
ft.
U.S. DEPARTMENT OF HEALTH.
EDUCATION A WELFARE
NATIQNAL INSTITUTE OF
!EDUCATION
U.S. DEPARTMENT.OF HEALTH, EDUCATION, AND WELFARE
THIS bonmENT HAS -SEEN REPRO-
Public Health Service 'MAE!) ()wry( AS tECEIve0 FROM
National Center for Health Services Research THE PERSON OR ORGAPIATION ORIGIN
ATING IT POINTS OF yIEW OR OPINIONS
STATED 00 NOT NECESSARILY REPRE-
SENT OFFICIAL NATIONAL INSTI TUTE OF
DHEW Publication No. (PHS) 78-3219 EDUCATION POSITION OR POLICY
2
 'yr

National Center for Health Services Research

Research Report Series
The ReiearchReport Series is published by the Na-
tional center, for Health. Services Research
-(NCHSR) to provide significant research reports
'in their entirety. Research keports are developed
py the principal investigators who conduct the re-
search, and are directed to selected users of health
services research as part of a continuing NCHSR
effort to expedite the disseMination of rkew kno I-
edge resulting from its project support.

Abstract
.
The purpose f this paper is to review, evaluate
critically, an synthesize the literature oh quality
assessment nd assurance, including the appro-
priateness f use of service, in order to arrive at a
'cogent, d umented, and authoritative assessment
of. the s te-of-the-art. In addition to addressing
quality ssessment as a research tool and quality
assura ce as an administrative tool, an attempt is
- blade to prbvide an understanding of the
the
de-
etiology 'of q -as a prerequisite to
s'
slcpi medical c rams and systems. Major
components of qtranitY:which are discussed in-
clude: definitions; quality assessment and progradi
evaluation; relationshito,of quality and 'quantity;
relationship of quality and c t; strategies of care;
. structure, prixess, and wilt ome;
monitoring ,:wer-
sus research; and the use bf outcomes. Rcom-
mendations for further reseaF.ch in the assessment
and monitoring of the quality Vif medical care are
.'presented.
I
 11 This NCHSR Research Report was written by Ayedis
Donabedian, M.D., M.P.H., Professor of .Medical
Caie Organization, Department of Medical Care
Organization, School of Public Health, The Uni-
versity of Michigan, Ann Arbor. This report, an
extended version of a paper prepared for the De-
partment of Medicine and,Surgery of the Veterans
1 Admini tration,. is one product of the work cue-
rently eing supported by the National Center for
Hea .Services Research under grant number HS
020 J. The final report from the work supported
by NCHSR is not expected to be completed til
June 1979..Dr. Donabedian was also supporte by
,

The University of Michigan, through sabbatical

leave, during part of this study.
Additional copiesof this report may be obtained
on. request from the NCHSR Office of Scientific
Ad Technical Information, 3700 East-West
ighway, room 7-44, Hyvasville, MD 20782 (tel:
17436-8970). Other current NCHSR publica-
i' tTons are announced on the last .pages. of this
publication. ,

C
 Prior to 1965, the year in which Medicare became In the past decade, a considerable body of HI
a'reality, efforts to improve the quality of medical knowledge has been gathered which requires
care were involved mainly with the self-regulation thoughtful review, evaluation, and synthesis in
activities of the medical profession, although order, to assess the present state-of-the-art and to
sporadic research had been conducted -in the areas allow meaningful projections of further research
of medical care process and patient outcome as far strategies. It is in this framework that Dr. Dotiabe-
back as the mid-nineteenth century. Utilization re- dian suggests that "it is necessary from time
view was emphasized with the passage of Medicare time to pause and take stock of what has been
c 1 portended a new awareness of public interest done, so that it may be clearly understood and
.1111 the quality of care. Dr. Lionabedian had the bp- future effort redirected." The synthesis which he,
rtunity during the period of 1965-1967 to re- provides is intended to mold together the re-
view and assess the state-of-the-art of quality as- search, operational, and policy concerns of the
sessment methodology and responded with several health establishment with regard to the quality of
publications which have become classics in the medical care at a time when major changes in the
field,. Since that time, there has been a period of financing and organization of health services are
significant, if not remarkable, growth in the body on the horizon.
of knowledge encompassing' quality -assessment
and assurance. A notable development in this area
was the implementation of the Experimental Med- Gerald Rosenthal, Ph.D.
ical Care Review Organizatibn (EMCRO) program Director
in 171 by the National Center for,Mialth Services National Center for Health
Research. This program esta ed the model fol., Services Research
lowed in the development f the Professional
Star Bards Review Organization (PSRO) pr9gram July 1978
and encountered many of the problems sub-
sequently experienced in that program. The
EMCRO program-addressed problems of criteria
development aid evaluation, organizational eat-
terns, development of assessment and assurance
techniques, impact evaluation, an&many of the
other emerging issues.of the day.
In October, 1972, Pu)blic Law 92-603 established
formal PSRO review of medical services reim-
bursed under th Social Security Act, and interest
in all facets of quality research received added im-
petus. The PSRO legislation has the potential for
profound impact on the cost and quality of medi-
cal services and the form of health services deliv-
ery; however, serious concern- has been voiced
concerning the wisdom of its current mode of im-
plementation, aspects of which have not been
rigorously validated. -. ° .0"
 #1 1°REWORD
vii NTRODUCTION
1 FRAME OF REFERENCE
41
Definitions t,
Quality Assessment and Program Evaluation
2 Quality and Quantity.
Quality and Cost
3 Strategies of Care
4 Structure, Process, Outcome
5 Monitoring Versus Research
The Uses of Outcomes

PROPOSALS FOR RESEARCH

Basic Explorations and Studies of What Constitutes' Quality
Specification and Assessment by Modeling
9 Empirical Testing of Strategies of Care
Increments of Cost and of Quality
10 Studies of the Client-Practitioner Relationship
Views of What Constitutes Quality
Continuity, and Coordlnailon as Attributes of Quality
11 Description of Prevalent Patterns and Strategies of Care
Studies of thd Elements of Clinical Behavior
Identification of Styles and Strategies
12 Comparisons of Norms with Practice
Homogeneity and Heterogeneity of Performance
The Epidemiology of Quality
13 The Epidemiology of Quality Among Providers
The Epidemiology of Quality Among Clients
14 The Relationship of Structure to Process or Outcome
Development of Basic Tools for Assessment
Specification and Measurement of Outcomes
16 Improvements in the Medical Record a

The Criteria and Standards of Quality

17 MOnetiry Measures of Costs and Benefits.
16 Spilcification and Testing of System -Desiga Elen nts
Specifying the:Appropriate Cut-Off Pointfin Sta ards
Sampling and "Enrichment:' Technique,
19 Larger Elements of Design 1-

dompirative Studies of Quality Using Different Approaches\

Furthei Development of Promising Current Approaches
Outcomes as Measures of Quality -

%Qtitcomes.as Cues. and Motivators for Process Assessment

..---\
) C
vi 20 The Otcurrence of Preventable Adverse Events
Disability
The Occurrence of Preventable Progression di illness or
21 Second Surgical Opinion Programs
Computer -Aided Management
...Integrative Measures of Quality.
Applications to Special Areas
22 COnsumer Perspectives and the Consumer's Role
Representing Client Values in the Definition of Quality
Clients as Sources of Information
Participation in Monitoring
Review Committees
23 Membership in Auditand Utilization
Patient Contributions to Care
Quality Assessment and Monitoring as a Social
Process
It
24 Effectiveness and the Factors That Influence
25 Changes in Physician and Client Behavior
Effect of Technical Design Characteristics
intra-Institutional "Social Design" Characteristics
and Medical Technology
28 Supra-Institutional Characteristics
and Medical Technology
27 Social-Organizational Characteristics Programi
Case Studio of Successful and Unsuccessful
jitiVEffeciveness and Cost-Benefit Studies

28 EPILOGUE \

29 References,
36 Curreit NCHSSI-Ptiblications
-114,F

Not too' long ago, the quality of medical care was :mite claim that the concept of quality is enriched vii
a matter almost exclus ely in the professional ben this happens. In my opinion!, this concept
domain. Any introduction of the subject in a more ca ) also become attenuated and less useful.
public setting was to be done, if at .ell, gingerly, Whether I ant right or wrong, it is clear that I must
almost apologetically, surrounded liy a cloud of circumscribe my subject rather narrowly if' I am to
caution designed to appease the wrathful and con- complete this paper. For the same reason, the
temptuous professional. How times have changed! delineation of the frame of reference must also be
"Quality" is now a term perhaps too easily bandittd 1, sketchy, for a thorough specification of the
about; and there is little hesitance in proposing framework is a formidable undertaking' in itself.
that quality can be measured, or that itican be en- All that is necessary. for now, is4kihat we establish
forced as a matter of public and administrative some common ground upon which to build.
policy. But this mood of almost belligerent confi-
dence is perhaps premature, for there is much
about the concept of quality that is elusive. unde-
fined and unmeasured. Our, knowledge- of_how 10
go about assuring quality is equally frail.
The academician who seems to he pleading l or
inaction while proposing further research is a
stock figure of ridicule in our gallery of public
fools. This is a role'l shall tryio avoid. My thesii is
that while some of us go about doing the best they
can with. what is known, the effort to examine
critically what is being done, and to find ew and r
better ways of doing it, should not be rel ed.
4,
This pa r is offered as a modest contribution
in this fu her. exploration. It will present a catalog
of needed research that is sufficientlyotganized to
avoid, being a mere haphazard listing of things.
But first, a general framework is needed that
explains and justifies the choice of research topics
and their organization into a classification. The
framework must also indicate what subjects are
excluded from consideration. For if reasonably Jr
strict limits are not set, at the very beginning, the
concept of quality has a tendency to expand, so
that it embraces every evaluative statement about
any aspect of the health care system whatever.
 A frame of reforms*
I
4

Definitions sonably liberil definition of his r )1e. It is not use- I

ful to define he responsibility f a health care
In some ways, my definition of quality will be practitioner, fir. any aspect ASt h th, so that it
tar sually narrow. Quality is taken 0) signify a, goes b.eyond hi)X)uthilly and p tfOssionally
judgment on a rather limited segment of the per- §a nut ioned role, oet heInstrumentalities actually or
formance of some professional personnel. Primar- potentially urger his citntrol.
ily, our concern is with the services that are re-
ceived and consumed personally and directly by
individuals, and with those profess ale who pro- Duality assessment and program evaluation
vide such services. Thus, for instant environ-
mental services and admin'strators f personal 64
Program evAluation differs from quality a3se.ts-
health services arc clearly tsidr ou scope; but mem by being tnore inclusive: It deals Jvith ac-
the patient -care service o physician. , dentists.
tivities of health practitioners in addition to direct
pharmacists, nurses, socia workers, . rid other patient care. It also includes the performance of
practitioners are clearly included. I belit e that it professionals and nonprofessionals who provide
is also reasdnable to include the services o es- ).no dlr t patient care. As a result, there is more to
sionals, such as pathologists and .diagnostt m evaluation than 'quality assessment; al-
radiologists, who generate the data that are used though quality assessment is part of program
in direct patient care.
evaluation, and in many instalrces its most imBsor-
Traditionally, the activities of these several pro- tant part.
fessionals are seen ws se arate contributions. In addition to this difference in extent or in-
Hence, one speaks of the q lily of physician care, clusivity, there are differences that flow from the
nursing care, social work, a so on. In this paper, level of aggregation at which the assessment is
I shall fo is traditit by focusing almost ex- performed. -Program evaluation deals at the
clusively on e services of physicians, but this is aggregate level with the manner in which a com-
not by pref7r n9e. It is a necessary concession, in plex formal organization functions, and whether it
part, to the reiStively less deNelsoped state of qual- meets its socially legitimate goals or some specified
ity assessment outside medicineybut, mainly, it is a needs or wants of its clients. In so far as the health
reflection of my ignorance of much of the work in care function is concerned, program 'evaluation
assessing the quality of performance in the other places greker emphasis on access to care, on other
professions. In fact, an essential component of the aspects of retionrce allocation, on cost and effi-
.toncept of quality is the interrelatedness of the ciency,- Ind on ,overall impact on a clientele, a
',contributions of the several professionals in the community or .a population. By contrast, it has
management of .'a patient', illness and health. A been customary for quality assessment to focus on
major- item on the research agenda is, therefore, the appropriateness of care at the level of personal
the development of "integrative" measfltres.of qual- interaction between' individual patients and prac7
ify that take into account this interrelatedness. titioners, as judged by the proper application of
science .to the needs that the patient and the 41.

Another way inwhich the concept of quality can practitioner have jointly defined. However, as
be expande4 is by. acCeptirig a definition of health health care becomes more formally organized, and
that is -considerably "broader than the traditional its financing collectivized; there is increasing pres-
- emphasis on physical-physiolOgical function. I sure to introduce into the assessment of quality at
shall includeipsychological and social well-being as the 'individual level concerns that were previously
necesiary components of health, but only to confined to the collective level. of analysis and
extent that responsibility for them can be assessment, This ha's raised serious ethical prob-
priately undertaken Wry Fakysician, und lems for the practitioner who is now caught be-
9
 Ion nil 1116411104 alum of Irmill11 14, W lit il is poor
tweet' the iwo milltonei Cif reoptinsibility Inwards unlot nog to
the huh..V.I.1111.
I patient Iimi the toilet tivits. It has quality at the aggiegmlfr level. It I%
I
alto blurred lrmerls t !eat er distini trans between are that, in this 11141Alitr, fir ifrilii of 'Call
program evaluation and gustily arsrsrnlrnt filAt icillimisiliir at liie level of i ulividual patient Late
.11111 at lilt, 4 (r( tivr level I all Ito
thr same be-
have now become outtally different es in emphasis
ha V 14U , train atiog ti fiatillisfliolIN 4 iniaelt e of
and in the detail* of humiliation, Some unin&
quaint assritsm4:and plow ant evaluation,
Hurts of this blurring mid overlap will be t !milted
Subiequelit sections otthis paprt. ()tn. esittoi le that ant evaluative statement
about theTplatility of tare:beyond a mete deicrip
lion of it. is a judgment about some aye( 1 of goal
Its. Flu that 1 eason,..mtlitation teview and quality
Quality and quantity assessment at e iiirxti ii .11)15 intet twined. mid will
Lou' so) 4 ottsitlted in this paper.
e overlap riWeeli iliiaTifiTV and quaint-- is title
con it through whit Ii plow ant evaluation and
2 clu,tlie Assessilicill flow into each other, obvious's,
quantitative adequat s is .1 net resat 5' pi et ondition Quality and coat
for quality tate. This means that access tot ale and
insufficient Tries.of servile are legitiniate elements We have alreadt had intimations that t ost is in)
in atisesking the of ( .11 lilionel II Ans- pin algid Fn the cow pt of quaint in a manner that
:11 well as of tin' ()grant .1% A u 'Waif is likely to (mist us problems. I am not referring
11(1(111 A« C'5% .10)11 use heed mtitiiiiiing simply iv the fat t that the mpit it al relationships
attention het use so mans plow ams air obss- between quality and cost ate essentially
-, 'thy conceptual
sivelt pivot c.upied. %till' I uring utilization and explored. Ain (.1 MCC r
I ut'

cutting costs, even though they mat pas lip service toonctions in the defiition of quality itself'.
The basic onsiderations that link quality to costs
to the important r of eliminating insuffit ient i arr.
Excessive ((Sr of servile usually pool (Iti:11- are essentially the same that connect quaint to
lit for a va (*it Of reasons. For one, implies a
it quantity: monetary cost, benefit to health, and risk
laik of ski in the «intim I of patir care: all in- to health.
There are two ways in whit h the monetary costs
:titans to t tweed from step to step along the. most
dire( th, selecting the pre( est's necessary and of inputs into care can increase without any in-
sufficient procedures to arrive at a diagnosis an k.7 crease in the quaint of care. First, the elements of
to institute treatment. This model of logical and service that go into patient care can be provided
parsimonious progression has traditionally "wen ineffitiently:l'flus, hospital care will l costlier
the hallmark of virtuositt in medical care, and has .necessary T hitspital beds are emptv., or the
mans still hold it as the ideal, although much of hospital is improperly managed: or if phi sicians
current acceptable practice MAN have departed do the work of nurses and the latter the work of
from it. Note, how-ever, that this is an ideal of Andes. Second, the elements of care it be com-
minimum redundancy, but not necessarily of bined and sequenced in a manner thai does not
minimum cost. It 'is possible, for example, that realize their NI) potential to improve health.
using current technology, a strategy of extensive, The fundamental attribute of these deficiencies
almost indiscriminate, initial testing will lead to an. in the organization, production and application of
earlier diagnosis, With lower likelihood of error. at care.is that there are added costs withoift either
lower average cost, at least in some situations. 1 added benefits or added risks. Can this be con-
so, the criterion of lowest cost could 1w in Conniff strued. as poor quality of care?* I have already ar-
with the criterion of least redundancy as elements gued that at the collective level of analysis the an-
in the definition .of quality. We shall return to this,. swer could legitimately be in the affirmative be-
interesting confrontation. cause, when resources aft scarce, wasteful produc-
\- In other ways*, the identification of poor quality tion and implementation of care reduces the po-
With excessive use is less problematic. Medical
tential benefits of care in the aggregite.,he an-
procedures, though intended to be beneficial, are swer could also be in the affirmative frthelevel of
not without risk which, in some cases, is consider- the individualfpractitioni-client interaction if re-
sponsibility for inefficiency falls, at least in part,
.able. It is reasonable to assert that the concept of
quality includes the criterion that in no case should on the practitioner. To justify this conclusion we
the benefits expected from any procedure he less can draw on two arguments that I have already
than the risks it poses. If so, a problem in assess- used. First, inefficient use of resourcesasuggests
mem is the measurement of risks and benefits in_a lack of skill or judgment' in the conduct of care: it
manner (fiat permits a comparison. is a manifestation of "logical redimAncy. Skc-
Still another undesirable consequence of exces- ondly, it is a Misallocation of resources, no less at ,

sive use is that by allowing some to have too much, the individual level than in the aggregate. The (
individual pays more than he ought to for care \-
there is'less available to others. There is an obvious qt
 either immediately and directly, or in the future its making such Judgments. using the relatively in.
the consequences of program inefficiency eventu- complete information available to them, anti when
who must, ultimately, foot the bill.
Not ever
/
ally work their way back to burden the irliVItial

ne will agree that wasteful tart is cafe

or poor qu y. It is nut Important to have agree-
the ahrtnatives are rather clearly demarcated. But
in older to make is more wedge determinetion,
the data on costs and benefits must he much, more
accurate, and reducible to , a common U1111 of
ment on th . What Is important is that, in evalua. comparisonhtr 4141111de. dollars.
dons of quality, the element or wastefulness hr The solution we have described is clearly in the
4'1/4 dearly identified and maimed. Once this is dense, mainstream of the (finical tradition beluse it rec-
$!e could be made either to keep it
rate or to merge it into an overall measure of
r
ognites that patients differ, not only in the . t illy
"wait al" feAtures of their illness, but also i the
quality. valuations ;hes place on the costs 01 care and its
There_k_nti disagreement that the balance of. expected treneficii Carried it, an -e-xtrrr., , tths
health benefits and risks is at the tore of qualiv Immolation would support the gut feeling mans
assessmeht. To do no harm is the most hallowed IA c link Unix have that the quality of care rests tit. o,+
precepts in the clinical tradition. No element of many individual xariltions to permit a general
care should he used if its risks exceed its expected standard. What prhcets it from going quite that
benefits. Similarly, the combination and seqtfant fat is the large area (4 agreement among patients
ing of elements shotild realite the largest het tits and prat titioners on the valuations placed on costs
relative to the risks incurred. I shall return t6 this and benefits.
last stipulation when I deal with the stcittegies of Under the solutiot proposed above:the quality
care. . or of tare at the collective level would be the sum
If quality is measured in trims of actual VP total of the quality of care determined case by case.
expected benefits to health from a specified course litl1 IhirTfIrtrUS1011 could be invalid if the costs
\of action, the relationship between costs and qual- were collectivized, for example through health in-
ity depends on the manner in which the factors I surance ;'if the I mit tells of hcahh #arc were shared
described above enter into care. 'Die elimination by more than the i.ecipiero; and if there were some
or reduction of wastefulness will allow us to have socially legitimate reason Tor valuing the health of
the same quality at lower cost, or higher quality at some, for example children, more than the health
the,-same cost. This also holds to the extent that of othetis, for example the aged. To the extent that
current care includes components that carry un- the collective interest differs from the individual
-..necessary risk. But the presence of such compo- interest, and the health practitioner is made the
nents
nents adds another, more sinister, aspect to the custodian of bola., serious strains are introduced
gs.icture. It is possible for care to actually deterior- into the patient-prattitionet relationship, and the
in quality as costs increase above a certain definition of quality postv moral dilemma.
po t, if the added services include a large enough
component of high-risk, low-benefit items.
Let us assume that the elements of are are Strategies of care
produced and applied as efficiently as possible,
and that only elements of care are used that have a Patient t or is a planned activity which involves
demonstrable benefit. The relationship between the choice of specific elements ol4are from a po-
quality-end costs, as more and more elements of tentially large pool of such elements, and their
care are added, is empirically undetermined-One proper sequencing in order to achieve specified
can, however, reasonably assume that the most diagnostic and iherapeuslit objectistes. A plan of lc-
beneOcial elements are likely to be used first and lion, as well as the panels-it of actions it generates.
that, as care becomes increasingly elaborate may be referredito as a strategy. A very simple
through the addition of more and more elements, example can be ssenIbled from the work of sev-
the relatiie gains in benefits and quality become eral investigators who Ave dealt with the choice of
smaller and sma n other words, there are strategies for the management of acute pharyngitis
diminishing ret ns and, in the end, very small in childrep and adults. '
advantage to be erived from further enrichments According to Brook, the development of process
in care or addit nal quantities of it. If we'accePt criteria in cooperation with infectious disease ex-
this picture. the question becomes: At what perts resulted in the following - recommendations
level of inputs into care should the standard of for the treltment of adults with sore throats:
quality be set? If individual patients paid all the "If history and physical indicate eukclence of a sore
costs and reaped all the benefits the problem throat, a throat culture should be clotie. If the cylture
woultilke easily solved by saying that the. patient is positive, she patient should make a repeq via
and his physician cap jointly decide at what point within 24 hours and be given a shot of procaine
alf added costs are not worth the added benefits. I (short-acting) penicillil; he should then wait 30 min-
utes in the office so thlt he caribe promptly treated if
bilieve that patients and physicians are capable of an anaphylactic reactibn occurs. After another 24
-1-1
 It no print Olin
i
sit alrgiroi, whit 11 memos 11141 the mysteries of dine
.

hours, the pandit should 'Immo. 14) lodgment ,rte amenable to ieliling their clerk
the mire %cuing none,
Allergy has 'limn kora null 111
he should he *tyros 01 shad beatiailinsi thing aciiugi est se, 1 els I hrsr owls Air Oct ision A11444111 Mid
prisktIlin ' the analysis 411 (nil el let tiveness Alld isatthellerhe.
rt Al a smiler lint, although these tools at AVilildhle, the ilata
In the example filed hs that 41 r needed tin then poet-tie application are
stresiegst is det Lard. milt the atithoins ,,t ryes growl ails 141 king, We need more Alt mate inlOrf
.represtitt "optnital-spialitv of (Mr I Initsik in
el of
1114114 .41ut I he' as t "mem r of illness and
al. propose to frL whether-yr stnsltat silategs Is, in a 11nt al and lahni 41.11 v hinting% in association with
fat I, optimal !iv corn pit! MR thi cc II v pot hent al sot It illitov; ahittot the Mollehil y 1 MIS and other
ittategies lot the panagrinrot of mote throat tit mks assay laird with diagnostit ',mediae*. or
Val 4011s sequences of them, in t ortectiv identifying
11111)1 s I11 1,(11 .111111) 41
prisons who eh) i in( Iyavr
level I he dove abr. DallVes it elitist% anti el I ntiellUsly missing
trigs illiir when
4 re rusting illness. ,ifAnit the ria,,,s, frordent- and-
tionirt.o S Imo, ad .thrIluilve therapies as rt)M
-A ihr a anielith leoullnatral :11.urgi. an intim,. pared to 011'11 1 1)1111 a/11111)11S In health; Acne the
the lin nut la All 11.4(trnt ...41 fit 11/".4/ 141111 V1111111

111 ilifille Volilli1C a 'Muir. Mg' 11411111/41'

11, gt (mil A 'dative valuations to br placed on various man-
tutrptut IH I I. ifestations ad health or ill health so that these tali
II he rat all imi irony wai prim illm %.. uhtun tiwg a be added up ism; a weighted sum that act taillely
1111)141 t ninny.. telly. Is then total impact; and, where costbeuefit r
t nrilltrt II ulitite nut 11c.11 .1111 nalic111 I
1 111111411111111 .ter to be made, a MI111111111 unit of /
1.0 1 otripli4 air matteis !indict, these so mews ate IllrallIlleliteln is Ileedet lh that permits a comparison/
tested foi epicleinn versus rildenin (Ha all 1(11(1' of Even when all the In ii inatnut needed is not
the clisrase, lot oial set sus paienteial porn t illin, available. 111411V klIISIg11111 11111) 1111111M judgment can
and for chi leurrit espy( teal tiles all ihittive I hi nal be gained by making use of what is already knawn,
(-tailor in the population a rpm (nig lot a ;ate. supplemented by the opinions and valuations of
1;111A1 II) has tamale the Inn ificatinli of \Iitategies those tv. II note prat time is imalt.r study, and of their
taut the management ol
at lite ph.1 t s ng tv rsru patients.
more complex by itittodin Mg as a hist- step rules Ot1V101114. a (lest ription of the techniques which
that. on the basis of (Italica, rindings. (Lissa,. a airs we have mentioned is beyond
the scope 7of this
as ( I ) most probably (auscal
In group A beta paper. Lusted has provided a reasonably simple
hemolytic streplocmc its. (2) most pi obabls not so exposition of the clinical applications of decision II
With ;I analysis.' A recent paper by McNeil et al. may
(aimed. and (3).(plestionable as to ethology',
further subdivision Of (awl into adults and c hil- t
serve as a quick intro( iction.a The more ambitious
is
alien six categeiries of patients ate 1111'1101rd: three reader with an aptittu e for quantitative methods
by etiologs and two In age. For rat Is of these it is referred to a. more rigorous exposition by Raiffa.'
possible (I) to treat all cases without prior ( tailor- kiarinato has published a brief account of cost-
ing. (2) to culture and then neat, and (3) not to effectiveness anti cost-benefit analysis as applied to
treat at all. As to treatment with print illin, it can excellent
medical technology, anal provided an
be either anal or parentetal Illus. their is
a bibliography.' Further discussion of the basic .
matrix of 36 possible combinations, and for Cal 11 is methods referred 10 a nd a large number of
applations to 111C ass ent of strategies irr----
is possible to test the consequemes.i s

)
Elaborate as )I f t his sees
m to be,thct\experienced
clinician will Fri qoile, it as an oversimpliticatitut
surgical care, . to he In
edited by Bun er et al.' I
in an excellent book
refer to other work
of a situation that is itself rlatisels simple. More in a subsequent secti of this paper; but infly-The
complete specification of strategies requires the surface tip be s med. iince this is an area
construction of rather elaborate protocols, al- cnrrently tinder intensive investigation, We are, at
' gorithms or trees. Some reference to this last, experiencing a major advance in our under-
work will be made in a subsequent section. All I standing of key elements in the quality of care. It
want to do do now is to la v. down the foundation is a very phlegmatic person indeed
who will not be
for asserting that the description and assessment stirred as he first sees, as it were frcim a moun-
of the elements of care, one at a time, misses the taintop, this new and enchanting landscape!
design, the rationale and the implications of the
strategy as a yehtde. including the consequences of
taking as well as pot taking certain actions. In my Structure, process, outcome
opinion, the very essence of quality, that elusive
but all-important ingredient that we (-all clinic-al In our framC of reference, when a judgment is
judgment., resides in the choice of the most appro-
made on the quality of care it is taken to be, by
priate strategy for the' management of any given definition, a judgment primarily on what profes-
situation. I also bell ve that we now have the sionals do, and how they behave, as they interact
necessary tools for tipecifying and testing such
 ,11

-directly with their patients.,'Hence, it is the process ing knowledge of that relationsheis used to olk
of sate hat is: ultimately; the object of quality as-, din information about the, behavior of profes-
iessme t. Quality is defined as the degrecof con- sional personnel or of the larger sYstem- I
formance to, or deviaiiOn from, normative. be- But, dogs monitoring have a subsidiary risearcir , A
.havior. 'this formulation, both structural attrib- function? Is "'monitoring analbgous to clini(il
utes arfd outcomes areindirect means cif obraining _medicine, where a physician Learns how to manage
information About the hormativeness of prifiCess. cases by observing the outcomes of cafe? I think
',The rather secondary role assigned to the as- the answer is bOth 'rtes" and ':No." It is "Yes", itra.
sessment of structure. is not to- be 'seriously restricted sense: if 'the. occurrence of unekpected
.ettallertged.'ACcordirigly,In4rder to ynal e my task good, or bad ottcomes leads*to '4 review of pfoCess" ,'
'Manageable,:l shall have little to say. about struc- in the light' or aken fly .krlwn redlio.nshipi -I,:,e-
turein, this paper; althOugh it will; not be entirely tween the two; and if, iik the event cutrent sknowl-
-precluded. Tho,situation is eidicrely different 'with edge does- not iirov,ide.an answer, questions' are
.resnestao__outsomos,, since,-_ according to ,a --large . raised about that knowledge so at,t,o suggest fur-
c
-body of opinion, including that 'oft inany leading , tiler research. per answer is alinott always "No," if
, experts lualify.asternent, it Lsthe outcome Of .1
we we 'expect tile -moniteri eclianism_itself to'.be
care that is'the pri miry objectt.concern, and that, further research :' Th establishment of new,,
process 'only' a means_ to -the attainment ---.--iinkages 6etween process OM outcome-can only be-
come. It is ,with'clifficlenc.e, 'and with some-apology, achieved,.With'illy .gertatuty,, through carefully -
'that a Methodbased on process. cap, -controlled and meticulously 'conducted, clirtical
I/entitle inta:this hostile enyiron'tnenwrhereas_it is - trials ..,It is unreasonalste and ,iclingtiions So. expect
'a proud badge OC,hondr, assuring- alnadst instant tterery PSRO. or its analogu can asa
'attention and respect, to 'say, dfat a inethodt. research'agency't hat tells us w good medicine
"Outcome,oriented$3,.;Some have ito hesitation to is. One might wiihetual reason a Frt, as has been
even distort reality, relabeling process elements as done to the `past, that: the best teskofthe useful-
otftcomes, in orcleX to avoid . the 'obloquy that
attaches to pioc-ess Aid to bay in the approbatiun -
ness ora drug is the sum of the.judgmehts' of indi- ..
vidual physician's as they y-okderve its effects on the
t;

thit;outdomes confer.. Of course, this picture is - management of 'their individual patients. .0.,./
linveidr,aw,n, but not by much! 4.1

It is not true that outcomes are a more valid

;mtasure-of,quality than is process, as it. is faihion- 'The uses of outcomes
sable to say. It is true that proceSS -pleasures are
valid indicators of quality only to the extent that, Nothing I have said sO'far Should be taken to
they relate to relevant outcomes. But it is equally mean that,outcomes a4 not important iniquality
"- true that outcomes are valid measures of quality
only to the extent that they relate to the antece- assessment and monitoring.,puite the reverse is
dent process of care. Fundamentally, 'validity de-, true: Lei me count the ways.
pends on the strength of the relationship between Outcome-s, usually undesirable ones, can be used
process and outcome, and on our understanding as 'a method, of sampling 'or screening in order
of that regionship. If that vital link is weak; or in to increase the yield of process assessment by
doubt, Reither process nor outcome can be used to concentrating on cases with such ,,butcornes.
astessTqu-ality; the validity of both is attenuated, Perhaps the clearest example of thiS strategy it
and to an equal -degree. This, means that, in 'this to b found in the "problem-status outcome
: instance, we do not know what constitutes qulity. method described by MuShlin et al.9 It is also vis-
Further'. research is needed; the link befween i_ible as an important element in the Performance:
procesi and:outcome must be stigated. In such Evaluation Procedure (PEP) advocated by the
an investigation outcomes are the,,only reasonable 4 Joint Commission on Accreditation of liospi-
criterion. There can be-n<disagreement about tals.10
that, provided the proper outcomes are selected' 2. Outcomes can be tied as a proxy for elements
and appropriately measured. of process which are difficult: to treasure, or
about which information is hard to get or is ab- 8
sent, provided the causal linkage between out-
, come and process is reasonably well established.
Monitoring versus researeh 3. Outcome items can also serve as a supplement to
monitoring process in order to ensure that im-
It seems to me..that much of the emphasis on the portant process elements have not been over-
primacy of outcomes in quality assessment arises lodked. In this case, they provide an added tier,
from a fundamental confusion between research so to speak, in the s.eillance system.
and monitoring. In research, new knowledge is 4. Outcomes can also serve as a feedback
sought about the relationship between outcome .amechanism that may lead to questions not only
and process.. In assessment and monitoring, exist- about whether certain process 'elements are
 "adequately represented in,Toricoring but, more treated disease may makg it iinpossible to
fundamentally, about assumed relationships by specify accurately future .outcomes, limiting
tweet] proceis:and outcome. As previously dis- their-use in certainassessment schemes:."
cussed; this latter is the point where-monitoring The duratio9 of outcomes as well as their
and/research are most likely to intersect. magnitude must be taken into account. Hence
on a long-term perspective, 'often
5 The inclusion of outcome assessment can also as long as a lifetime, ind, the measurement of
asemphasis
serve to reinforce problem=orierited'11 ,manage- health stattis.' 4
.. ment, or "management by objectii.res." 5. As 'a corollary to the aboie, pne needs to
6. Attention to the' attainment of prestated out- consider the possibility -of trade-affs between
conies can motivate a more serious exatniation magnitude and duration for any one outcome,-
'of process and be a powerful spur Iii reform. In
Williamson's work on "health accounting," for )and among severs) outcomes. For example a
... .

example, one finds not only an-implicit prefer-

shoertr,,life.at a higher level of function may
. have: to .be weighed against. a ,longer life -bur-
ence for management by objectives, but, ''also, a dened with greater disability. )
deliberate reliance on confrontation with fa ire
6 .
.
t. As ano Pier corollary, accurate information on'.
, irrordar to jolt physidans, into action.' r ant ottlAtikes must be available and
7-Outcomes reflect the impact of all the elements \ the
'Na.! go intecare,_ and of much else besides. the ou omits Must be subject to 'reasonably
'They have, therefore, an integratiwe, property precise measurement. Since. definitive:out-
which allows them to represent the contribution comes- rhay 'not appear except after long
periods of time, inforMation may be hard to
.- 'of all the healtS prOfessionals to patient care. get:
They also triclude the contribution of the patient.
7. It is necessary to examine the consequences not
to his own care. Unfortunately, this useful ay- only of taking-a specified action but, also, of
ity to pull together all these influences is also a
weakness, if cknewishes to explain how the ob- not taking such actiot 'in order to get a
complete picture. Thus, in evaluating the ef-
m

served outcomes come about.'

8. Outcomes can not only include the patient's con-
fectiveness of a sargicarprocedure; its is neces-
tribution to care, but also provide a means to..en- sary to follow not only those who have had,
operatiois but also those whO might have been
list the client in the process of quality assessment 8
.
candidates but were not. .
and monitoring. The specification of the techni-
cal elements of, process is an esoteric profes- 8. The attainment of outcomes cannot stand
sional enterprise about which the client can only alone as a measure of success. The means used
have rudimentary and, possibly, distottted in achieving. these outcomes have; also to be
I knowledge. By contrast, the patient ,has a great considered, unless it is assumed that resources
deal to say about the interpersonal component are un'imited, which is far from, true. This
in the process of care. He has at least as much to simple truth hasj-only occasionally been ap,7_
say about the outcomes he expects, to what ex- ;predated in assessments of quality, as distinct
tent these are attained, and how different out- from utilization review;35 but it has received
comes are valued relative tos.each other and to much attention in program evaluation, under,
re costs incurred' in attaining them. the heading of cost-effectiveness and cost-
I have been successful in my exploration of benefit analysis.' In this context, it is useful to
the role of process and outcome in quality assess- remember that the costs are not only in re-
ment, one should never again hear a preference sources expended or risks taken; any assault
for one, over the 'other, except by the poorly in- on established values and social norms is also a
formell. But, I am also realistic enough to know 'cost which deserves serious attention, unless it
how forlorn a hope this is. No doubt the debate isr regarded as desirable and considered to' be a%
will go on. In the meantime, it:` may be useful to benefit.
piesent some general principles that govern the 9. At an equally fundamental level, the problem
use of outcomes in quality assessment. of attrib has-to be handled. Outcomes,
1. Obviously, the outcomes selected should be whether od or bad, must be attributable,
relevant to the goals of health care in any par- first, to medical care and, then,' to the per-
ticular situation.
forniance of those unde- r .,assessment. Natu-
2. This al,o means that the outcomes must be rally, the longer the time that has elapsed be-
achievable by giiod care and that, the in- tween a specifipd activity and its consequence,
strumentalities needed are available to, and the more opportunity there has been 'for the
intervention of other factors, and the more dif-
under the control of, those whose performance
ficult it becomes to assign responsibility for ob-
is being assessed.
S. The outcomes must be. specifiable in ,mag-
served outcomes. /
nitude, frequency and timing. The paucity of 10. To conclude this, decalogue, when adverse out-
information about the course of untreated and comes are used to assess quality, one must al-
,

14
 most always, examine the antecedent process/of
care to find out what Went wrong, and hOw it
might be corrected. The search for causes and '
remedies, will often lead, ever) beyond process,
to an examination of tire.; ural characteris-
tics that have encouraged *scouraged speci-
fied behaviors.,The quality:of care cannot be
fully comprehkuded Or successfullyared
a without understanding how- structure infl*
ences protess,and process influences outcome,
No*atter where one starts in this chain!, one
must ultiniately deal with it as a w,}Kile.

f.

15
 Proposals for research

Morin of algorithms or decision trees, are not only

l begar ihis paper with a' general frame of ref-
8 morp precise and realis(ic representatioris of what
ere,nce hoping -that it would give meaning and
.

is c nsidered to be goods 9r , but also provide ihe .,

perspective to the listing 'of research proposals; ' "opportunity to test altern ve strategies and to
but, also, that it would generate a corresponding. confirm or modify normatie standards.. Such
classification. Unfortunately, no truly satisfactory formal testing is/necessary b use in these corn-
classification was found; because there is a great plex'situations, the best course of action that intui-
dpal.of overlap among the several categories that tion dictates may not be the best indfcated by deci-
emerged. Nevertheless, I hope that the following ,
is a reasonably orderly presentation in which the i....sion analysis.
The usefulness of this effort, then', is-that it pro-
reader can discern the major features described in vides the basis for formulating criteria for assess-
the introducston. There is also some attempt to ing the quality of care. The models are also an im-
have a very rough progression from studies deal- por.tant tool in medical education, as' a vehicle for
ing with basic concepts and measurement tools, to specifying and communicating to studen the in-
those that deal with implementation; but too much tellectual operations that constitute cli I judg-
should not be .made of this. Finally, there Is'the
problem of specification. Obviously, it is not possi- ment. Moreover, th -attempt to construct such
models reveals deft 'encies in current information,
ble to be very specific in a general review like this. identifies th st critical defects, and suggests
The text only sketches areas for research. But, the research 'needed. to obtain the required
whenever I could, I have cited examples of re-
ported studies that might serve as more detailed information.
will find not only con- Illustrative, examples of this apprOach to the'.
models. In these the reader exploration of diagnostic strategies may be found
crete embodiments of the more general descrip- in the work of McNeil et al. on' screening fqr
tions in the text, but also; I hope, the raw material hypertension" and on diagnosing pulmonary e&;
and inspiration that generates new research. Since bolism;22 and in the work of Neutra on the
the studies cited are meant only to be illustrative, decision to operate for apRendicitis.." Examples of
the list of references does not serve as a systematic the specifitation and testing of more complete
bibliography.. strategies that include -treatment as well as diag-
nosis are pr ided by Schwartz et ak for essential
hypertep d renal artery disease;" by McNeil
studies of what consti- Adelste for hypertensive renovascular
Basic explorations and disease;2° and by Tompkins et al. for acute
tutes quality pharyngifis.2
A review of these publications illustrates the im-
Specification and assessment by modeling As a portance of costs as a factor i4 the analysis. Non-
first step in exploring what constitutes quality, at monetary 'cost' in the form of risk is always a factor.
ast in the technical sense, diagnostic and In many instances monetary cost also figures
erapeutic management should be described and prominently, for example in the papers by McNeil
assessed as planned and sequenced activities or et al." " and Tompkins et al.2 The problem of
strategies. The models of such strategies that placing values on' the consequences of various
emerge will have a certain newness, although they k courses of action bedevils the analysis. by Schwartz
use only existing knowledge and opinion. This in- et al)" Methods for valuing the quality of life,.in .
formation is obtain d from reviews of the litera- addition to its mere duration, are briefly discussed
ture, from retrospective review of case records, by Abt. "
....- ,
from the opinions and values of expert ,clinicians,. r`
Even a nodding acquaintance with this work.
from all of the
f and from reasonable extrapolations
construct& Pti the suggests opportunities for further research, be-
above. The models that are .

16
 nlhtl its extension to cover a wider range of tio-n of the I b at the end of three years, mortality
`51nical situations. wOne line of inquiry is to go from the procedure, the occurrence of immediate
,Yond determining how to best arrive at a certain complications, the loss of patients due to non-
lugnosii by studying manner in which cases . attendance, the current monetary costs of the
resent clinically ide tillable problems can be procedures, and t it indirect cogi as represented
---3'itned several diagnoses. The work of Ginsberg by travel time for t atments, dayptaken off workre..
may serve as an illustration.22 Another interesting and loss of earni s. There was no attempt to
study
Da would.be compa'ring tire stepwise, logically construct 1 singl measure for costs,and another
approach to diagnosis by the "shot- for benefits, but the advantage was clearly with
gun' approach; which uses a very much larger sclerotherapy. The implications of such studies to
fitnnher of tests, at least as a first step, to quickly uality assessment and to social policy are obvious
b e 4rowdew nthe realistic alternatives. The possi-

gains in time and in the discdvery of illness

And important, Lssuming that their findings can be
accept as valid.
should
be weighed against the monetary and non-
m o netary costs.
I 9
3
IncremInts of cost, and of quality As I tried to
.. E show in"the introductory section of this paper, the
14Pirieal testing of strategies oreaire Modeling, relationship between irwzements of,quality and of
wrie'ts Illatter how ingenious or creative, is limited by cost is a matter that deseNes special attention. One
' -.0'44'
tionshipiS curr4ntlY known or believed about the rela- way of studying it is through modeling and simula-
process and outcome. Additional tion. Using this method, McNeil et al. have esti-
tli:cellviedge must come from empirical studies of mated that it costs $8,485 to identify the last one
elements of care or the more cdmplex
strategies out of 20 cases of pulmonary embolism in a'young
of management. This may b ccom- adult population with pleuritic pain, whereas the
:rexpriments,.;
Srled by observing what might be called 'natural preceding 19 cases could be identified at an aver-
but definiEive conclusions can only age cost of $595 per case.'7 Even more striking pre
me from controlled trials, which cap b9 re-
g Ned as an extension of research in clinical
the cost stimates of Neuhauser and Lewicki fdr
proce te for finding cases of cancer of the colon
a
medicine. gut clinical research is primarily cdn- l' hypothetiCal population of 10,000 persons 40
Rcirhed with .the comparative effectiveness of alter- years old or older." The procedure is to first do a'
natives;
and relative costliness, or the balance of costs stool gimiac test and afterwards to x -ray, tho who
trialsbenefits, is seldom an issue. When clinical show at least one positive test. What is varied he
expand their scope to include not only risks number of successive guaiac tests that are incl ed
''-`tlincl benefits td health, but also monetary costs and in the first step. Obvi?iusly, as the number of uch
tiellerits, they 'address questions of social policy tests is increased, the number ofcases who sho at
at are also essential ingredients in quality.
asset least one positive test increases; more cases wit
-1. 11-8thent. - cancer are detecte4, but there are also many more
"e algorithms that currently define acceptable false alarms to be allayed by subsequent barium
Practice
ion, derive their authority from expert °Erin- enema. Accordingly, the cost of finding' an addi-
The work of Meyers et al. is one rather tional case of cancer rises steeply as the "quality"
41.1a1 example that is a 'reasonably direct ,em- of care, judged by the amber of stool guaiacs
1,31:rical test of such an algorithm: one fof the prescribed, increases; so much,so that the cost of
diagnosis
work of meningitis in children.23 Some-of the finding the additional Case is_ estimated at an as-
al. tel, cited earlier, for example that of MCNeil et tounding $47 million when 6 tests are done,
4 the diagnosis'
be regarded in the of pulmonary embolism, can' whereas it is only about $1,000 when one test is
reCortis, same light, since it,uses patient used as the Screen, and about $49,000 when three'
nostic though retroactively, to construct a diag-
diag- tests are used. The values that are generated by
Stu ri kt ra teg Y . 1 7 A modest proipective study by these models are, of course, heavily influenced by
,), t-evant and Stern that tests the ability of tb models' assumptions; and they could be inac-
r-Ysiciarts to predict the finding of a stone on
cholec 'curate. My intent is to emphasize not the findings,
esti YstograPhY
, .is relevant because- analogous but the nature of the method and its utility.
bette tes are used in decision arialysis, when no As an extension of such explorations, under es-
more data ate available.24 An example of a much sentially hypothetical conditions, itynight be useful
quettc_.8 mbitious clinical trial that tests conse-
. to have clinicians formUlate a set of ,,increasingly
foutig; additional to, medical outcomes .may be stringent criteria of the quality of management, as
econo,in tha,livork of Piachaud and Weddell on the well as an estimate of the gains in health expected
study "l ics. aP treating 'varicose veind.25 In this from such stepwise increments of quality. Cost es-
tive merits of injection-cortipression
scleroti..tbe,rela timates could then be made, and the expected
Ai erapy and surgery were compared in benefit compared to 'the cost. Hardwick et al. im-
controlled
Ned eXPe riment, using as criteria the condi= vieTnented what is in effect, the firSt phase of such
 in the other, and actively engaged in research i
a procedure'when they asked house staTeand gen-
eral practitioners to specify the diagnositc tests both.
they would perform for specified conditions (1) a,,1 The literature pertaining to the client
Practitioner relationship is too vast to permit a
an absolute minimum, (2) routinely, (3) if .tie case quick summary..A text book review by Bloom and
were&to be presented at a medical gran lc:kinds, Wilson serves as a useful introduction." Lebow
and (4) if it were to be included research
has reviewed the literature on patient satisfaction;
study.27 discussed some of the problems of method, and
Needless to say, embirical studies are necessary made suggestions for further research." A specific
to establish the relationships between quality and example of a clinical trial that examines the- conse-
cost. A study of secular, trends in the inputs into quences of a change in the client-practitioner 'in-
care for carefully specified conditions as compared teraction is the study by Inui et al.,- which shows
to the outcomes of such care could be one initial considerable improvement_in the-control of hyper-
approack-A. good example is the study by 'Martin
-tension wher4hysicians shift their attention froin:
et al. of inputs into coronary care compared to the the manifestations of the disease to thGtbehavior of
mortality during hospital stay.28 While such studies
10 the patients." These encouraging findings can be
are of great interest, contemporaneous compari- placed against the discouraging experleztce in
sons of costs for similar services at different levels another, cAnical trial of the lack of success with
of quality may be' more credible. In one such either of two strategies, "augmented convenience"
study, Jackson and Smith found no relationship Or "mastery learning," to improve medication
between estimates of the quality of pharmaceutical compliance in hypertension."
services in community pharmacies and the charge
per pre ripti .22 Similarly, Schroeder et al.
fot*sd no, r relationship between the cost of
laborator7 services ordered by residents and the Views of what constitutes quality The research we
assessment of the competence of those residents by have reviewed and pcoposed so far is expected tp
their supervisors, even though thereJwere wide give a mordprecisP answer to' the question: What
variations in both estimate§." Rubenstein et al. constitutes quality?, Another approach' is to study
have reported' findings that suggest diminishing the opinions held,,on this subject by clients,, ad-
returns in outcome thtingswiih increments in the minisa-ators and professionals. A* comparison'
quality, of the process{ of:care.31 These are only a should be made of the attributes that are believed
handful of studies, but they show the way for to constituitnuality and of their rankings in
more. importance. The findings would -be germane not
only to quality assessment, but also to under-
standing problems in system performance.
'Studies of the client - practitioner relationship An example that fits precisely in this category-is
The research I ...have _proposed so far defines the study by Smith and Metzner of the opinions of
quality in terms of the wchnical elenients in clini- patients, physicians and nurses on what constitutes
cal Judgment. There is a corresponding universe quality of care in prepaid group practice." Mar--
of concern with the judicious management of the ram's work on how' much credence nurses are will-
interpersonal relationship. between clients and ing to give to the opinions of patients concerning
Rractition.ers. Here, 'also, there are differing different aspects of nursing performance suggests
modes of interaction and varying styles and an interesting topic for further study." The litera-
strategies of management that are more or less ture on patient satisfaction to which we have al-
successful in achieving deOrable outcomes. These ready referred includes much material from which
outcomes include satisfaction, knowledge, change inferencei may be drawn about how,clients view
in attitudes and behavior, including compliance the quality ofcare. In this respect, patient satisfac-
with recommendations. All these mays, in time, be tion has an interesting dual nature: it can be re-
related to success or failure in altering health garded not only as an outcome of care, but also as
status. Fundamentally, there is little difference a judgment by the client on the quality of care.
between studying -the effectS of two drugs on
'hypertension and a study of the effects of two ways
of managing the interpersonal. relatiorfship on Continuity and coordination, as attributes of
compliance with the drug regimen itself. Both are
clinical trials, though they may draw on different quality There is a general presumption that the
bodies of theory and concepts. But, despite the continuity and coordinationrof care are impostent
e considerations in the assesiment of quality, even
.similarity, the study of' the effect of drugs is
considered, squarely within the domain of clinical though it is not quite clear what these phenomena
practice, whereas no one is quite sure where the are, or how they fit into a formulation of what
quality is. De. Geyndt accords them a central place
study of the interpersonal relationship belongs. in the definition of quality by considering them to
Ideally, clinicians should be as interested in one as
1.8
 be aspects of the4rocess" of e as distinct from Several studies of the use of laboratory and .

its "content."" I have preferr re them as other diagnostic procedures an be cited as a good
an aspect of the organization of whar I h4ve called example of -this type of research. The findings of
'`,process. "" More portant than these formal Childs and Hunter. suggest that monetary return
distinctio,vs is the pr cise definition and measure- On lik.investment in an x-ray machine may be a
ment of these, attri utes, and the study of their factor in recommending radiological procedures.5°
contribution to th Outcomes of care. Empirical On the contrary, the °persistence of large differ-
Trudy should also include the degree to which ences in the use of laboratory and, other services in
existing differences in, Or purp ive manipulations stwitg. s where there is no direct finanCiakincentive
of, certain structural features mg' about differ- to use such services," or where the incentive is the
ences in continuity and coordina ion of care. I same for all physicians, 52.53 suggests that Other fae-
would also like to see/ a test of the hypothesis that Cors are equally or more important. Schroeder and
planned tjAnsfers of the responsibility o£ care his associates in a series of studies have looked into
from one physician to another might actually the correlatObf .lifferences in, laboratory use and
improve care by creating an opportunity to assess shown that these differences are net related to dif-
it anew, and by making the performance of physi- ferences in corripctence," Or to differences in out-
cians more visible among colleagues. comes, or "proddetivity";" and thavphysicians
Important Contributions to a more peecise defi- may respond to a "cost audit" by red cing labora-
nition of the concept of continuity may be found tory use."
in papers by Shorten" and by Bass and Wi'ndle."
However, the tendency to.4eraitiOnalize con-
tinuity to mean care by one fhyNician or. a single IdA ittfivition of styles and strategies
, I Have
source of care, except by referral, does not seem to already defined a strategy as a plan 'for action, and
me to capture the essence of the concept of cussed the iIportance of dealing with strategies
continuity. Almost all the studies I have seen have in t definition and assessment of quality. A
used this definition."42-47 Bass and indle tried to "style" lay be defined as a habitual preference for
assess the relatedness of past to present care." certain ,modes of decision making which would
Shorr and Nutting defined continuity as the com- manifest itself in components of strategies or
pletion of a needed sequence of are." Becker et strategies as a whole. For example, a physician may
al. describe a well-controlled tri of the effects Of--7 exhibit-a persistent' and pervasive preference for
having children see the same physician at each errors of commission over errors of-omission, pne
clinic visit, and give a gobd review of the relevant manifestation of which may be a large redundancy
literature." in gathering information-. He may give evidence of
more than usually routinized or stereotyped be-
.havior. These and other persistent yet undefined
Description of prevalent patterns and strategies propensities requiri precise conceptual formula-
of care tion Ind empirical study. c.
The notions of style and strategy have applica-i
In a Previous section we discussed th modeling
-tion beyond the solution of clinical probleis. I
have already suggested that they can also be used
and testing of strategies of care. In this section we to study the clie t-practitioner relationship. They
focus on a complement to the research described
earlier, namely on the identification of how physi-
also apply to the way in which a' titioner
manages an enti case load, hoping to a hieve the
cians behave in the real world, described in terms most efficient allocation of his time, attention; and
of the elements of care, as well as bundles and other resources among competing calls on them.
configurations of such elements. Here we seek to Styles and strategies can be inferred from physi-
describe what goes on, what factors. influence it, cian behavior in real-life situations or under more
and what the consequences seem to be. artificial test con itions. Information on the
rationale employed y the physician can be ob-.
tained more directly y having him explain, as he
Studies of the elements of clinical behas;.ior Much works, thereasons f. doing what he does. Fattu
moil information is needed about differencrs in has summarized some f the early work using this
the patterns of care among practitioners within a. method, known as rifle n /Attlee, as well as other
given setting and across settings. We need to, methods in studying pr lem solving." A more
understand what, factors are reorisible for such recent example of the use of reflexion parlee in
differences and what the costs and other corise- exploring clinical decision making has been re-
quences 'are: The factors invplved could inckide ported by Kleinmuntz.57 A study of the diagnostic
personality attributes, knowledge, training and process by Leaper et a). notes variations in the
socialization, position within an organization, re- degree to which the1clinical interview is either
sponse to role models within the organization; and "stereotyped" or "adapted" to the problems of
financial incentives. Fach patient, as inferred by an observer."
1
 Obviously, strategies That are identified through explicit, or occupy such a central place in the
design of the method. $ ,
empirical study become candidates for testing, as
described in the opening section on research pro- Kessner has tested his assump/ion Of
posals. I alsOolieve that strategies used by "good"
horpogeneity and found it, at bet, frail." LKons
pit ysicians are a 'more valid basis for the formula- and Payne have reviewed the literature and done
tion of expliCit process criteria than is the practice further ,testing of the degree of compliance with
of having physicians list all the things that should `normative standaids of management by individual
be donefor cases with a specified diagnosis. physicians, across diagnoses, in office practice,"
and in hospital-practice." s tended
to be low. There was, however, a sugg dot} that
, greater homogeneity might be found in the work"
Comparisons- of norms with practice The litera- of phySician subgroups'who havea more restricted
ture teems with observations that' physicians fall domain. This clustering was also found in a Study
short of the normative standards of care. In some of the office care of a set of preventive and illness )
cases it has, bee suggested that physicianS\do not situations in children. Homogeneity of perform-
follow, the sta Ards which they thlemselves, as a ance was reasonably high within each of these :
group, have rmulated." Some hale claimed that categories of conditions, and it wa,$ high r fcK
mashy errors in care are due to inattention by the pediatricians than for family physicians."
overworked physician." Others have shown that
part of the deficiency in performance is slue to
lack of knowledge, while another part is due to not
acting on what is known." In my opinion, the
explicit criteria lists which are often used to judge The epidemiology of quality
performance are, themselves, often faulty. One
important inadequacy is that they fail to take 'ac- Variations in the quality of-care are, n t simply a
count of the many contingencies, including multi- random phenomenon. They are highly terued,
ple diagnoses, that modify the strategies isf man- and responsive to causative factors ttfat we.need to
agement. Moreover, it may be inappropriate to identify_ and understand if the quality of care is to
-apply to office practice, especially to that of the besuccessfully safeguarded. The first step in this
generalist, criteria derived from strategies of man- exploration is to.'answer the classic questions of
agement that are suitable in 'academic settings demiological investigation: How much? elVho?
where highly specialized physicians are involved in ere? and When? The results of these observa-
the,polution of difficult diagnostic and therapeutic tions maff suggest answers to the most critical of all
problems. , questiobs: Why?,;The causal hypos eses that
Unfortunately, much of theabove is only conjec- emerge could, then, be tested through mor or-
ttire; d there is an urgent need for studies that
ture; ous observational studies and confirmed by actual
atte t to understand why physicians conduci experimentation. But, for now, even\the simplest
care in the way they do, before passing judgment of deScriptive studies would add a great deal to. the
that what they do is inapppriate. Nile we know.
On a larger scale, we can 74 say *lost nothing,.
about the quality of care for the nation as a whole,
Homogeneity and heteroge sty of performance
. or for reasonably large populations in thtr natu-
An interesting question wit many practical impli- ral 4abitats,otiher than what can be inferred from
cations' is whether physicians and other practition- ;crtele mortality, morbidity and utilization data.
ers perform equally well across a range of activities The one exception to this generalization that I
and functions, or whether they do well in some know about 'g the study of a segment of cars for
and not so well in others. This involves examining the residents of Hawaii by Payne and his as-
the htogeneity of performance in the practice of sociates." We are similarly in the dark about time
indivi al physicians, as well as the ability of an
trends: Is the quality of care improving, and how
institution to 4-educe variation in practice across rapidly? Thii question is difficult to answer be-
physicians.
cautv it requires the separation of two phe-
The relevance of this issue to quality assessment nomena: changes in the science and ,technology of
is most apparent when it comes to sampling. In medicine, and changes in the application of that
one 'method, the "tracer methodology" proposed science and technology. Both phenomena need to
by Kessner, there is an explicit.assumption that the be assessed.
performance of an entire system' can be mapped The epidemiology of quality can be viewed as
by judicious selection of a small number of condi- manifesting itself in two populations: (1 the pro-
tions that can stand for all the rest." In many yiders, and (2) the clients. 'Obviously, these are not
other studies, when a all number of diseases or two separate compartments. Variations in the qual-
conditions are seleCted fo ssessment, there is a ity of care received by clients are probably largely
similar assumption, even th ugh it may not be due to the kinds of providers who care for them. I
20
 mondeil to what extent the reverse could also he the characteristics of physicians that perform par
par-,
true. ,
ticularly well or badly could be a useful way of
. ,
$ generating hypotheses about the determinants of ,
performance. f N

Tim- epidemiology of quang' among provider' iAmong the recent studies that have attempted to
The studies that might belong in this category A mecasure the magnitude of the effect of each of
overlap.rith those I have already described in the / seieral variables on iperfnance are those of,
. ptevious section.on Description of Prevalent Pat- Rhee7' and of the Stanford University group
terns. If there'is a istinction at all, in the earlier responsible for the Institutionaltifferences Study.
section 1 .included tudies that dealt with the 72 Rhee used data from the study by Payne et al."
cletailed ontent of\ care and the rationale that
.
in' which the dependent variable was a perform-
.,, .esplaiteddifferences in that contentfiere, we are ance score based on compliance with explicit progb .-

... more interested in who provides 'care of good, bad ess criteria. Notable among his findings was the
or inclifferett quality, and under what circum- large effect of hospital,cliaracteristics compared to'
stances. Much of the literattire of quality assess- the effcct of specializaition and of organization into
mept deals with this question; but the restricted large,nultispecialty groups; and, evepnore 13.
scope of most studies, and limitations in their striking, is the magnitude of the unexplained
design and analytic methods, inakes generalization variance: In their study onpost-operative surgical
hazardous. It is clear, however,' that performance mortality and morbidity the Stanfol-d group., npt'
is related to attributes of practitioners, attributes only found unexplained variance of a similar
df the organiz*ional settings in which they work, order of magnitude, but also failed to confirm the '
Ind interactiOrA between the two. Among the
.
effect of factors usually thought to be conducive to
attributes of the providers are education, training, quality, for example hospital size and university
specialization and length of practice. The role of affiliation. Both of these studies can serve as
personality characteristics including motivation, models for future work. The Institutional Differ-
while suspected to be large, has nocreai4ed much ences Study is particularly notable for the methods
attention, except in studies of the academic per- it developed to control for risk factors that influ-
' formancle of medical studenas. Among the attrib- ence the outcome of s rgerr and td specify. and
utes of organiiatibnal settings, usually hospitals, 'measure organizations variables. Its findings
that have been found'to be influential, perhaps the could perhaps be better Aderstood it samples of
. most important has been affiliation with a medical. records in the hospitals involved were to be sub-
ca school. Other attributes, are involvement in resi- jected to an assessment of the process cif care. I
dency and internship training, auspices, sqffing, would accord such a study a high pri rity in this
financing, size, and organizational contrIlls on prospectus.
staff appointments -and activities. Unfortunately, - ' The correlations bitween organizati nal charac
theie is far fr%n unanimity on whether these teristics and perform&nce are, of course, extremgly
factors are influential and what their effects are. important for system design. It would in interest-
For example; we are still not certain whether the ing, therefore, to go.one step further and deter-
organization of physicians into prepaid groups /mine whether the-same physicians will, alter thei
behavior when placed in differeirit settings.
Results in better quality, and, if so, to what extent.
tThe 'numbe of studies that have attempted ac-
-,
Atually to qu ntify the influence of each factor
separately an in combination with others is par- Epidemiology of quality' among clients Unl the
tic larly small. Whenever this has been done, what emphasis on provider characteristics, there has
is ost impressive is the very small amount of been very little attention to client characteristics in
. v riance explained by the variables in the analysis, studies of the quality of care. Although the receipt
' suggesting that we still know very little about the of quality care by disadvantaged populations'is a
determinants of the quality of care. matter of great social concern, and there is much
". The literature relevant to this section is so large public debate about it, the information bear ng on
that even a partial review would be a herculean the question is indirect. It deals main with
task whiCh I shall not attempt. Only some .exam- differences in levels' and patterns of utilization, in
ples will be given. Among the earlier studies sources of care, and in morbidity and mortality
particularly outstaistling is the work of Peterson et
al." and of Morehead et al." Among the more
data. Without minimizing the relevance of such .
information, it would be useful to have more
recent work is the Hawaii study by Payne and his direct and definitive assessments of the quality of
associates," and the study of Medicaid benefici- care received by persons differentiated by age, sex, .
aries in New Mexico by BrEioke and Williams." Of educations, color, income, occupation and other
'particular inteiestin the latter is the special wen-. demographic and socioeconomic variables. Much
tion to "outlier" physicians, those whose perform- of the differences would probably be related to
an.fic was markedly deviant.70 Further studies of differences in sources of care. But a question that
.
21 ---.
 '

'needs to receive special attention is whether the be o c e. No program that provides care to
same institution, and the same practitioner, give' clients of widely varying background* can afford
different types oriiare .0to patients with similac, to igncir the possibility of discriminatory behavior
medtal conditions because of differences in the in the application 'of careNrrco*itravention of the
demographic and socioeconomic characteristics of most sacred traditions oLthe titling profestions. ".
the latter. Some adaptations of care to such
charactefistics are; of course, not onlylergitimate #i relationship of structure to process or
but, also, desirable and necessary. The' issue,is to
determine *icttether the adaptations are made to
maintain a ,.high level of quality or-whether quality
come.' .
. t
, The eader will recognize that the epidemioloii:
suffers.4
cal-stu es skeichefl out above often deal v#th.sob- '
The literature having' a bearing On uality of served relationships beta Teen structural afara
care for disadvantage populations-44,sbeen 're- teristics (attributes of practitioners and instil -
viewed by Broo d !Hams." In another publi- dons) and process, or outcome, or both. The more
14
cation they d scribe t eiown fiticlings in a study definitive verification of such relationseips will re,-
of Medicaid eficiaries." Lyons and Payne h Ye
nship Izetween age and Ilte quire controlled experimentation. The major pur-
described the re pose of such studies is to safeguard and improve
quality of care in 'their studies ill Hawaii."'" the quality of care. However, at the same time,
essner et al.- hasie described the ',relationship they can elucidate the operational meaning of cer-
between
1K syrem performance and various demo- tain concepts, for example "continuity." Tothe ex-
graphic at socioeconomic factors, as well as
tent that attributes of structure are 'found to be
of care, revealed 'by an application of the regularly related to performance, the more gen-
"tracer" method, in selected populations in the eral use 'of such attributes as measures of quality
Washington, D.C. area.63 Griner and Liptzin have will be mot firmiblistied.
described the effects of patient characteristics such
as age, insurance status and ward or private
accommodation on the use of laboratory tests in a Development Of basic tools for assessment is
teaching ho s al." .

The ident cation of time trend's is an important Many of the studies mentioned in previous sec-.
tool in epideviological analysis that has been tions, as well as many-still to bc_ described, cannot
seldom employed in studies of quality. Hence, two be done well unless certain' bas c tools. a7 avail-
studies that have information on this subject are able. Thus, the refinement of existing instruments,
particularly interesting. The first is a study of and the development and testing of new ones, is a
survival after cancer of' the cervix uteri treated a necessary part of research in quality assessment.
decad apart. The findings suggest that improve- At issue are the reliability, validity and cost of the
ments have been due mainly to a diffusion of basic instruments of assessment. A few of these,
knaaedge from rrilijOr centers to community hos-
. that I consider most important, *ill be selected for
Ifftals, and to a much lesser extenCto an improve- further attention.
ment in medical science." The second is a study of
maternal mortality in Michigan from 1950 to 1971
showing that, in spite of spectacular declines in
mtirtality, the percent of deathsNonsidered "pre- Specification and measurement of outcomes. I
7:':-ventable" by the States maternal mortality corn- have already indicated the ways in which the
,mittee fias increased markedly, from about 60' measurement of outcomes fits into quality assess-
percent to about 80-.percent." A retroactive reas- ment'. and monitoring. In this section, l sha
sessment of the Committee's file of cases, applying describe briefly some ways of measuring outco
current standards of preventability could be very One approach is to develop and use indicators
revealing when compared to the contemporaneous health and social well-being which permit a gen-
' assessments. eral oversight of a community of population. This
,,Most of the work on differences in the manage- approath is typified by the "sentinel events" pro-
ment of patients by the same piovider, whether an posed by Rutstin et al." A useful area of research
institution or an individtial practitioner, has been would be to specify appropriate indicators, de-
done in the field of psychiatric care. Perhaps the velop methods for data collection, actually imple-
best known of these studies is the work of Hol- ment data collection and interpretation, and de-'
lingshead and Redlich in New Haven," but there termine the usefulness of the information in
bringing about change. Such a system would, of
are many others. In a later work, Duff and Hol- course, have to be adapted to the special needs of
lingshead showed that such differences in care can
also be observed in a teaching hospital engaged. in its users: whether, a planning body, a program, or
providing genepai medical care." The under- an organization that provides care. Any of tifiese
standable reluctance to look into this matter must agencies may need to supplement information
2-2 1`.
 ii,
which it collc4s with 'information from otl-ar above will plibably be found to be 4.cking in sen-
sources, including census data and information ,sitiyity and specificity when used aI measures of
from.the.National Health Survey. Naturally, the the quality of ar . There is an opportunity to
indicistor conditions need not 1,4 only outcomes; a reihedy these ects by developing analogous
variety of process elensknircan, also be included. measures of the uratio .. quality of life of per-
Moreover, the ,system will be relevant not only to sons suffering from specific c ditions, fbr,exam-
determination of quality, but also ay need and ple a particular cancer. The mea ures of function
unmqt need, resource use, and so on. Many or- and dysfunction induct/et can th be tailored to
ginizations are,'Ipo doubt, already involved in data the condition, whivatten an gii n to including
gathering activities of this general kind. Perhaps manifestations that can be pr ted or remedied
the first step wouk be to review all that is being by proper care. Thetesting of such measures will
done, tg document Its current use, and to'assess its als provide an 'opportunity to study the course ok
potential usefulness._ It islimpbrt4nt to rixnember ess and identify-Mdditional outcomerrat cat
that a . system overladen' with data that are not serve as measures of quality.
.useful or are not us4d, can be as bad as one that
gish4s t little information. In any event, a pause for A fourth line that might-be pursued under the 15
rea ment could be most helpful.
general heading of outcome measures is the de-
velopment of condition- specific of out-
A second line to pursue is the development of come. Here, only key indicators are used singly or
"integrative" 'measures of health status that can in a profile; there is no attempt to integrate them,
represent" the outcome Of all the factors that together with losses from mortality, into a single
influence health. The distinctive features of such a measure, as envisaged in the preceding section.
measure are that; (1)4he impact 9f mortality and But, obviously, there is a relationship betiveen the
morbidity are combined; (2) morbidity is repre- two approaches, since the identification of indi-
sented by a gradation of mutually exclusive vidual indica rscmust be a step iff the construction
categories of dysfun`Ction; (3) dysfunction includes' of an integrativ measure.
social and psychological, as well as physical, disabil-
ity; and (4) the several dysfunctional states are The time e sed since the institution care is
weighted and summed into a single measure,-The an important classifying variable in o tcome
object is to arrive at a summary representation of studies. Accordingly, the indicators of ou ome
the quantity, and quality of life of a cohort of indi- can be contemporaneous with care, br can f ow
viduals over a period of time, often a complete life care, in which case they are proximate (short-term)
span. or remote (long-term). Short-term outcomes areof
special usefulness in monitoring becatir they can
Elsewhere, I have briefly reviewed the earlier be used to identify cases that require further
stages in the develypment of this approach study:. Thfris a method of venerable ancestry,
through the work of Sanders,'Chiang, going back to the _classic work of Codman," and
and Fanshell and Bush." Since then, this area of earlier. Its more recent manifestations in,the work
endeavor hasoexperienced an almost explosive of Williamson and his students have already been
growth which includes further work by Bush and noted.'2 A particularly useful model of the kind
his associates,"" the work of Torrance," and the of research and development needed for con-
work of Gilson and her associates on the Sickness structing short -term indicators of outcome, is the 0
Impact Profile." " Three collections of papers will work recently reported by Brook et al." What still
provide the reader with a concentrated and rea- remains as an essentially unsolved challenge is the
sonably current overview of the field." development of concurrent monitoring, using out-
Perhaps the central problem in the construction comes during the process of care. Of course, the
of integrative measures of health siatus is that of conduct of care from day to day is constantly
valuation. There is need for empirical studies of guided by such outcomes. The difficulty has been
the valuations placed by clients on different de- in adapting this everyday occurrence to a formal
grees and kinds of dysfunction. Expecially in- system, of monitorrng, other than direct supervi-
teresting would be differences in relative valuation sion by peers or superiors. Some of the recent ad-
by persons who are currently experiencing a par- vances i computer-aided management do; how-
ticular level of disability. The effect of length of ever, su est a possitle solution." What seems, to
time in any level of dysfunction should also be be necessary s the constant feeding of selected in-
examined, as suggested by Torrance." Another formation into a computerizedsystem which raises
line of development might be to try out .a totally an alarm when certain, prespecified, configura-
different method of valuation, comparison with a tions of events occur or fail to occur during sped-,
standard population, as proposed by Breslow and fled time intervals.
his associates." A final line of inquiry derives from adopting a
A third line of development is the construction definition of quality that includes phenomena such
of integrative measures of health status that are as satisfaction, attitudes, opinions, knowledge, ill-
condition - specific. The global indices described ness behavior, and the like. This opens up the
23
 whole a ea of methods in behavioral research allwed e traditional content of the medical rec-
ord to tate what is included or not included- in
which ca be assessed'and implemented by inves-
assessme is of quality. In this way, the record con-
tigators aving the necessary pleparation. A
model for such research that is closer to home may strains the definition of quality, allowing the tail to
be foundA the work of Hulka (and her associates, wag the dog. It seems to me that 1% is mote reasons
first, in h development.of a scale of client satis- able to begin-by defining quality Independeptlyof
faction and,othen, in using that instrument' to the record and, then, to design the record so that.
study its epidemiology.'' -95 alone or in combination withOther specified
sources, it can provide the information that cprfe2
sponds to the initial definition of quality. In such
an. enterprise there would be a great, temptation to
Improvement,' in the medical FKa6-1The medi- demand an im/Sossibld degree of completeness.
cal record is almost always fllie key document , One must be adamant In resisting it. Th more
which contains e information for assessment reasonable and. challenging objective would be to
of care. Judgn nts bf quality are heavily influ- define and implement the near-minimal set that
16 enced by the nature of the record. There is also would permit p oper management and assessment.
the possibility that the record is, itself, influenced I have alre y referred to the uses of com-
by quality assessment activities.33.37 Unfortunately, puterized records in computer-aided management,
in spite of its key role in patient care and its evalu- and to the affinities between the latter and concur-
ation, the record is often inadequate or poorly rent monitoring of care. Setting out to design such
adapted:so these purposes, especially in ambula- terns is an)excellent opportunity to rethink the
tory care. The follow' ig are some proposals for record and .adapt it more suitably to its several
()remedying this situatiob. . ^; t
uses. The releyant literature is immense, By way of
While the record is often recognized to, be in- examples, I have already referred to the work of
complete, the accuracy of the nformation that it McDonald" and of Schmidt et al." Other work in-
does contain is seldom questio ed. The early ,work cludes that of Wassertheil-Smoller et al.1" and .,
of Lembcke" is an excepti t this generaliza- Barnett et al.'" .
tion, as is the more recent w k of Wiener and
Nathanson." It would be usef to test, by seeking
. independent verification, th accuracy of the his-
tory, physical findings, results of diagnostic tests, The criteria and standards of quality No assess-
and so on. As a second step, it would be interesting ment.is possible without some.standard for com-
parison. In studies of quality, two or more provid-
to see what effect corrections of these errors would
make on an independent judgment of quality' ers may simply be compared. Another very ,preva-
based on a review of the record. lent approach is to judge performance by the
The completeness, and some aspects of the accu- extent to which it attains normative standards.
i The urgent need to develop realistic and valid
racy, of the record can be studied by arranging for
independent direct observation of they client- criteria and standards for condition-specific out-
practitioner encounter, or by recording it on comes is implied in my previous discussion of the
videotape. Use of the latter method has been re- measurement of outcomes, and can be seen as a
ported by Turner et al.'°°, Zuckerman et al.,101 and parallel activity. As Brook et al. have shown, the
SteWard and Buck.'" key steps are: (1) the identificatinRof relevant out-
Alternative ways of designing records should be comes; (2) ordering them by importance; r3..)-find-
developed and tested as to their usefulness in the ing reliable and valid means to get information
management of care as well as its assessment. Con- about the outcomes and to measure them; (4)
siderable work of this nature has been done in specifying when dui.ing care or following it each
connection with the problem-oriented record. outcome is to be measured, so that it is most dis-
criminating (sensitive and specific) as a measure of
Examples are the studies of Tufo et al.103 and
Simborg et al.10' Other work has been reported by performance; and (5) specifying the degree of
Grover and Greenberg.'°3 In work of this kind, progress toward each outcome that can be ex-
pected by good care, given certain attribUtes of the
,.., the objectives include not only completeness and
ccuracy of information, but also ease- in finding patient and his illness." In addition to using such
what is needed, and the ability to identify the prac- outcomes for retrospective review of care, there is
titioner's intent and reconstruct his rationale. need to develop methods for using 'them in con-
Another objective might be the inclusion of the current monitoring, as has already been discussed.
contributions to care of nurses, social workers and Criteria and standards for the assessment of the
proCess of care are very widely prevalent and are a
other professionals, so that assessments can be
more inclusive. In a subsequent section I shall deal basic tool in current assessment, monitoring and
with the feasibility of even including entries by the control activities. In spite of their central impor-
patient himself. tance as a measurement device, the criteria lists
.At a more fundamental level, we have generally have attracted little serious scientific analysis: I
24
 shall devote the, rest of this section to proposing .
ways of remedying this deficiency.
abseCrice of one critical element in care render ?
rende the
entire care disastrous, no' matter how many brow-
perhaps the first' step is to develop a taxonomy nie points the care can earn in other respects: Of
of criteria lists and similar formulations based 9Iii course, this could be handled by assigning near-
key attributes of their design and its underlyiig infinite weight to such elements, provided the con-
logic. Next would come an analysis of the possible figurations that render them critical can be de- /I
implications of these features for quality assess- fined in advance.
dent. The twork of Rosenberg" is an example of a.".
The problem of weighting could be investigated
an initial exploration in this direction. Much more through comparing the items on a criteria list with
work is needed, and soon. . the corresponding algorithm, and to both implicit'
.,,,. As a result of the above, or independently, work judgments of quality and `the outcomes of care.
should proceed on developing and testing alterna- World that has'a bearing on the question of weight--
tive cr*ria designs. One way to go is to develop ing includes that of Richardson," of Hoekelman
algorifc formats that define more precisely op- and Peters," of Lyons and Payne," of Hopkins
, timal or acceptable strategies for management, tak- et al.,"3 and of Novick et al"
ing account of frequently encountered contiogen- Another interesting lire of inquiry would be to 17
cies. The worktof Greenfield and his associates is subject a set of recoPd to as ssment Using differ-
an excellent example." Initially, these algorithms
*derive their validity from etpert opinion. Ulti-.
ent types of criteria. The co mplications of_ ac-
tually satisfying different type f criteria should
t
-8 lively, they should lie tested emeirically, as indi- also be determined and compar d to the expected
cated in an earlier section. and, where feasible, the actual outcomes'of care.
When a system of monitoring is designed, it The social process, including group interaction,
might be useful to consider the use of several sets that leads to the formulation of, and agreement.
of criteria in stepwise fashion. For example, a Sim- on, explicit normative criteria has been, to my
ple list could be used for screening, with a more knowledge, an almost totally neglected area of re-
elaborate algorithm to be used for more definitive search. It would be useful to know how leadership
s'udgments in cases that fail the screen, possibly is exercised, dissent handled and differences re-
upplemented by a sample of cases that pass. A solved. The effect of including health profession-
ombination of a concise algorithM with judgments als other than physicians, and of administrators or,
using "implicit" criteria may be tried out. The even, consumers should be studied. Similarly, the
work of Mnshlin provides an example of the lat- inclusion of physicians from a broder range of
ter.' Rubenstein et al. give an example of how speOialties, for example psychiatry, physiatry or
. 'criteria with "laundry list" and algorithmic com- public health and preventive medicine, might have
ponents can be combined into a "decision index. "" an interesting effect. The content of the criteria as
As mentioned in the recedi g section there is a well as the process of arriving at them might be
., mutual- interdependent betw en recording'and influenced. As a subset of such studies, it would be
assessment. Hence, o part of the effort to de- useful toilook into 'ways of expediting the ocess
velop and use alter tive criteria designs is the of peer concensus and improving the dec sion by
woirk needed to r esign the record so that the staff work that provides necessary infor ation,
criteria can be more efficiently applied. forms or worksheets, and otherwise structures of
The application of criteria lists to the assessment the situation. Brook et al. demonstrate the Useful-
of process results, initially, in a "profile" of indi- ness of such staff work and, incidentally, comment
vidual criteria that are met or not met. The deriva- on the impact of including a psychiatrist on the
tion of an arithmetic average weights each item panel dealing with the outcomes of breast
equally. Differential weights may be assigned to surgery.'3 We are indebted to these investigators
the several items and a weighted average obtained, for giving us this information. There are other
as in the work of Payne et al." These procedures workrs in the field who also have considerable
lead to several difficulties. First, a given score can experience in such matters, but who have not
be obtained
.
by different combinations of perforrh taken the time to describe it for publication,
ance and non - performance of the several criteria perhaps because they have not realized how im-
on the list. Are these different combinations portant it is. 'As a simple first step, I would suggest
equivalent, or are there combinatorial interactions that this Bind of information be tapped, even if it
that are missed, by simple summation? Second, we produces only descriptive. accounts .end informed
don't know what any given numerical score means. 2
guesses about what works and what does not
Is a score of 65 "good," 'fair" or "ppor" ?.Third,
we do not know for certain what the basis for the
weighting is, and -how valid the weights are. Fi- Monetary measures of costs and benefits The
ney, a related matter, the construction of an av- need to measure the monetary costs of inputs and
erage, weighted or unweighted, does not accord the monetary equivalent of ben fits arises fre-
with the intuitive view that in some instances the
J quently in assessing quality, as we have seen
25
 Lelopsnent and 'lest' of these proposals as mat-
already. Precise and valid cost and benefit meas-
urements are also required to assess the ters of high priorit .

of utilization control and quality monitoring

systems. In addition to devising rigorous and 1

standardized cost accounting procedures, there Sampling and "enrichment" techniques There
are some important conceptual problems that re- are two aspects of this subject that tend to overlap
quire attention. The problem of arriving at mone- and become confused. The first has to do with the
tary equivalents for nonmonetary costs and bene- kind of probability sampling designed to obtain an
fits has alreadyleen mentioned. In assessing the unbiased picture of a specified universe of
effectiveness of utilization control proedures it is phendmena. Obviously, this is a critical issue in
possible to overestimate savings, and be unaware many assessment efforts; and much work is
of shifts in the cost of care. Wyszewianski and I, needed to develop efficient means of stratification'
have indicated asome of the ways in which this may and sampling. Some studies mentioned earlier,
happen)" A *port by Ike Lnititute of Medicine bearing on the heterogeneity and homogeneity of
provides a, good surnmarr l':ictors.to be consid- performance and the factors that influence it,
18. ered are that. the days of care saved nA be less Would contribute to the knowledge needed to
costly than the average cost per day. that capital sample more efficiently.
costs are nut reduced proportionately to variable,
There is a ther kind of selecticy which is not
costs, that the hospital may function.inefficiently if
sampling in the/ statistical sense, but a method of
beds remain 'empty, that the physician may not be
screening. its Intent is to increase the yield for
as productive in caring for some patients outside monitoring: to hit pay dirt, so to speak. Ideally, a
the hospital, that expenditures for carei)given method Is wanted that would pick up every case
outside the hospital in place of hospital care will
rise, that these expenditures may not be covered that is managed suboptimally, while it excludes
every case that is managed at an acceptable level of
by insurance, and that nonadmission to the hb4pi- quality. In other words, a screen is wanted that is
tal or premature discharge may generate future 100 percent sensitive and 100 percent specific. In
costs.
the real world We, obviously, have to settle for
something considerably short of this.
In considering,further research in this area, let
Specification and sting of system-design ele- me begin by pointing out.that not enough' atten-
ments tion has been given to whether total coverage is
necessary if a monitoring system is either to give a
Several activities al ady described fit under this fair representation of performance or to hie effec-:
new category of rese h and development pro- tive in achieving improvement in performance.
jects. These include th design of alternative Samples could be no less effective in achieving
criteria formats and record s tems. Selected addi- both these objectives. Pilot studies to verify this
tional features of system desig will be described possibility would rank high in my list of priorities.
below. A fair amount of work has been done on de-
veloping selection or screening methods that are
intended either to increase the yield from monitor-
Specifying the appropriate cut-oir points in ing or, also, to bring about the most efficient sep-
aration of questionable from non-questionable
standards The determination of the appropriate cases. Examples are the work of Wolfe,'" Riedel 0
cut-off points or levels in the standards for al.,"s Rubin,60 and Glass et al."' Brauer briefly
monitoring is a critical design element because
both the yield and the cost of the monitoring ef- describes a variety of selection mechanism-1 and
gives a longer account of an "adaptation of the
fort are heavily.influenced by that. There may als,o method described by Riedel et al, to rev,iewing psy-
be other consequences, for example to the social
acceptability of.the system and dysfunctional adap- chiatric care.'" Certain procedu-res that begin the
tive reponses to it. An excellent example is the de- process of review after observation of poor out-
terfnination of the most appropriate "check comes, such as the methods used by Williamson"
and by the PEP, system,'° can also be regarded as
,points" for recertification of further hospital stay. concentration, enrichment or screening
To make this determination it is necessary to mechanisms. Despite all this, relatively little is
specify.the factors that. go into the analysis, to
identify what information is necessary, and to im- known about the effectiveness of such screening
plement the analysis, first, in model form and, techniques as measured by sensitivity and specif-
later, in practice. Wyszewianski and I have indi-
icity ratios. Mushlin does provtide a test of his
scheme, which depends on rev4wing the records
cated one possible way to proceed." Averill and
McMahon have offered a mathematically more of Those whose original symptoms have not im-
proved within a month of reporting for care. For
rigorous model."' I consider the further de-
26
 and specificity of this method are quite impresl proaches
sive.. By contrast, the method advocated by Riedel
et al., which is essentially the selection of statisti- Irwrecent years, a number Of promising new ap-
f ally deviant cases, d s not seem to perform very proaches have been developed and tested more or
well,: ipdging by the inary reports that I less rigorously. In most cases, a great deal remains
have seen." to be done. To select one or more of these ap-
One particular method of selection, that in- proathes and invest in, further. testing could be
., volved in the "tracer" methild developed by Kess- very rewarding, since it would, build on existing
ner et al.,12 ." has been described in an earlier work and benefit from''the advice andrcollabora-
section, tion (and, sometimes, the afterthdughts)___of their
originators. In fact, many of the proposals for re-
searth that I have included in this review are de-
Larger elements of design In this section, I want rived from a critical examination of what has been
to call attention to the design of the assessment published about these new approaches.
and monitoring, endeavor as a total system in At the .riskor some repftition, let me mention
which there is a mutually supportive functional re- some of the approaches that I think are most
lationship among parts. I have tried to develop promising.4'he order in which they appear is not
this idea to ap earlier work.'" Various expressions intended to signify either perceived impottance or
of it are daily seen,, in the design of a variety of personal preference.
.
. systems including that of the PSRO program. The
conceptual apparatus and the methods-for testing
such constructs will probably come from systems Outcomes as measures of quality. The major re
en sneering and analysis, and will embrace both ,cent exemplar of this approach, the Institutional
tec nical and social phenomena. Since systems Differences Study, demonstrates both its useful-
\virialysis is another of the many subjects about ness and its limitations.72 The fundamental as-
which I know next to nothing, all I can do is to sumption in this and similar, studies is that when
express the hope that it has something to contrib- statistical corrections are made for known risk fac-
ute. If it does,not, the work of designing and tors that influence outcome,a great deal, if not,all,
testing these larger systems s ill not stop, but the of the variation that remains is accounted for by
underlying principles hat govern design and ef- differences in the quality of care. I believe that a
fectiveness will have to be formulated as the direct test of this assumption should be attempted
,/ con1ruction goes on. by independent assessment of the process of care,
using a variety of methods. I suspect that the re-
sulting' correlation between process and outcome
would be low, suggesting .the need Tor better
Comparative studies of quality using different methods to standardize for risk factors, as well as h
approaches more fundamental approach to process' assess-
ment. In particular, the decision to operate or not
A great deal of insight into alternative methods was not subjected to assessment in.the Institutional
of assessment can betibtained when they are Differences Study, as the investigators take pains
applied to the same set of records, and the result- to point out. I suspect that many of its anomalous
ing estimates of quality are compared. An findings are traceable to this basic weakness.
excellent model is the early work of Brook,"4,par- The method of assessing proximate and inter-
ticularly when studied in its more dttailed mediate outcomes of care, so well described 'by
More recently, Brook et 1. have -pro- Brook et al.," is another promising approach that'
posed that separate sets of process and outcome requires and deserves much further development
criteria be developed simultaneously for a number along the lines indicated by its proponents. This
of conditions, with subsequent comparison of the includes extension to other diagnostic categories
ratings of quality accorded to the same care using or conditions; actual implementation of the
both sets." The primary purpose in such studies is criteria already developed to test feasibility, accu-
not to find new facts ,about' the link between racy and cost; and comparison with simultaneous
process and outcome, but, given existing knowl- process assessments.
edge, to determine.whether the proper formula-
tions of process and outcome criteria have been
made. However, when discrepancies between rat- Outcomes as cues and motivators for process as-
ing based on the two sets are found, they could sessment There is no sharp line of demarcation be-
lead to questions about the validity of what was tween the use of outcomes as measures of quality
thought to be known about the process-outcome in their own right and as screening devices to
relationship. 'select cases for process review. In fact, my attribu-
2?
 don of primarily one unctiop to some methods deaths in hospital, has been reported recently
frothiEngland."° The key feature.Of these studies
and another function to ther methods is likely to is that the problerp of attribution is handled by a
be. challenged by their o iginators and advocates. case-by-case analysis that leads, to a determination.
However, I do perceive t e approach developed by .of whether there were preventable or avoidable
Williamson" and used by Mushlin,9 as well as adverse circumstances and by assigning responsi-
some basic- elements i the PEP method of the bility for these. Broader application and testing of
Joint Commission on Accreditation of Hospitals,10 this method, using a wider range of conditions,
to fall more comf itably in the category of seems to be well worthwhile.
monitors of outco rather than assessments of
quality based o'n o tcomes. The method described
by Brook et al." appears to be intermediate be-
tween the puree forms of the two classificatory The occurrence of preventable progression of s
categories that have proposed. But none of this illness or disability This is another member of the
discussion on cl ssification need deter usnfrom not- family of outcome-oriented methods which cannot
ing the potenti I usefulness of these methods and be fully differentiated,from some of its compan-
20 investing effor in their further development. I am ions, except in emphasis. It has particular affinity
particularly mpreised by the simplicity of to the preceding category, Preventable Adverse
Mushlin's ap roach, and its great success in Events, except that, in this instance, the focus
separating sign ficantly deficient care from accept- the preventable progression of illness from an
able care. Whe her this will remain true when ad- earlier stage, which is presumably more amenable
ditional conditi ns are tested remains to be seen. to treatment, to its later, more advanced and
As to the PEP pproach, it offers many opportuni- recalcitrant manifestations. This approach owes its
ties for further development,, for example the de- more recent saliency to the work of Gonnella and
termination of esponsibility for complications that his associates who refer to it as the "Staging
occur in -the h spital and the. inclusion of sonic Concept."129.1" In one way, the \staging of disease
outcomes that appear after discharge from the creates more homogeneous categories, so that the
hospital, as pr posed many years ago by Cod- attainment of outcomes can be compared with
man." To retu n to the distinction which forms greater confidence that differences are attributa-
the basis of my Classification, if these methods to ble to care. However, Gonnella -et al. also argue
indeed use outcOmes as monitors and screening that the stage at which a disease comes under care,
devices, rather than as more complete representa- either initially or at some later date, telts us
tions of quality: a dear recognition of this distinc- something about earlier ,access to care 'and the
tion could lead to simplification of the measures of quality of that care, if care is Provided. As in all
outcome and to their assessment in terns of their outcome studies, the problems of interpretation
screening effidency. I believe that this Would be a are many, but this approach does simplify popula-
very useful and important t4velopment. tion monitoring to some extent.because the neces-
sary data can be obtained from within the patient
populationjor example, a hospital may be unable
to precisely identify the population it serves or to
The occcurence of preventable adverse events obtain useful.. population data, 'but it can ,charic-
Rutstein et al. have recently reminded us,of the terize all, or,a sample of, its admissions as to stage
potential usefulness of this, time-honored method of illness and preventability of progression to such
In
of monitoring the health care of a population." a stage. Further empirical testing is warranted.
this Method, attention focuses on, outcomes and
other events that are preventable, at least to a sig-
nificant degree, when good care is available and is
used. Obviously, this is little different, in principle, Indicator conditions:. "trajectories" and "trac-
ers" There is a large number of studies in which
from the approaches described in the preceding one or more conditions are selected and the career
section, except that-we are now speaking of popu-
lations rather than of patients. A historically signif- of patients with these conditions is followed as the
icant method of quality assessment and control patients proceed through, the System. This could
be call the "trajectory" approach, since the em-
that probably helongs under this heading is repre- phasis is on what happens at each successive step
sented by the activities of the maternal mortality in a progression that is, too often, a tragic Odyssey
and perinatal mortality committees originating in of accumulated failures. Examples are provided by
the landmark studies of the New York Academy of Brook and Stevenson,'" Starfield and Scheff,132
Medicine.""" Since then, much information of Novick et al. "4 and Shorr and Nutting." The
this kind has been accumulated over long periods "tracer" method developed by Kessner et al. can
of time in may states and localitiet. The useful- be seen as a highly systematized selection of such
ness of this historical material is demonstrated, by a indicator conditions, each with its distinctive
recent analysis of Maternal mortality data in trajectory. The systematizing or organizing device
Michigan.11 A current application using infant
28
 is a prior conceptual mapink of a field and the integrative, The approaches that Use'''the "trajec-
. purposive selection of conditions "lo represent all
tory" or "tracer" methods also have an integrative
the major elements in that fie41.9313 Another property since they often include outcome as well
characteristic of the "tracer" method is its em- as process measures, and follow the course of care
phasis on combining population and patient &lull thrinigh successive stages, levels and sites, so that
to achieve an epidemiologic investigation of the they reflect the cinnulation yf deficiencies at each
..

problems of medical care. However, this of these junctures.

epidemiological perspective could be incorporated' There k need to develop methods that incorpo-
in the "trajectory" approach, in which case each rate process and outcome elements that are expe-
trajectory becomes a tracer (more accurately, the chilly selected to represent the contributions of the
path of a tracer). Such semantic. games aside, there several professions that are involved- in patient
is much opportunity for further work in this area. I care, because these elements are particularly .re-
particularly like the notion of a planned selection of sponlive to the contributions of each of these
% tracers, with, a view to systematically, sounding the several professions.
corpus of medical care-in its totality and to include Another way of taking a more complete or
both patients and non-patients. However, since this integrated look t performance is to use as a unit
requires, a massive effort, more modest and cir-
.
of analysis thee tire case load of a practitioner or
cumscribed applications should be tested first. institutional prov er, so that the assessment in-
eludes not only th adequacy of care, but also the
optimal allocation o esources'among cases.
Second surgical opinion programs An idea that
has captured a wide audience is the possibility of
controlling unnecessary surgery through either
making available or requiring second surgical Applications to special area*
ofiinions.133 This is a particularly interesting ap-
proach since it is truly preventive. There are many On the one hand, there is need to develop
opportunities for research and development here, integrative-veasures of the quality of care. On the
including tests of the reliability and validity of other hand, `there is need to adapt assessment
multiple surgical opinicins, acceptance by clients methods to special populations, to categories of
under voluntary and mandatory systems, accept- care, to particular professions, and so on. This is
especially true when an agency or organization
ance by physicians, effect on relationships among that provides patient care includes very large ele-
physicians and surgeons in a community, effect on ments of very diverse programs, including general
initiation of recoinmendations for surgery, effect ambulatory and inpatient care, long term care,
on the client-physician relationship, and so on. To physical rehabilitation, care for mental illness and
answer the question of validity:long term -alcoholism, etc. Within each of these categories,
follow-up under, controlled conditions would be care could also be differentiated according to pro-
necessary. Stich studies are now in progress.'" fession, for example: pharmaceutical services, den-
tal services, nursing caresocial work, dietetics,
physical therapy, radiology, pathology, anos-
Computer-aided management I hive discussed thesiology, and so on. It is impdssible for any one
earlier the affinities between computer-aided reviewer to encompass the literature in all these
management and several facets of quality assess- Tecialized areas. I shall not even try to offer 'a
ment, including .the design of records, the identifi- partial description. However, the reader seeping
cation of critical events, and concurrentimonitor- an introduction might find useful some selected
.
ing. I. see this as an area of much fruitful further items that have come to my attention.
development I Freeborn and Greenlick have attempted to sys-
tematize approaches to the assessment of ambulat-
ory care.'35 Christoffel and Lowenthal have re-
viewed the newer methods that are available.'35 A
Integrative measures of quality publication of the American Society of Internal
Medicine provides a recent anthology on ambulat-
I have already said that most studies of quality, ory care assessment."' The American Nurses As-
focus on a relatively small segment of care pro- sociation has performed a similar, function for
vided by one professional, and, that there is need methods of assessing nursing quality.'" A collec-
for more "integrative," measures that include the tion' of papers in the cfuality of pharmaceutical
., contributions of several professionals during corn- services can serve as an introduction to that very '
plete episodes of care and sequences of such interesting and 'important area of application."9
episoldes. I have also pointed out that outcome re will be found in. a publication by Knapp and
measures, especially those that are more inclusive t .140 A collection of papers on the assessment
in extent and duration, are by their very nature, of al care, though not so recent, could also be .
 )
helpful,"1 Much information about the assuiment
other times, the expectations of the well-informed
client can be quite valid, for, example when more
of mental health services' in genlorwl, together with
complete prenatal care is demanded or too ready
detailecttinformation ab6ist one approach,. will be
use of some x-ray examinations or 'of antibiotics is
found in, a book by Riedereval.'" resisted. In any event, it would be interesting to
i.
study the Influence of such expectations on pro-
fessional performance. of
Consumer perspectivis and the consumer's I
role I have already commented on the dual nature
client satisfaction: aran "outcome" of care, and as
There are many who are distressed by the near a judgment 'on care. In the latter instance, it could
of
total- domination that.. the professionals have exer- be seen as the patient's estimate of the quality
cised over quality assessment, fro its deepest .care. It is remarkable how infrequently the-pa-
roots to its most slender branches. nd yet, it has tients' estimate of the quality of care he has
been very difficult Ito involve the consumer of in received in a defined instance has been compared
quality assessment in a meaningful way. Aware to that of a professional estimate of the, quality of
think
the same care. The only example that I can
22 this problein, The Institute of Medicine has iden-
tified consumer participation as one of five of was reported by Ehrlich et al. many years
"priority areas for quality assUrance," and has ago."' This is an obvious area for further study.
devoted considerable space to it in-its "research
agenda." " I' In this" section. I shall deal with some
, ways in which consumers can
participate in quality Clients as sources of information For some data
assessment, hoping that I will stimulate further used in assessment the patient is the only .au- pa-
thought, research and.adevelopment. , thoritative source: for example, data on
tient's knowledge, opinions, satisfaction aandnd the
like. For other data, the client is an alternative or
verifying source: for example; concerning services
Representing client. values in the definition of received and assessments of function or disability.
quality At' the root of quality assessment is a view There is .much rooms for research on ways
of
of what attributes of care constitute quality, which obtaining reliable and valid information at low
imprints itself. on evernhing that follows. ft is, cost, and its incorporation into quality, assessment
therefore, important, as 'I have argued earker, that and monitoring activities.
the clients' values, preferences and expectfions be It is perhaps ironical that "the patient's' record,"
included in the definition. of quality from the very in any medical setting, is not only totally
barred to
entries. by,
first. ; the patient, but also.cdntains no 'direct
The choice of outcomes as a measure of quality the patient concerning how he feels, what he
is, in itself, a means for making the notion of qual- knows, or how he perceives.his care. It ould be a
ity easier for the patient to understand, and closer fascinating experiment to see whethe it would be
to his concerns. But not all outcomes are equally possible to have the patient tries into the
comprehensible or relevant. Outcomei that are de- records, eitherin narrative form tor'as checkmarks
fined in terms of physical or, social function are on a list of questions. It has been suggested that
much more meaningful than clinical, physiological patients might be unwilling to express- negative
iraluations that are
or chemical measurements. The feelings for .fear of reprisals. This is subject to
placed on alternative outcomes, when there is a testing and, if verified, entries might be made, on a
choice, could differ not only between clients and separate document that only later becomes part
of
professional's, as groups, but also among clients as the record. Even the answers to the iluestion,
individuals. All these speculations are subject to "What was the one best'thing that happened to you
empirical study, as is the degree to which the con- today?"would be most revealing.
duct of care takes account of client values and
preferences, collectively or as individuals.
, As we said earlier, clients also have preferences Participation in monitoring Participation in
with respect to the process of care. Generally, monitoring can perhaps occur indirectly. and in-
these focus on, the management of the interper-- formally through the manner in which the well
sonal relationship. One cannot emphasize too informed patient responds to physician initiatives
- strongly that in this one aspect of care the patient and participates in the client-practitioner interac-
is fully as expert as the professional, if not more tion. This is a matter that can be studied.
so. .In fact, a good case could be made for having
in this A specific example of this more general category
the client set the criteria of good care is the degree to which patients participate in sec-
retard.- Clients also have views about the-technical and surgical opinion programs and how they react
component of the process of care. Sometimes, to non-copfirrhation of the initial reCominenda=
'these are of debatable validity, as when an injec- tion. There is alt(eidy evidence that participation
don is demanded or, even, a hysteiectomy: At ,
 In voluntary programs is not high and that patients factors that influence that contribution, is a legiti-
do not always act in accsrd .with a second recom- mate and important. area of research in its 'own
mendation, whether at is confirmatory of the first right.
or not.18$0" 4ttempts could be made to alter client'
behavior and to test their success.
Some have argued that the paienti medical
record should be accessible to him at all times. Quality assessment and monitoring as a social
Shenkin a d Warne have proposed that this ,prooess
should be required by law, and have speculated on
''the possible advantages and disadvantages that Perhaps the most difficult problem's in estab-
might ensue.'" Stevens et al. described experience lishing quality and utilization control mechanisms,
with- an actual trier phiast.ggests that at least and in operating them effectively, are social rather
some patients can ',participate usefully in monitor- than technical. It is of the utmost importance to
; ing their own care, while others either catinot or redress the imbalance in past and current research
are, unwilling to try.,'" More work is cal for. by paying at least equal attention to quality
It, is not clear, to What extent patients can monitoring and control as a social process. In my 23.
participate in a more structured and formal man- earlier work, I have described briefly the kinds of
ner in. monitori the care they themselves receive, factors that provide a context of quality assessment
but there are many opportunities to find out. For and monitoring.'" I have also speculated on the
example, Bouchard et, al.'" and Burger"' have nature of the factors that irlfluente the effective-
reported on experience with involving the patient ness of monitor\ng and review.'" More recently,
in auditing' the problem-oriented record-by having Freidson has presented an incisive analysis of the
the physician's evaluation sent to the patient for social processin the implementation of PSROs "as
comment. part of a larger class of issues connected with the
social psychology of work and its control."'5° Ad-
7:
Membership in audit and utilization review
ditional speculations, from a more operational
viewpoint, can be found in a series of comments pn
committees As far as I know, the inclusion of a paper by Morehead,'5' in a paper by
clienis on audit and utilization review committees Bellin.'5° Jacobs et al. describe and discuA factors
is virtually unexplored territory. Goldblatt et al. in the implementation of the PEP system of as-
describe professional reailance to inclusion of sessment developed under the auspices of the
Joint Commission on Accreditation of Hospitals.",
consumers on such committees in a project on
mental' health evaluation.'" But, they also note Goldblatt 'et al. give an excellent account of ex-
very briefly that a "consumer opinion subcommit- perience with utilization review committees .in a
tee" that :tindeperidently investigated clients' collaborative project in mental' health evalua-
opinions about care and analyzed complaints a tion'4° Using these 'sources, and others like them,
referring institutions and practitioners" influences it would not be difficult to formulate a series of
selection of topici as well as cases for review. In hypotheses as a starting point for research. But
this way, "consumer dissatisfaction could be there remains a great and urgent need for an
tutVed into criteria .for good care."'" In the same approach to the study of the implementation of .
sway, consumer participation on grievance corn- quality monitoring that rests on a sound, and
mittees'and similar bodies could be linked to the systematic conceptual base. The development of
activities 'of quality assessment. There is urgent this conceptual foundation is, itself, an area of
need for careful trials of a variety of mechanisms scholarly research. ,However, it is likely that no
for involving consumers directly and indirectly in unifying framework will emerge, and that this
quality assessment and assurance. social phenomenon, like all.others, can be seen and
understood using a variety of theoretical con-
structs that are provided by economies, political
science, sociology, organizational theory, an-
Patient contributions to care. The importance of thropology, history, law, and so on. My pessimiim
the client-practitioner relationship,to quality notwithstanding, the student of medical care or-
sessment has been a recurring theme in this paper: ganization will 'continue to hope that in some way,
The care bf the patient is a joint enterprise which relevant contributions from the theoretical trea-
includes contributions by the patient as well as the sure of all these disciplines can be brought to-
professional. Quality assessments of care may give gether coherently, so he can understand-fully the
the professional too much credit for success or too problems that he faces.
much blame for :failure unless careful attention is At the broadest level, there is need to under-
giVen to the role of the patient. Btu, there is much stand what might be called the politics of quality
; more involved than simply deciding who is respon- monitoring and control. This could include 'the
sible for what. The assessment of the patient's role of government, of the, organized professions,
contribution to his _own .care, and. study of the of the organized constituencies, including the
31
 derstand how they work With each other, and how
health insurance sector. The interrelationships be-
and the or- each rikated structurally and functionally to their
tween regulatory government agencies peer groups, to the internal utilization
ganized profession is a particularly critical area for
respect
and audit ,cominittees, the cliriical department
study. All these forces need to he understood as chiefs, the chief of staff, and the 4ospital adminis-
they interact at national, state and local levels.
trator. There are usually others who are important
At the level of thethospital, :mil similar idstitut as leaidets:Ahough informally; and still others
who,,
tions, it is important. to
understand the- power
like the pathologist, can have disproportionate in7
relationships Within the hospital as well as those fluence because they control infotmation critical to
.between the hospital and, its environment. The the assessment and, its conclusionii.
organization of physicians in the immediate emit- And, finally, what of the individual physicians
,ioninent of the holipital and .within it is a particu- themselves, who are the object of this seeMingly
larly critical element. Comparative studies of unrelenting scrutiny? The manner in which they
hospitals that have different goals
and/or are
respond individually, through/ their formal and in-
differently organized could be quite revealing. format] groupings within the Aospital, and through
Such studies might include comparisons of their many-tiered and interilocking- professional
24 government-owned to community hospitals, and of ogranizations, including 'the PSRO, determines
hospitals that emphasize teaching and research to whether the program achieves its objectives, or
those that confine themselves to patient care. whckher it is so watered down and subverted that it
The local PSRO functions as' a key link between becomes a ponderous,and costly apparatus, 'mak.-
the hospital and the organized profession in, its ing i brave shOw, but achieving little.
environment, and between the latter and gov-
ernmental ,regulatory and financing agencies, as
well as other third party 'payers. The dynamics of
PSRO operations should Jiecome, therefore, an Effalveness and the factors that Influence it
object of intense scholarly scrutiny. Unfortunately,
it is quite likely that this will be countered by an Finally there is no way of escaping, the most
equally intense determination to avoid being Momentous of all questions: whether vality and
studied. Nevertheless, careful study of the PSRO utilization review .activities are effective, and suffi-
as a social organization is absolutely necessary,
and ciently .so, to justify their immense social and
openess to such study should be a condition for monetary cost. It is remarkable that even now,
formal recognition. when we have made a political decision to con-
In this progression from larger to stnaller,social struct the awesome machinery or the PSRO, the
units the next critical-level is that of the audit and answer is that we do nor know. ome'"fyears. ago I
reviewed, what was' then known about the effec-
utilization review committees, where much of work
tiveness and costs of 'quality and utilization review
of assessment and monitoring is done.-The struc-
mechanisms.123 Very recen , a committee of the,/
ture and roles of these committees should be
Institute of Medicine reasse sed the situation not
studied in. differently. organized hospitals. with
differing linkages to the local PSRO. The only by reviewing the lite tune but also by ob-
dynamics and effectiveness of the committees are taining information directly from operating pro;
likely to be influenCed by the structure of the grams.'" My conclusion was that hi some instance
there was success, whereas in others there 14as
tommktees themselves, including variations in
Who is represented on the thembership.roster, and failure, and, that we did know for fertain what
linked, structurally and accounted for either, though one could speculate
how the committee' is
functionally, to centers of power and inflionce in at length on the matter. I understand the cdo0u-
the hospital as a whole. ran earlier section, sitaus of the Institute of Medicine to be not much
mentioned the, importafce of studying the cf ferent. It is still, not clear what hospitaLmedical'
dynamics of developing thte criteria and standard's au its atcomplish, if anything. by contrast, the
occa-
that ,operationally define quality. ut ztion control programs of hosprtals
sionally report savings, but these tend to ver-
At the most disaggregate level of analysis one estimated because of improper account
,
as-
finds, as always, individuals who hold key rolesi sumptions. Ambulatory care claims review, where
and who carry on their shoulders, as it were, the studied, has been cost effective, but this is mainly
burden of the, most massive of social enterprises.
Depending on whether these persons are seen as or entirely 'due to the administrative component,
as distinct from professional peer eyiew. All these
prime movers or mere pawns, their behavior
.

"savings," when they do occur, are 'to the fiscal

either determines. or reflects the manner in which intermediaries. The social costs' can be shifted. As
the total enterprise ultimately functions. In this to the effect on the health of people, almost
instance, I see as if at the focal point of some giint
nothing can be said.
lens turned to' the sun, the lonely figures of the There are many reasons for this rather dismal
"nurse coordinator" and medical adviser." If I had state of affairs. The most fundamental are the
and try to un-
a choice, I would go first to these,
 O.%
.
absence of sound conceptualization of the problem 1 included the client in the title of this s ction
to be addressed, the deficiencies in the basic tools because of a conviction that any change in t eliv-
of measurement, and the absence of soundly de- havior of the practitioner is likely to have
signed and properly controlled studies. No ac- terpart in the client. Almost no considerao n has
cumul don of case reports from operating agen- been given to client behavjor in studying the of
cies, : I) matter how lengthy, can resolve these feels of quality and utilization control mechanisms,
problems. Case studies do, however,'have the vir- and I am not sure that there will he any, unless
tue of generating hypotheses for further testing. considerable costs are shifted to the client.
In subsequent sections. I will suggest some areas Nevertheless it is interesting to speculate whether
for study, without,trying to cover the crone range patients may be morekely tq patronize physicians
of possible research. It will become obvious to the who are less scrupulous in accepting the strictures
reader that there is much nverlaiihetween these on hospital admissions and length of stay. Might
proposals anti others that were made earlier 'under The patient develop symptoms when informed that
different headings. In particular. there is a close the approved hospital stay is about to expire?
tie between the specific considerations that come Could the local hospital lose community support
up in this section .and the issues thaqt were very for seeming to repeatedly refuse admission or to 25
broadly sketched in the preceding section on Qual- throw patients out?
ity Asserent and Monitoring as a Social Process.

Changes in physician and client behavior It is Effect of technical design Characteristics I have'
important to document the changes that occur in discussed in an earlier section some technical de-
the behavior of physicians and other practitioners sign elements that might influence the perform-
as a result of instituting quality and utilization re- ance of the quality monitoring system. The rela-
view mechanisms. Rather simple before-after tionships to effectiveness could be inferred from
studies are useful: but, where possible, contem- observations of existing variants or tested by inten-
poraneous controls should be provided. In these tional manipulation under Controlled conditions.
studies, it is necessary to keep in mind, and look Examples include studies of the effects, of differ-
for certain "dysfunctional" behaviors, such as a ent criteria formats, of varying the hospital stay
tendency to lengthen stay up to the "checkpoint," recertification checkpoints,. and of testing the
or to discharge prematurely, as a response to a cer- cost-to-yield ratio of alternative sampling and en-
tification program: the. likelihood that physicians richment schemes..
will "manage by criteria," resulting in many re-
' dundant procedures: and the possibility that eva-
sive actions will be take, for example by using a Intra-institutional "social design" characteristics
different diagnostic nomenclature or by moving This category subsumes a very large area much of
patients to other settings. It is safe to say that ev- which I cannot see clearly. .1 will mention only a'
erything that human ingenuity earti devise will be few of the more obvious kinds of studies.
used to tame a regulatory mechanism, and the re- It seems to me that the legitimacy of the quality
searcher must be prepared to anticipate and study monitoring effort in the hospital, and the degree
such behavior. Some attention to possible adverse of commitment to it by its key figures would be a
or unintended effects can be seen in an interesting very important factor in the effectiveness' of this
paper by Brian on the impact of a utilization con- effort. The- written and verbal declarations of
trol program -in California.'" Brian concludes that board members,Administrators_and physician
the 'program -was effective 'without evidence that leaders would be one source of direct information.
needed care was denied or costs shifted to others. Perhaps more valid would be the inferences drawn
In direct contrast, two reasonably well controlled from how the assessment and monitoring effcirt is
studies of a hospital-stay. recertification program structured and how it functions. Much can be
.in Pennsylvania showed no effect on hospital use learned from an examination Of who chairs the
even though the state,Medicaid agency was much audit and utilization review committees, and who
impressed by the reduced rate of unjustified the members are. It is also important to know"what
stays.'".'," A reasonable, though unverified, ex- decision-making power the committees have, how
planation 0.1-this discrepancy is that the program their recommendations reach the executive eche-
did not alter hospital use, but did improve tab lon in the hospital, and the degree of influence the
documentation' needed to obtain, payment far committees have on..tlie key centers of power in
care.=s' If so, it is only proper to ask whether this' the hospital.
new documentation is a better representation of The. legitimacy of the criteria and standard
the truth, or only a more credible distortion. To incorporated in the monitoring system could be a
the extent that it is the latter, we may, a% a society, particularly important factor. One significant 'vari-
be turning o t ?he mo,st expensive fiction every able is whether the criteria are externally imposed,
written! developed internally or a mix of the two. If the
33
 , . Pour performance is not always primarily at-
'crite?ia aee, at least to some extent, internally /'

tributable to individual failure. Quite frequently

developed, the degVe of participation in criteria
1 formulation may influence the adherence to. the there are organisational problems that 'interfere
criteria by the physicians as a whole, or may dif- with good work by practitioners. In that case, and
ferentiate between participants and nonparticip- often when individuals ate at [atilt. it is necessary
ants with regard to adherence. If the criteria are to make changes in the organizatiiln. This requires
externally sponsored. the identity of the sponsor the full support of those who hold executive
could be important: for example, whether it is an power. Ilence, the manner in which the quality
insurance carrier, a government agency, the local 1 monitoring system is linked to the executive, and '

PS110, the Joint Commission of Accreditation of the mit ore of its influence at that level, become
Hospital's, the American Medical Association or critical elements in studying the factors that mod-
one of the specialty societies. Ina large system ifyperfoance.
Education, either alone or in conjunction with
such as the VA. sponsorship by the central office
versus the local institution. could be a differentiat-
sanctions. features prominently in attempts to
bring about clinge in the behavior of practition-
26
01
4
ing variable! .

e structure of' sanctions and incentives as it

ICS on individual practitioners cannot fail to
ers. The variables that are likely to influence -elec-
tiveness include, first, the relative emphasis placed
on education as compared to sanctions. Thep,
have an important effect. One le.atue is whether there are different educational strategies that
the monitoring system confines itself to studies of could be more or less effective. On distinction
patterns of care or whether it goes beyond that to that has been the.subject of much sp Illation is
identify individuals whose practice is called into that between an educational program b ed on
question. 'I'he methods used to communicate and topics of general interest and on that is' ed by
interpret findings are determined to a large extent 'audit results. The latter approa can' be highly
by' the initial decision to identify or snit identify individualized by tailoring continuing education to
individuals; but, irrespective of that, there are the deficiencies in an'individual's performance..It
many options. Nelvsletters., general staff meetings, is also claimed that participation in audit and re-
( and meetings in smaller departmental or sub- view activities is itself educatilmal and helps*moti-
departmental units could have different impacts. vate change in behavior. All these speculations, as
It is also itriportat who is responsible for com - well as other hypotheses about the differential ef-
municating the information:. the lialtire of his fectiveness of alternative' educational strategies,
involvement; and whether that person meets with are open to testing.
the entire. Staff, a small group, or it single indi- Strategies of client education are also a relevant
r-
--' vitlual. variable, since patient cooperation may be an im-
More important still is the manner in which portant factor in achieving the objectives of quality
findings about performance become instrumental and utilization review.
in influencing the careers of individual practition-
ers. fit may be that the major consequence. of
nonadherence to standards is 'approval or disap-
of payment by an insurance carrier or Supra-institutional "social design" characteris-
prov- 1
tics The sanctions and incentives inherent in ,a
g ernment program. While important, occasional' quality monitoring and control system may act on
brushes with a third party payer may not be as
effective in influenscing behavior as would be the
the institution as a whole, in addition to 'their
certain prospect that the physician's performance effect on individual practitioners. The interests of
the institution are likely td have great impact on
record will be considered in determining his prac-
tice privileges, promotions, access to prestigious how committed it is, as a collectivity. to quality and
appointments, and other organizational .rewards. utilization control. For example; it is generally
The use of . rewards and their possible impact believed that when beds are plentiful, neither the
shoulcUreceive serious attention. Would it be pos. institution nor individual physicians are motivated
sible, for -example, to grant priority in admiSsion to to keep patients out. Self-interest works in pre-
cisely the opposite direction_ to .the objectives of
the patients of physicians Oho have ,a consistent utilization control. The same; is true ifr budget
record*;4, very few unnecessary admissions or
allocations to an institution depend ,on occupancy
stays? While thiy'absence of a reWard'is, itself, a
punishment it could well be that thereis a signi& levels,
.
and if savings from more prudent manage-
cant difference between a system that focuses on ment cannot be retained by the institution. In
punishing wrongdoers and anotlier that stresses some situations, it may be possible to vary such
factors under reasonably controlled conditions and
' .0rewards to (hose who have an excellent record of
performance.' Finally, any incentive system will be to study the consequences.
inoperative if knowledge about it is not shared and
The nature and extent of participation in
monitoring and control by an external agency is
if its. certain and impartial implementation is in probably a critical factor in effectivenef. Such an
question.
.3 4
 external agency might 'be an insurance carrier, a explanations of what these eveuts Mean. These
goveunnent agency, or a PSRO. In a system suai are. in a way, fragments of theory and snippets o
air that of the VA, the central Office probably.Oeurs hypotheses; and in these bits and scraps the aler
a paaallel relationship to the local hospital. 'The investigator tint)/ find the roue outlines of som
relationship ,Between the external ageiicy and he new insight.
intra..institutional quality monitoring apparakus It seems reasonable to begin the investigation by
can be structured in a variety, of vfays, and the examining' rply contrasting situations of brit- '
consequences examined. For example. under the liant succe 4 a ect failure. The factors that
SRO program, the hospital may be delegated, contribut. V5neastatould be more oharply.
partially delegated or nondelegated with respect to diffOnt se' large contrasts: But, fur-
quality and utilization control activities. The ther .e ugh the examination of in-
PSRO, or the analogous external agency, could terniediat cl`hirvegis Follow. A
-Obtain repons from the institution under his must lead to More.rigorobs testing of specific
supervision, duplicate certain review activities, in- hypotheses. '
dependently-obtain data additional to those hvail-
able to each institution, analyze and interpret data, \ .
engage in consultatory and educational activitifs, Cost.e(feetivtiness and e st-henefit studies So far,
and so on. It may even be feasible to have hospitals in 'this section, I e dealt with documenting
take turns reviewing each other. The effects 61 ). changes in beh and in identifying the factors
each of these activities, and of the various ways in( that Seem to enhance or deter such changes. But,
which each can be implemented, are legitiniate ultimately, a determinalion has to be made
end important objects of research. whether any given monitoring mechanism is wor-
thwhile, and which among alternihtive mechanisms
is to be preferred. This requires the comparisan at
a Social- organisational characteristics and medical' lept of costs and effects, and, preferably, of costs

, technology Based oh more general work in

ganizational behavior, Scott et al. have
hypothesized that a given organizational form may
or- and benefits. For example,. a committee of the
Institute of Medicine has estimated. flat an exten-
sion of PSRO activities to cover all inpatient and
be more or less effective in influencing perform- ambulatory care would require a yearly expendi-
ance depending on the technological characteris- ture of 11/4 billidn dollars."4 What do we get in
tics of the work being done." Somewhat along the return? .
same line, though independent of the work of The returns to a program of,monitoring quality
Scott et al., Wyszewianski has hypothesized that and utilization are partly nionetary savings due to
the effectiveness of a specific quality control pro- reduction in "unnecessary"- care. These savings
cedure is influenced, among other things, by the have to be set .aginst the expenditures that might
degree to which it' fits, speCified technological result from`care that is added in order to improve
characteristics Of the task whose performance it is quality, but also due to redundant care masquerad-
designed to influence.'" Further research along ing as "quality." I have already discussed briefly
these lines may provide insights into the effective- the difficulties in measuring these savings and ex-
ness of discrete quality control practices, while penditures, and emphasized the need tticlearly
making a contribution to organizational theory. identify their social incidence. When thatYs done,
the cost of establishing and operating the monitor-
ing apparatus must be carefully determined so
Case studies of successfil and unsuccessful pro- can be set against the monetary balance of its con-
grams. There is lively debate about the usefulness- sequences. But,Ino m4tter how careftl and accu-
of case studies as a method for research. The rate,the balance sheet of savings and expenditures
contention that ease studies aread( aluable method is; a definitive judgment cannot be made unless it
for generating hypotheses is co tered by the is possible to measure the impact on health. The
argument that it is impossible to understand any- megsurement healih and the assigning of a
thing about real life situations unless they are moetary value to it has also been discussed in an
'viewed and interpreted in preformed ways. Case earlier section. Seeing how difficult a task this is,
studies cannot gengate knowledge out of ignor- we need not wonder that the effectiveness of so
ance. However, giien some general conceptual much that is done in medical care and its Organiza-
framework that suggests to the investigator where tion remains less than completely evaluated.
to look and what kinds of things to look for, case
studies, I believe; can nudge the mind into new
formulations that draw on what was formerly
latent or only dimly perceived in, the investigator's
thought. Besides, the 'participants in the events
that are under study have their own views and
..35
 e..

ti
e

a 1
28 The task of eviewing the extent of current
ignorance and o indicating ways of remedying it
calls for an approach that the reader may find
overly critical of what has been accomplished: and
of what can yet be done_ , in the world of actiqn. As
I said at the beginning, it would be foolish to argue
that all efforts to monitor quality must cease while
we seek certainty about a near-perfect solution. On
the contrary, we must continpe to act based on
what we now believe to be reasonable and feasible.
But, we also need to find out whether we have
been corljece in acting as we have, and to learn how
to dO better in the future. I hope that this paper
has made a small contribution to that continuing
quest.

36
 a

1. Brook, R.H. and Allyson-Davies, A., Quality 12. Williamson, J.W., "Evaluating Quality of Pa- 20
Assurance and Cost Control in Ambulatory Care. tient Care: A Strategy Relating Outcome and
,Santa Monica: The Rand Corporation, July Process Assessment." The Journal of the Ameri-
-1977. can Medical Association 218:564.569, October
2. Tompkins, R.K.; Burner, D.C. and Cable, 25, 1971.
WI.. An Analysis of Cost-Effectivenins of 13. Brook, R.H.; Davies-Avery, .; Greenfield,
Phary,pgitis Management and Acute S.; Harris, L.J.; Lejah, T.; Solomon, N.E. and
Rheumatic Fever Prevention." Annals of Ware, J.E. Jr., "Assessing the Quality of
Internal Medicine 86:481-492, April 1977. Medical Care Using Outcome Measures: An
3. Forsyth, R.A., "Selective Utilization of Clini- Overview of the Method." Medical Care Sup-
cal Diagnosis in Treatment of Pharyngitis." plement to Vol. 15, No 9, September 1977.
. The Journal of Family Practice 2:173-177, June 165 pp
1975.
14. Fanshel, S. and Bush, J.W., "A Health Status
4. Lusted, L.B., Introduction to Medical Diagnosis. Index and kto Application to Health Services
Springfield, Illinois: Charles C. Thomas, Outcome." Operation.; Research 18:1021-1060,
1968. November-December, 1970.
5. McNeil, B.J.; Keeler, E. and Adelstein, S.J., $15. Hirsch, E.O., "Utilization Review as' a Means
"Primer on Certain Elements of Medical De- of Continuing Educajion." Medical Care
cision Making." ,New England. Journal of 12:358-$62, April 1974.
Medicine 293; 211-215, July 31, 1975.
16. McNeil, -B.J.; Varady, P.D.; Burrows, B.A.
6. Raiffa, H., Decision Analysis: Introductory Led and Adelstein, S.J., -"Cost-Effectiveness Cal-
tures on Choices Under Uncertainty. Reading, .culations in the Diagnosis and Treatment of
Mass.: Addison-Wesley Publishing Comparty, Hypertensive Renovascular Disease." New
1968. England Journal of Medicine 293:216-221, July
7. Klarmah, Hi., "Application of Cost-Benefit 31, 1975. s

Analysis to the Health Serviies and tht Spe- 17. McNeil, B.J.; Hesse!, S.J.; Branch, W.T.;
cial Case of Technological Innovation. in-
ternational Journal of Health Services 4:32 Bjork, L. and Adelstein, S.J., "Measures of
352, Spring 1974. . Clinical Efficacy. III. The Value of the Lung
Scan in the Evaluation of Young Patierits
8. Bunker, J,P,, Barnes,.A.A. and Mosteller,- F., pith .Pleuritic ess Pain." Journal of Nuclear
Editors, Costs,, Risks and Benefits of Surgery. Medicine 17 :169-160, March 1976.
New York: Oxford, University Press, 1977.
18. Neutra, R., "Indications for the Surgical
9. Mushlin, A.1.; Appel, F.A. and Barr. D.M. Treatment of Suspected' Acute Appendicitis:
"Quality AssuTance in Primary Care: A A Cost-Effectiveness Approach." In Bunker,
Strategy Based' on Outcome Assessment." J.P.; :Barnes, B.A. and Mosteller, F., Editors,
Journal of Community Health, in press. Casts, Risks and Benefits of Surgery. New York:
10. Jacobs, C.M. Christoffel, T.H. and laixon, Oxford University Press, 1977:
Measurit' ig the Quality of Patient Care: The 19. ;Schwartz, 'W.A., Corry, G.A., Kassirer, J.P.
Rationale for Outcome Audit. Canibridge, /and Essig, A., "Decision Analysis and Clinical
Mass.": Ballinger Publishing Company, 1976. Judgment." The American Journal of Medicine
11.. L.L., "Quality Control and the Medi- 55:459-472 (October 1973).
cal Record." Archives of Internal Medicine 20. McNeil, B.J. and Adelstein, SJ., "The .Value
127:101-105, January 1971. 37 of Case Finding in Hypertensive Renovascu-
 lar Disease." New England journal of Medicine 32. Bloom, S.W. an I Wilson, R.N., "Patient.
293121.2113 (July 31, 1975).
Practitioner Rel tioisships." 1u Freeman,
11.F.: Levine, S. awl Reeder,. 1..G., Havtdbooli
21. Abt, C.C., -The Issue of sal Costs in of Medical Sociology, Englewood Cliffs, NJ.:
CostBenefit Analysis of Surgery." In Prentice-Hall list.. 91kimti Edition, 1972,
Bunker, J.P.; Barnes, B.A. and Mosteller, E..
Editors, Cost). Risks and Benefits of Surgery. 33. Lebow, 1.1... "Consumer Assessments of the
New York: Oxfurrcl University Press, 1977. Quality of Medical Care." Medical Caro
12:328-337. April 1974.
22. Ginsberg, A.S.. Dicii4on Analysis in Clinical
'Patient Management with an Application to the 34. Inui, T.S.; Yourtee. E.L. and Williamson,
Pleural-Effusion Syndrome. Santa Monica, "Improved Outcomes in Hypertension
Calif Tire Rand Corporation,- July 1971,
.
After Physician Tutorials: A Controlled
Trial." Annals of Internal Medicine 84;846-851,
23. Meyers, A.; Brand, D.A.; DoVe. H.G. and June 1976.
Dolan, T.F. Jr.. A Technique for Analyzing
Clinical Data to Provide Patient Management 35. Sackett, D.L.; Gibson, LS.; Taylor, D.W.;
Guidelines: A Study of Meningitis in Children. Haynes, R.B.; Hackett, B.C.; Roberts, R.S.
New Haven, Conn.: Center for the Study of and Johnson, A.L., "Randomised Clinical
Health Services, Yale University, January Trial of Strategies for Improving Medication
1975. ,
Compliance in Primary Hypertension." The
Lancet 2:1205-1207, May 31, 1975.
24. Sturdevant, R.A.L. and Stern, D., "Accuracy
of Physicians' Predictions of Cholecystog- 36. Smith, D.B. and Metzner, C.A.. "Differential,
raphy Results." Medical Care 15:488-493, Perceptions of Health Care Quality in Pre-
June 1977. paid Group Practice." Medical Caro 8 :264-
275, July-August 1970.
25. Piachaud, D. and Weddell, TM,. "The Eco-
nomics of Treating Varicdnie Veins." Interna- 37. Marrame S.D., "Patients' Evaluations of
tional Journal of Epidemiology. 1:287-294, Au- Nursing Performance." Nursing Research
tumn, 1972. 22:153-157, March-April 1973.
26. Neuhauser, D. and Lewicki, A.M., "What Do 38. DeGeyndt, W., "Five Approaches to Assess-
We Gain From the Sixth Stool Guaiac?" New ing the. Quality of Care." Hospital Administra-
England Journal of Medicine 293:226-228, July tion 15:21-42, Witer 1970.
31, 1975.
39. Donabedian. A., "Some' Basic Issues in
27. Hardwick, D.E.; Vertinsky, P.; Barth, R.T.; Evaluating the Quality of Health Care." Is-
MitChell, V.F.; Bernstein, M. and Vertinsky. sues in Evaluation Research. Kansas City, Mo.:
I:, "Clinical Styles and Motivation: A Study American Nurses' Association, 1976.
of Laboratory Test Use." Medical Care
13:397-408, May 1975. 40. Shona. 9.M., "Continuity in Medical Caste:
Conceptualization and MeasUrement. Medical
28. Martin, S.P.; Donaldson, London, Care 14:377-391, May 1975.
C.D.; Peterson, O.L. and Colt .,"Inputs
into Coronary Care During S rs, A Cost 41. Bass, R.D. and Windle, C., "Continuity of
Effectiveness Study." A Internal Care: An Approach to Measurement." Ameri-
- Medicine 81:289-293 eptem , 1974. c Can Journal of Psychiatry 129:110-115, August
1972.
29. Jackson, R.A. and mith, M. ., "Relations
Between Price and Quality in Conicnunity 42. Pugh, T.F. and McMahon, B., "Measurement
Pharmacy." Medical Care 12:32-39, Jiptuary of Discontinuity in. Psychiatric inpatient
1974. Care." Public Health Reports 82:533-538, June
1967. -r
30. Schroeder, S.A.; Schliftman. A. and Piemme,
T.E., "Variation Among Physicians in Use of 43. Mindlin, iR.L. and Densen, P.M., "Medical
Laboratory Tests: Relation to Quality of Care of Vrban Infants: Continuity of Care."
Care." Medical Care 12:709-713, August American Journal of Public ,Health 59;1294 -
1974. 1301, August 1969.
31. Rubenstein, L.; Mates, S. and Sidle, 44. Breslau, N. and Reeb, R.G., "Continuity of
"Quality-of-Care Assessment by Process and Care in a University-Based. Practice." journal
Outcome Scoring: Use of Weighted Al- of Medical Education '50:965-969, October
gOrithmic Assessment Criteria for Evaluation 1975.
of Emergency Room Care of Women with
Symptoms of Urinary Tract Infection." An- 45. Breslau, N. kid Haug, M.R., "Service Deliv-
ery Structure and Continuity of Care: A Cate
nals of Internal Medicine 86:617-625, May 3s
Study of Pediatric Practice in Process of
1977.
r
 Reorganisation." Aerosol te Health and Social 58, Leaper, D.J.; P.W.; Stattiland. j.R.:
Behavior 17:339-352, December 1976.
"formats. J. and DeDomhal, E. .. "Clinical
40. Start leld, B.H.; Simborg, D.W..; Horn, Si), Diagnostic Process: An Analysis." BritiaA
and Younee, S.A., "Continuity and Coordi- Medical fining/ 3:569.574, Seinember, 15,
nation in Primary Care: Their Achievement 1973,
and Utility." Medical Cary 14;625.636, July MI. Hare. ILL. and Barnoon, S., Medical Care
1978. Appraisal and {lustily &SIMMS in the Office
47. Bice. T.W. and Boxerman, Sit., "A Quan- Practice of Internal Medicine. San Francisco:
titative sure of Continuity of Care." American Society. of Internal Medicine, July
Medical Ca 5:347-349, April 1977. 1973.
48. Short G.I. and. Nulling, LA. "A 60. Me Donald. C ..priv,x01.Based Computer
Population-Based AssessmentLAd the Clui- Reminders, The Quality of Care and the
tinuity of,' Ambulatory Care." Medical Gae NonPerfectability of Man." New England
15:455-464, June 1977. Journal of Medicine 95:1351-1355, December
49. Becker, M.H.; Drachman, R.11. and Kirscht, 9, 1976. 31
J.P., "A Field Experiment To Evaluate Vari- 61. BroWn. C.R. and Uhl, S.M.. "Mandatory
ous Outcomes of Continuity of Physician Continuing Education: Sense or Nonsense?"
Care." American Journal of Public Health. Journal of the American Medical Association
64:1062-1070, November 1974, 213;1660-1668, September 7, 1970.
50. Childs, .A.W. and Hunter. E.I)., "Non - 62. Kessner, D.M.; Kalk, C.E. and Singer. J..
Medical Factors Influencing Use of Diagnos- "Assessing Health QualityThe Case for
tic X-Rayby Physicians." Medical Care Tracers." New England journal of Medicine
10:323-335, July-August, 1972. 288:189-194, January 25, 1973.
51. Freeborn, D.K.; Baer, D.; Greenbelt. M.R. 63. Kessner, D.M.; Snow. C.K. and Singer. J..
and Bailey, J.W., "Determinants of Medical Assessment of Medical Care For Children.
Care Utilization: Physicians' Use of Labora- Washington Institute of Medicine.'
tory services." American Journal of Public 1974.
Health 62:846-853, June 1972. 64; Lyons. T.F. and Payne. B.C., "Interdiagnosis
52. Lyle,*C.B. Jr.; Citron. D.S.; Sugg, W.C. and Relationships of Physician Performance
Williams, 0.D., "Cost of Medical Care 'in a Measures." Medical. Care 12:369-374, Aril
Practice of Internal Medicine: A Study in a 197.4.
Group of Seven Internists." Annals of internal. 65. Lyons, Ty. and Payne, B.C., "Interdiagltosis
Medicine 81:1-6, July 1964. Relationships of Physician Performance
55. Lyle, C.B. Jr.; Applegate, %N.B.; Citron. D.S. Measures in Hospitals." Medical Care
and Williams,e0.D.. "Practice Habits in a 15:475-481, June 1977.
Group of Eight ,Internists." Annals of Internal 66. Osborne, C.E., "Interdiagnosis Relationships
Medicine 84:394-601. May 1976. Physician Recording in Ambulatory Child
54. Daniels, M. and Schroeder, S.A., "Variation Health Care." Medical Care 15:465-474, June
,Among Physicians in Use of Laboratory 1977.
Tests.' I. Iteration to Clinical Productivity
r)
67. Payne, B.C.; Lyons, .T.F.; Dwarshius, L.;
and Ouicotnof Care." Medical Care 15:482- Kolton, M. and Mortis, W., The Quality of
.437;June 1977. ,
Medical Care: Evaluation and Improvement.
55. Schroeder, S.; Kenders. K.; per, and
.
Chicago: Hospital Research and Educational
Piemme, T.E., "Use of Laborat gists and Trust, 1976,
Pharmaceuticals: Variations. Among Physi- 68. Peterson, 0.L.; Andrews, L.P.; Spain, R.S.
cians and Effect of Cost Audit on Subsequent and Greenburg, B.G.. "An Analytical Study
Use." Journal of the American Medical Associa- of North Carolina General Practice, 1953 -
tion 225 :969 -973, August 20; 1973.
1954." Journal of Medical Education Volume
56; Fattu, N.C., "Experimental Studies of Prob- 31, No. 12, December 1956, Part 2.
lem. Solving." Journal of Medical Education 69. Morehead, M.A.; Donaldson, R. et al., A .
39:212 -225, February 1964. Study of the Quality of Hospital Cart Secured by
57. Kleinniuntz, B., The Processing of 'Clinical A Sample of.' Teamster Fancily' Members in New
Inforthation by Man and Machine." In York City.New York: Columbia University
Kfuntz, B.K., Edito , Formal Representation of School of Public Health and Administrative
Hessen Judgment N w York: John Wiley and Medicine, 1964:
Sons, Inc., 3968. 3g0. Brook, R.H. and Williams, K., "Evaluation of
 the Ncw Mes144) Pert Review Ss aunt, 1071 to Cambildge, Mass Hai vaid Univelittv
Ness. 1973
1971" Afedual Carr Vol 14, No 12. Dr
(ember 1976, Supplement 122 pp MY Paoli k, 1.. Rusk. I.W. and Chew M.M.,
I owai41 an Opriattnal DrIllittion of
71. Rher, S., "Fa41111% Deteimining the Qua Ins
of Physidan Pei lot MAIII r in Patient Cate I /m/
Health rind Social Hehavior
14,6 23, Maids 1973.
Medical l:nre 14:733-75. keptetillsel If176,
8 Parini., D 1.. Hush. J.W. and (lien, M.M.,
72, Scott, Wit.; Foliest, W 11 II and Blown. "Methods lot Mrassiiing levels ill Wrll-
[VW.. "Ilospiial Slim lute mid Postoperative'
hnttig a Health Stains Index." Nerjlth
Mortality and kfcllstilits In Shot tell. S 'N1
and BilIW II, SI F4111111%, Organisational Re
,
Serial's Ile,raih 8:228.245, Fall 1973.
search or ilospo441-s.. Clm_ago... Blue tacos, AA ttf4- 4.144114..e, 1.61114k, WM. and Sackett,
%IX iiiii1111, 1976 I) L. "A' lttilits Maximization Model fur
Lvalualion of Health Cale Programs." Health
73. lirrnk, R.11 'dill! VS'illiams. k N "Qualits id,

Nero, el liesraidi 7: 118-133, Summer 1972.

Health Care Inc the Disadvaritageil"10arnal
32 ol Comnuay llealth 13'.!-156, %%'itirei 1975
1 Gilsnit, B.S.; J.S.. Brixiier M: ct al,
74. 1..vons, FF. and Pasiir. WC,. I hr Q41,1111% ill " I lit Sit klicv Impact 1111411c: Developinepi
4)1 Onteortie Measure of Ilealth Care."
Plivsiuji10%. Health/
Comparison Against "" 11"Cum-it.'
OptiniAl " lotA American journal of Puhlor Health 65:1304-
13111, December 1975.
Firatmeni 4)1 hit Fldri Is and Votingri-
Adidis in Commiiiiiis Hospitals." /owner/ 01 87, pollard. WE.; Bobbin. R.A.; Bergner, M.;
the Amernan Meduril .1%w, swum 227:92.5,928, Matins. add Gilson, B.S., "The Sickness
February 25, 1974. Impact Prnfile:Reliabilily of a Ilealiti Siatus
M canto c." Medico/ Care 14:146.155, February
75. Lyons, T.F and P.isite, It.(.. "Qt1.11ii of
Physician' (II lit c-Cate Pei hirman«.: DO lei 1976.

4..
cut leo Fidel Is I can lot l'ourig,-, ;Willis?" MM. Brig, RI.., Editor. Health Status indexes.
journal 4,/ the .4rner:«in .1leeli«il .f .or sauna, Chit-agb: Hospital Reseirch and Educational
229:16'21-1622, Spimbei 16. 197.1, 1 nisi. 1973.
76. Griner. P.F. and Liptzin. B "('sr 111 lite . 99. Elitism), J., Editor. "Socionicilical Health In-
1,41)rivtmv in .1 I vat hiug Hospital: Impina- dicators." International Journal o/ Health Serv-
tinits Inr Kit-let:arc, 1%4614.464m, and Hos- ices 6:337.538, 197(1.
pital Crisis." .4eora/s 01 internal Medic:t. Several Authors, "Health Stains Indexes-
75: 157-I63Augnsi 1971, Work in Progress." Health Services Research
77. Graham, J.B. and l'alourek, F. "Where 11:332-528, Winter, 1976.
Should Cancer of die Cervix Be 1 leated?- Breslow. I.., "A Quantitative Approach to the
.91
Amencars Journal 01 (1b.tetru% and Gynecology World Wahl) Organization Delitiitirm of
87:405-409, 094)be: 1963. Health: Physical, Menial and Social Well-
78. Schaffner, %N..: Fcrlei spiel, C.F.: hilton, beitig.'; international Journal of Epidemiology
M.L; Gilbert, D.G.and st,%(.11,;(m. ILA.. 1:347-355, Winter 1972.
"Mdtertial Morialiis in !slit higair An 92 Coditian, F..A.., A Study in Hospital Efficiency
Epiderniolrigit Arials %is, 144,0- I 97 1." ,4mers- As Demonstrated In the Case Report of the First
ran Journal o/ Publu Health 67:M21-829. Sep- Five Years of a Private Hospital. Boston:
tember 1977. Thomas Todd Co., Circa 1916.
79. Hollinkshead, A.B. and Redlich, F.C., Social 93. Hulka, B.S.; Zyzinski, S.J.; Cassel, J.C. and
Class acrd Mental lllnr.o. A l:orrirnuatty turly. "Fhompsyn,- Si., "Scale for the Measurement
New York: John Wiles & Sims, Inc.. 1958. of Attitudes toward Physicians and Primary
80. Duff. R.S. and Hollingshead, A.B., Sickness Medical Care.- Medical Care 8:4449-435.
and Society. New York: Harper and Ross. September-October 1970.
-"--968. 94. Hulka, B.S.; Zyzanski, S.J.; Cassel, J.C. and
Thompson, "Satisfaction with Medical
81, utsteina D.D.; Berenberg, W.; Chalmers, .

1.C.'; :add, C. 3rd; Fishman, A.P. and Care in a Low Income Population. Journal of
errin. E.B., "MeAtierints. the Quality of 'Chronic Disease 24:661-673. November 1971.
edical Care: A Clinical Mettod." Nf IV Eng- 95. Zyzanski,- S.J.; Hulka, B.S. and Cassel, IC.:
nd Journal of Medicine 294:582-588, March "Scale for the Measurement of 'Satisfaction'
1, 1976. with Medical Care: Modifications in Content,
Format and ScoPing." Medical Care 12:611- .
82. nabedian, A., Aspects of Medical Air Ad-
0 620. July 1974.
, stimistration: Specifying Requirentes for Health
 96. .Sehmidt, E.C.; Schall, D.W. and Morrison, Medical Care 13:966-975, November 1975.
C.C., "Computerized Problem-Oriented 109. Greenfield, S.; Lewis, C.E.; Kaplan, S.H. and
Medical -Record For Ambulatory Practice."
Medical Care 12:310427, April 1974. Davidson, M.B., 'Teti' Review by Criteria
Mapping: Criteria for Diabetes Mellitus, The
7. Grimni, R.H.; Shimoni; K; Harlan, W.R. and Use of Decision-Making in. Chart Audit."
'Estes, E.A. Jr., "Evaluation of Patient-Care Annals of Internal Methane 83:761-770, De7,.
Protocol Use by Various Providers." New cemller 1975.
England Journal of Medicine 292:507-511,
March 6, 1975. 110. Richardson; F.McD., "Methodological De-
velopment of a System of Medical Audit."
*98. Lembcke, P.A.., "M ical Auditing by Scien- Medical Care 10:451-462, November-
tific Methods." Jou 1 of the American Medical Deceniber 1972.
Association 162:64 -655, October13, 1956.
1 ft.:tIoekelman, R.A. and Peters, .E.N., "A
99._Wiener, S. and Nathanson, M., "Physical Supervision Index to-Meatire-Stand="
Examination: Frequently Observed Errors." ards of Child ,Care." Health Services Reports
Journal of the American Medical Asiociaton 87:537-544, June-July 1972:
236:852-855," August 16, 1976.
112. Lyons, T.F. and Payne, B.C.., "The Use of
100. TUrnet, E.V.; Helper, M.M. and Kriska, Item Importance Weights in Assessing Physi-
S.D., "PrediCtors of Clinical Performance." :clan 'Performance with Predetermined
Journal of Methcal Educatio'n 47:959-965, De- Criteria Indices.:Medical Care 13:432-439,
cember 1972. May 1975.
a-
101. Zuckerman, A.E.; Starfield, B.; Hochreiter, .113. Hopkins, C.E.; Hetherington, R.W. and Par-
C. and Kovasznay, B., "Validating the Con- sons, E.M., "Quality of Medical Care: A
tent of Pediatric Outpatient Medical Records Factor Analysis Approach Using Medical
bi Means of Tape-ReCording -Doctor-Patient Records." Health Services Research 10:199-208,
Enconniers." Pediatrics 56:407-411, Sep- Summer 1975..
tember 1975.
114. Novick, N.R.; Dickinson, .1(; Asnes, R.; Lan,
102. Steward, M.A. -and Buck, S.W., "Physicians' M. S-P. and Lowenstein, R., "Assessment of
Knowledge of and Response to Patients' Ambulatory Care: Application of Tracer
Koblems." Medical Care 15:578, July 1977. Methodology." Medical Care 14:1.12, January
103. Tufo, H.M..; Eddy, .M.; Van Buren, H.C.; 1976.
Bouchard,-.R.W.; Twitchell,.K. and Bedard, 113. Donabedian, A. and Wyszewianski, L., "The
L., "Audit in. a Practice Group." Pages 29-41 Numerology of utilization Control Revisited:
In Walker, H.K.; Hurst, J.W. and Woody, When to Recertify." inquiry 14:96-102,
M.F., Editors, Applying the Problem-Oriented March 1977.
System. NewYork: Medcom Press, 1973.
116. Institute of Medicine, Assessing Quality in.
104. SimbOrg; D.W.; Stanfield, B.H.; Horn, S.D. Health Care: An Evaluation. Washington,
and YOurtee, S.A., "Information Factors Af- D.C.: National Academy. of Scienes,
fecting Problem Follow-Up in Ambulatory November 1976;
Care." Medical. Care 14:848-856, OctOber
1976. 117. Averill, R.F. and McMahon, L.F., "A Cost
Benefit Analysis of Continued Stay Certifi-
103. Grover, M. and Greenberg, T.SQuality of cation." Medical Care 15:158-173, February
Care Given to First Time Birth Control Pa- 1977.
tients at a Free Clinic." American Journal of 118. Wolfe, H., "A Computeriied Screening De-
Public Health ,66:9867987, October 1976.
vice for Selecting Cases for Utilization Re-
106. Wassertheil7Smollei, g.; Bell, B. and Blaufox, view." Medical Care 5 :44 -51; January
M.D., "Computer-Aided Management Of February 1967.'
Hypertensive Patients." Medical Care 119. Riedel, D.C.; Fett r, R.B.; Mills, R.E. and
13:1044-1054, DeceMber, 1975. Pallett, PJ., Basic tilization ReView Program
107.. Barnett, G.0,; Winickoff, R.; Dorsey, J.L. (BURP). New Hay , Cohn.; Health Services
and Morgan, M.; "The. Role of Feedback in 'research Progra Yale Unidltrity, 1973.
Quality Assurance-An Application of 120. Rubin, L., Comprehensive Quality Assurance
Computer-Based Ambulatory. Medical In- System: The Kaiser-Permanente Approach.
formation 'System." In Assessing Physician Alexandria, Va.: American Group Practice
Performance in Ambulatory Care. 'San Francisco: Association, 1975.
AmericanSociety of internal Medicine, 1976.
121. Glass, R.I.; Mulvhill, M.N.; Smith, H. Jr.;
108. ROstnberg, E.W., 4!What,Kind of Criteria?" Peto, R.; Bucheister, D. and Stoll, BJ., "The
41
 4 Scqre: An index for Predicting A Patients' of Screening by Consultants on 'Recom-
Non-Medical Hbspital Days." American Jo Ur- mended Elective Surgical Procedures." New
ofPublic Health 67:751-755, August 1977. England Journal of Medicine 291:1331-1335,
December 19, 1974.
122. Brauer, L.D., "Selection of Cases for Indi-
vidual' Review." In Riedel, D.C.; Tischler, 134. McCarthy, E.G., "Mandatory and 'Voluntary
G.L. and Myers J.K., Patient Care Evaluation Second Opinion Programs in the Greater
New York Area with National Implications."
in Mental Health Programs. Cambridge, Mass.:
Ballinger PublislAing Company, 1974. The Dorothy Eisenbeip Lecture, Department
of Surgery, Harvard Medical School, June
123. Donabedian,, A., A Guide to Medical Care Ad- 12, 1976.
.ministration, Volume II: Medical Care
Appraisal-QUality and Utilization. New York 135 . Freeborn, D.K. and Greenlick, M.R., "Evalu-
(now Washington): American Public Health
ation of the Performance of Ambulatory
_:___Asso'ciation,1,969,1.176 pp. (plus Annotated Cafe Systems: Research Requirements and
Sete ted Bibliography by A.J.
34 March-April 1973.
124. )ook, R.H. and Appel, F.A.: "Quality-of- 136. Christoffei, T. and LoWenthal, H., "Evaluat-
Care Assessment: Choosing' a Method for ing the Quality of Ambulatory Health Care:
Peer Review." New. England Journal of A Review of Emerging Methodologies." Med-
Medicine 288:1323-1329, June 21.,,1973. ical Care 15:877-897, November 1977.
125. Brook, R.H., Quality of 'Care Assessment: A 137. American Society of Internal Medicine, As-
Comparison of Five Methods of Peer Review. sessing Physician Performance in Ambulatory
Washington, D.C.: U.S. Department of Care. San Francisco: The Society, .1976.
Health, Education and Welfare, Bureau of
Health Services Research and Evaluation, 13 8. American Nurses' Association-, Issues in
Evaluation Research. Kansas City, Mo.: The
July 1973 Association; 1976.
126. New York Academy of Medicine,)Corrunittee
on Public. Health Relations, Maternal Mortality 139. Several Authors, "Symposium on the Quality
in New York City: A Study of All Puerperal of Pharmaceutical Services." Drugs in Health
Deaths, 1930-32. The . Commonwealth Fund, Care 2:3-66, Winter 1975. .

New York: Oxford University Press, 1933. 140. Knapp, D.A. and Smith, - M.C., Evaluating
Pharmacists and Their Activities. Washington,
127. New York Academy of Medicine, Committee D.C.: American Society of Hospital Phafina7
on Public Health Relations, Petinatal Mortality cists, Researth and Education FoundatiOn,
in New York. City: Responsible Factors. Cam-
bridge, Mass. Harvard University Press, for 1973.
the Commonwealth Fund, 1955,. 141. Several papers in the Journal of the American
Dental Association 89: 842-871, October 1974.
128. Oakley, J.R.; McWeeny, P.M.; Hayes-Allen,
M. and Emery, J.L., "Possibly Avoidable 142: Riedel, D.C.; Tischler, G.L. and Myers, J.K.,
DeathS in Hospital in the Age-Group One Editors,' Patient Care EvalUiltion in Mental
Week to Two Years." The Lancet 1:770-772, Health Programi. Cambridge,' Mass,: Ballinger
April 10, 1976. Publishing Company, 1974. .

129. Gonnella, J.S. and Goran,1 M.J., "Quality of 143. Ehrlich, J.; Morehead, M.A. and, Trussell,
Patient Care-A Measurement of Changei, R.E., The Quantity,. Quality and. Costs of Mediial,
The Staging Concept."- Medical Care 13:467-, and Hospital Care Secured by a Sainple of Teams-.
473, June 1975. ter Families in the New York Area New York:
130. Gonnella, J.S.; Louis, D.Z. and McCord, JJ Columbia Unive ity, School of Public Health -
-."The Staging Concept-An Approach to the and Administra ve*Meclicine, 1962.
Assessment of Odtcome of Ambulatory 144. Shenkin, B.N. and Warner, D.C., "Giving the.
Care." Medical Care 14:13-21, January 1976. Patient His Medical Record: .A Proposal to
Improve the System." New England Journal of
.131. Brook, R.H. and Stevenson, R.L., "Effective- Medicine 289:688-692, September 27, 1973.
ness of Patient Care in an Emergency Room."
New England Journal of Medicine 283:904-907, 145. Stevens, P.D.; Stagg, R. and Mackay; I.R.,
October 22, 1970. "What Happens When Hospitalized Patients
132. Starfield, B. and Scheff, D., "EffeCtiveness of See Their OWn Records." Annals of Internal
Pediatric Care: The Relationship Between Medicine 86:474-477, April 1977.
Process and Outcome." Pediatrics 49:547-552, 146. Bouchard, R.E.; Tufo, 1.1.M,;- Van Baren,
April 1972. H.C.; Eddy, W.M.; Twitchell; J.C. and Be-
dard, L., "The Patient and His Problem-
133. McCarthy, E.G. and Widmer, G.W., "Effects
 Oriented Recoilr." In Walker, H.K.; Hutsts Regulation, June, 1975. DHEW Publication
J.W. and Woody, M.F., Editors, Applying the No.. (HRA) 77-621, Waihi gton, D.C.
Problem-Oriented System. New York: Me Fon)
Press, 1973. 151. Morehead, M.A., "P.S.R. Problems and
Possibilities." Man and Medicine 1:11'3-132,
147; Burger, C.S., "Audit in a Problern-Oriented Winter 1976.
Practice." In Walker, H.K.; :Hurstd.W. 152. Bernd, L.E., "PSRO Quality Control? Or
and Woody, MS., Editors,Apply'ing the Prob-
lems-Oriented 'System. New Vcirk: Medcom Giinmickry?" Medical Care 12:012-1018, De-
Press, 1973. cember 1974
153. Brian, E.W., "Government control of Hospi-
148. Goldblatt, P.I3.; Henisz, J.E. and Tischler, .tal Utilization: A California Expetience."New
G.L, "Utilization *view Within an Institu- England JoUrnal of Medicine 280340-1344,
tiOnal Context." in Riedel, D.E.; Tischler, June 22, 1972.
G.L. and Myers, J.K., Editors, Patient Care
Etiglua '''in-71te're:tai Health 'Programs. ..... 454-:---Glendenning; M:K.T Wolfe; -H.; Sh4m an, L.J.
bridge, Mass.: Bcallinger Publishing Com- . and Hither, G.A., "The Effect of a Target. 35
pany,,1974,. Date Based UtilYzation Review Program on
pany, 1974: 292 pp. Length of Stay." MediCal Care. 14:751-764,
Sep r 1976.
149. Donabedian, A., "Promising Quality 155. Lave, J.R. nd Leinhardt, S., lekn Evaluation
Through Evaluating the ProCess of Patient of a Hosp I Stay' Regulatory. Mechanism."
Care," Medical Care 6:1811202, May-June
1968. American Journal of Public Health 66:959-967,
'October 1976.
150. Freidson, E., "Speculations on the Social 156. Wyszewianski, L., work in progress, Depart-
Psychology of Local PSRO Operations." In inent of Medical Care Organization,, The
Proceedings: Conference on Professional Self- University of Michigan, Ann. A0or.
.

43
 (HRA) 77-3166 Foreign Medical S aduates: A Comparative
National Center for Health Services Research pub- Study of State nsure Policies (PB 265
lications of interest to the health community are 233) i'.
available on request to NCHSR, Office of Scien- (HRA) 774171 Analysis of Physician Pt,rice and Output De-
tific and Technical Inforination, 3700 East-West cisions (PB 273 312)
Highway, Row 7-44, Hyattsville, MD 20782 (tele- (HRA) 77-3.173 Nurse Prietponer and Physician Assistant
phone: 301/436-8970). Mail requests will be facili- Training and Deployment (PB 271.000)
tated by enclosure of aielf-addressed self-adhesive available NTIS only
miffing label. These. publicaficms also are available (HRA) 77-3177 Automation, of the Problem-Oriented Medi-
for sale.'through the National Technical Informa- cal Record (PB 286 88 1 )
Lion Service (NTIS), SPringfield; VA 22161 (tele-
phone: 703/557,4650). PB and HRP 'nurnbers in
parentheses are NTIS order numbers.. Publication,S
which are out of stock. in NCHSR are indicated' as Research Summaries
available only from NTIS. Prices may be obtained ty Series provides rapid aeCess
from the NTIS order desk on iliquest. The Research Sum
to significant resu s 'ol NCHSR-supported re-.
'arch projects. T e :series presentx.ekec the
summaries prepare by the investigators at Ghe
completion of the project. Screlific findi
Research Dig higleighedin a morecontise form than in th
report. The Research Summary Seri". iiintend for
The Research est SO provides overviews of health services administraiors, planners, and 'other
significant researc pported" by ..NCHSR. The research. users' who require recent findings rel-'
.A'serieideie 'lies either ongoing or completed prof- -eyant to immediate programs in health services.
ects direc toward. high priority health services
probleM . Issues are prepared by the principal in (HRA) 7:3182 Recent Stiitilqs. , Nealt Services Re-
vestigators perforMing the research, in collabora- search, Vol. 1. 974 t oughDecom-
tion with NCHSR 'staff. Digekts are intended for ber 1976) (PB 20 480
an interdisciplinary audienci of health services (HRA) 77-3183 Recent Studies I,n Health ServiceS Re-
planhers, administrators, legislators, and -others search, Vol. II (CY 1976) (PB 279 198)

who make decisions on research' applications. (HRA) 77-3176 Citality 'of Medical Care 'AssessMent Uetg
, . ar, Outcome Measures (P8-272 455)
(HRA) 78-3144 Evaluadon of a Medical Information System (HRA) 78-3193 Optimal Electrocardiography (PB 281' 558)
in a Community Hospital (PB 264 353) .

(HRA) 78-3145 CoMputer-Stored Ambulatory Record

(COSTAR) (PB 268-342)
(HRA)77 -3180 Prcram Analysiti of plipiclay Extendei Al- Policy Research
gorithm, Projec (P.B...264"e10, available ,
.

NTIS only) . The Policy Research Series desCribes findings from,

(HRA) 77-3161 Changes in the Costs of T-reatment of. the research program that have major signicc,ance
Selected. Illnesses, 1951-1964-1971 (HRP for policy issues of the moment. These papers are
v
001'4598) prepared by members of the staff of NCHSR or by
(HRA) 77-3163 Impact of State Certificate-of-Need Laws on independent investigators. The series is intended
Health Care Costs and Utilization (PB 264 specifically to inform those in the public gild prit
352)"
vats sectors who must consider, design, and ini-
(HRA) 77-3164 An Evaluation of Physician Assistants In
'Diagnostic Radiology (PB 286 507, avail-
..
plemein policies affecting the delivery of ,health"
able NTIS only)c
service's.
,.. 44
 :

(HRA) **74162 .Controlling thicost of Health Care (PB 288 (HRA) 76-3136 The Program in Health Services Research'
885) (Ravised 9178)
(HRA) 77-3158 Summary of Grants and Contracts, Active
JUne 30; 1976.
Research Reports (HRA) 77-3187 EmergenCy Medical Senitces Systems Re-
search Projects .( Active as of June 30,
'r The Research Report Series provides significant re- 1976) (PB 264 407, avallable.NTIS only)
.search reportsln their entirety upon the comple- (HRA) 77.3179) Research on the Priority issues of the Na-,
tion of the ,projeq.-geseareh Reports are yle- tional Center of Health Services Research,
Grants and Contracts Active on June 30,
veloped by the prince al investigators who con-
duCted the research, and are directed to selected
1976 '
(HRA) 77-3194 Emergency Medical Services Systems 9e-
users of-health services research as part of a con- search Projects, 1977 (PB 273 893).:
tinuing' NCHSR effort to expedite the dissernina- HRA) 7S- 4.
KCI4SELResearch..BibilogrePhy- (July--1
don of new knowledge resulting frill its, project o* 1978 to June 30, 1977) (PB 273 997)
support.
.37
(HRA)'76-3143 Computer Based Pail nt Monitoring Sys-
tems (PB 286 508) ,
Research Proceedings
4
(HRA} 77-3152 How Lawyers Handle Medical MalpractiCe
Cases (HRP 0014313) Research Proceedings Series extends the avail-
(HRA) 77-3159 An Analysis of the Southern California Ar-
ability of new research announced at key confer=
bitration Project, January 1988 thtough ences; symposia and seminars sponsored or sup-
June 1975 (HRP 0012466) ported by NCHSR. In additiOn to papers pre.
(HRA) 77.3165 Statutory Provisions for "Binding Arbitration sented, publications -in this series includes discus-
Of Medical Malpractice Cases (PS 264 40ft sions andoresponses whenever possible. The series
available NTIS only) , , intended to help meet the information needs of
.health services providers and others who require
.

HRA) 77.3184 '1g60 afici 1970 Spanish Heritage Fopula-

tion of the Southwest County (PB 280 direr ccess to concepts and ideas evolving from
656) the ex ange of research results.
(HRA) 77-3188 ,Pemonstration and valuation. of a Total
Hospital Information System PB 271 079) (HRA) 76-3138 Women and Their eith: Research impli-
: (IKA) 74189 cations- for a New E (VIP 264-359, avail-
Drug Coverrifie under National Health In-
surance: The Policy Options (PB 272 074) able NTIS.only)
(PHS) 78-3204 (HRA) 78-3150 interniountain Med cal Malpractice (PB 288
.Experiments in interviewing, Techniques:
Field. Experiments in Health Reporting (PB 344', available NTIS only)
/
.

276 080) (HRA) 77-3154, Advances In Health Survey 'Retearch

Methods (PB 282 230
(HRA) 77-3181 NCHSR Research Conference Report .on
Consumer Self -Ctrs. in Health (PB 273 811)
Research Management
(HRA) 77-3186 intern'ational Cbnference on Drug and
Pharmaceutical Services Reimbursement
The Research Management Series describes "pro- (PB 271 388)
grammatic .rather than technical aspects of the (PHS) 78-319t EmergenCy- Medical. Services: Research
NCHSR research effort. Information is presented Methodology (PB 279 098)
on rthe,'NCEISR goals, research objectives, a
v.
priorities; in addition, this series contaim lists
grants and contracts, and administrative inform Probram Solicitation.
tiOn on funding. Publications in this series are in-
<tended to bring basic. information 6n NCHSR and (HRA) 77-3196 Conference Grant Information
its prograrns to research plamirs, administrators, (HRA) 77-320-0 Grants for Dissertation Research Support
and others vibo are involved with the allocation of (PHS) 78-3206 Grants for Cost attainment Research. for
research resources. Health ,Planning

4.5
 BIBLIOGRAPHIC DATA I. Report No. ; ,
3, Recipient's Accession No-
SHEET
NCHSR 78-145 -

4. Title and Subtitle ..N...,

. .
5. Report Date
NEEDED RESEARCH IN THE ASSESSMENT AND MONITOR GIOF THE July 1978
QUALf pp. MEDICAL CARE; NCHSR Research Report Sgrias
',
7. Auth 6r(s) t ,..,

8. Performing OrganizStion Relit.

.

Avedis Donabedian .
Igo:,
9. Performing Organization Name and Address
The University of Michigan 10. Project/Task/Work Unit No.
School of Public Health .

11. Contract/Grant No.

Department of MedicalCare Organization' . ...
-Ann-Arbor, MI. 48104 AC 313/764-5433 .
HS 02081
12. Sponsoring Organization.Narne ape Address 13. Type of Rietrolotrte:iPmerilt
DHEW, PHS, OASH, Hational Center for Health Services
3700 East -West Highway, Rooi 7-44 (STIB) Research
5/1/76- - -4 /30/79
%
.
HyOtsville, MD 20782 AC'301/436=8970 14.

15; SupPlementary Notes '

,DHEW,Pub. No. (PHS) 78-3219. The final report will
.

1979. This is an interim reiOrt. not be. completed until approx. June
-

16. Abstracts .
.

/ The purpose of this report is to review, evaluate critically,

and synthesize the-liter-
ature on quality assessment and assurance, including the appropriateness Of
use of
service, in order to arrive at a ,cogent, documented, and authoritative
the state -of- the -art.
assessment of
In addition )to addressing quality assessment as a research tool
and quality assurance aslAn acflAistritive tool, an attempt is made to provide
derstanding of the epidemiology Of quality as a prerequisite an up--
to the.desigh of medical
care prograMs and - systems. Major components af quality which are discossed include:
(1) definitions, (2) quality Assessment and pr gram evaluation, (3)
relationship of
quality and quantityl (4) relationship of qua. ty a
cost,'(5) strategies of care,
(6) structure, process, and outcome, (7) monitorin versus research, and (8) the
.Of outcomes. uses
Recommendations f further,research in the assessment and 'monitoring of
1 the quality of medical'. care are, presented.
.

1Tribey-VaretrarrtImbarvinvetnn4ite+yriv7-49er./4..;...tiprora- '
.
1
.

. .
HCHSRpublication of research findings does not necessarily represent approval or' '.
'official endorsement by,the National Center for-Health Services Research or the
U.S. Department ol'Health, Education, and Welfare.
.

.
.
. , .

Diane N. Funk: NCHSR P.O.; V1301/436-8941: 1

4.. .

17b. IdentifierS/Open-Ended Terms .

.
.
Health aervices research .

Quality of medical care

Quality assessment ,
t
Quality assurance
Monitoring the,quality,of care .

, .

.
.. .

17c. COSATI Field 'Group 0 .

18. Availability Statement 19. Security Class /This

Releasable to the public. Available from National 21. No. of Pages
Report)
.Technical Information Service, Sp ingfield, VA g Est.: 50
ecurity C ass ( is 22. Price
AC 703/557-4650 22161 Page
4014M 447>S-35 MEV- 10,73> UNCLASSIFIED
i
4
ENDORSED BY ANSI A D UNESCO. THIS FORMMAY BE REPRODUCED uscomm-4C 5265-P74

6,