International Journal for Quality in Health Care 2000; Volume 12, Number 3: pp.

231–238

Methods for external evaluation of health

care institutions
LLUÍS BOHIGAS1 AND CATHERINE HEATON2
1
Director, Avedis Donabedian Foundation, Barcelona, Spain and 2External Peer Review Techniques, Bristol, UK

Abstract
Objective. To compare the methods used by external evaluators of health care institutions in Europe.
Methods. A common framework for analysis was designed by the authors and shared among the members of the ExPeRT
Project. Each member prepared a description of a model and the results were compared in two workshops.
Results. Programmes share similarities in the methods used, but they differ in the focus and purpose of the evaluation.
Differences in focus included whether a part or the whole of the institution is analysed and whether the review is patient
or system centred. Different purposes of the programmes are reflected in the emphasis and use of the methodological tools:
for instance, the appeal system is used only in the programmes that provide a certificate to the institution audited.
Keywords: external evaluation, health care institutions

The External Peer Review Techniques project (ExPeRT) The evaluation process consists of all activities that the
began, in 1996, to catalogue the range of programmes offering evaluator performs in order to give an informed opinion
external evaluation of health care facilities in the European about the quality and the performance of the reviewed
Union. These programmes fell within four main categories institution.
or approaches to the external evaluation of health care Each of the two activities analysed is comprised of a
institutions: accreditation, certification by the International set of processes. This article is structured following these
Organization for Standardisation (ISO. In Greek ‘iso’ means processes. For each process, its content will be explained
equal), award-seeking [European Foundation for Quality and information from the four different evaluation pro-
Management Award (EFQM)], and professional peer review grammes will be presented. The order of presentation will
or ‘visitatie’ (Dutch for ‘visitation’ or ‘peer review’). These be accreditation, ISO, EFQM and visitatie. Not every process
different approaches are presented in the paper by Shaw in applies to each evaluation method.
this issue [1].
The ExPeRT project analysed the strategic and the op-
erative aspects of these four approaches. The strategic aspects Methods
are presented in other articles in this issue; this article is
devoted to studying the operational activities undertaken by The comparison was made by means of a protocol that was
the four approaches to gather information and to make circulated among the partners of the ExPeRT project [1].
decisions about the institutions evaluated. Three crucial ac- Each partner prepared a summary of one programme. These
tivities in the operations of external evaluators are [2]: summaries were presented and discussed at a seminar in
Budapest, 6 October 1998, and a draft comparative paper
• the development of standards;
was analysed at a seminar in Stockholm, 24 May 1999.
• the selection, training and monitoring of evaluators; The partners reviewed the different systems. Lluı́s Bohigas,
Avedis Donabedian Foundation (Spain), prepared the analysis
• the evaluation process.
of accreditation programmes; Catherine Heaton, ExPeRT
This article will focus on the second and third activities. The (UK), analysed ISO certifiers; Iris Blomberg, National Board
selection, training, and monitoring of evaluators comprises all of Health and Welfare (Sweden), studied the European
the activities undertaken to prepare peer reviewers to gather Foundation of Quality Management; Jacqueline van Bat-
information about the quality of the institution evaluated. enburg and Bertjan Koekenbier of the National Organization

Address reprint requests to Lluı́s Bohigas, Director, Avedis Donabedian Foundation, Provença 293, 08037 Barcelona, Spain.
E-mail: [email protected]

 2000 International Society for Quality in Health Care and Oxford University Press 231
L. Bohigas and C. Heaton

for Quality Assurance in Hospitals (CBO, The Netherlands) EFQM assessors include some academics and quality pro-
studied visitatie. In the analysis of accreditation programmes fessionals but the majority are drawn from the ranks of
data was used not only from European accreditation pro- experienced managers currently practising in Europe.
grammes, but also from the programmes of the Joint Com- Visitatie schemes include the following selection criteria
mission on Accreditation of Healthcare Organizations for potential visitors (different scientific associations may
(JCAHO, USA), the Canadian Council on Health Services vary). A visitor:
Accreditation (CCHSA), the Australian Council on Healthcare
Standards (ACHS) and the New Zealand Council on Health- • has to be a registered specialist for at least 5 years;
care Standards (NZCHS). • should be independent of the clinical staff being sur-
veyed.
In particular the visitor should not:
Selection and training of evaluators
• be working in the same region as the clinical staff
The management of evaluators comprises two basic processes: under review;
selection and training.
• be the former trainer of one of the departmental
clinical staff members being surveyed;
Selection
• be the former promoter of one of the departmental
In accreditation, the main criterion for recruiting surveyors
is experience in the health care sector within the defined clinical staff members being surveyed.
professions of doctor, nurse and administrator or chief ex-
ecutive. All accreditors require a minimum period of ex- Training
perience in senior managerial positions that varies between In accreditation all surveyors undertake training at the be-
2 and 5 years. The accreditors that employ volunteers prefer ginning of their surveyor careers. Most accreditors require
their surveyors to be professionals currently practising in a 2–4 days of initial training, an exception being JCAHO which
health care facility: JCAHO requires nurses and administrators requires surveyors to undertake 15 days of orientation and
to hold a Masters degree; CCHSA requires its surveyors to training. Thereafter surveyors receive ongoing updates and
be employed in an accredited institution; ACHS requires education of between 1 and 5 days per year. The training
knowledge of the Australian health care system, good in- methodologies are always participatory. The content of the
terpersonal skills and commitment to ACHS accreditation; training usually includes the following areas: standards know-
NZCHS requires knowledge and experience in continuous ledge; survey processes; communication; interviewing; and
quality improvement [3]. report writing. Most accreditors use the observation of real
The ISO norm EN 45012 states that the certification body surveys as a part of the training.
shall ‘employ a sufficient number of personnel having the ISO requires auditors to be trained and assessed by ex-
necessary education, training, technical knowledge and ex- ternally recognized training bodies, who approve auditor
perience for performing certification/registration functions training programmes and officially certify auditors who have
relating to the type, range and volume of work performed’. passed certified courses. Some certification bodies run their
This norm does not require certification bodies to employ own training programmes. Most use auditors who have
‘peer reviewers’ for the health care sector. When assessing completed approved training courses and have been certified
health care facilities, many certification bodies in Europe use by the International Register of Certificated Auditors (IRCA)
auditors with some health care experience, making a certain or an equivalent organization. IRCA-approved training
level of relevant expertise mandatory. These bodies add more courses are required to last for 36 hours (soon 40 hours) and
specific requirements to their auditor selection criteria, for cover all procedures relating to the audit of facilities to ISO
example: knowledge and experience working in quality man- 9000 standards. Training courses are usually residential, spread
agement; understanding of certification schemes and a mini- over 5 days and include role-play, lectures and, in some cases,
mum of 20 days auditing experience; relevant health care a live audit. In addition, IRCA auditors are required to meet
qualifications, including medical, nursing or health care man- certain academic qualifications and have experience in a
agement; and seniority in the health care sector. Criteria working quality and audit environment, in order to become
for recruiting auditors include attributes such as maturity, certified. ISO standards require simply that an auditor’s ability
judgement and understanding, analytical skills; general edu- to meet ISO requirements is judged by an evaluation panel
cation and language fluency; management capabilities; training chaired by a qualified auditor and including members selected
and evaluation ‘to the extent necessary to ensure their com- by internal audits, customer audits and independent third
petence in the skills required for carrying out and managing party audits. Selection of the auditor or audit team operates
audits’; at least 4 years full-time appropriate practical work- under the general guidelines of ISO 10011.
place experience; at least 2 years in quality assurance activities; In visitatie the visitors receive 1 day’s training from CBO
and at least 20 days and four audits as a trainee auditor, on procedures, attitudes and techniques. CBO continues to
including documentation review, actual audit activities and coach them through their first visit with specific personal
audit reporting. feedback.

232
 External evaluation of health care

The evaluation process Having registered with an ISO accredited certification/

registration body and determined the need, purpose and
The evaluation process usually comprises three phases: general scope of the audit the client institution is encouraged
to show evidence of self-preparation in some of the following
• survey; ways:
• evaluation; • documentation;
• between surveys. • internal audits;
The survey is the primary fact-finding phase. It usually takes • management review;
the form of a team of professionals visiting the institution and
• pre-audit by an external organization.
writing a report of their findings. The second phase is
contingent on the survey: based on the report, the evaluator Within certain guidelines, ISO allows the content of pre-
uses criteria and scoring to grade and eventually give an audit documentation submitted to the certification body to
award or a certificate to the evaluated institution. The third be specified by the client. This should include information
phase occurs between the survey or certification and the next on general features and resources of the institution; a copy
evaluation [4]. of the institution’s quality manual; general information on
the quality systems in place and a description of those to be
Survey assessed.
EFQM promotes ‘self-assessment’, believing the process
The survey phase can be divided into three subphases, namely:
to be a catalyst for driving business improvement. The EFQM
preparation; survey; report.
definition of self-assessment is as follows:
Preparation (i) Self-assessment is a comprehensive, systematic and
The preparation for the survey comprises all activities carried regular review of an organization’s activities and
out by the accreditor and the institution to prepare for the results referenced against the EFQM model of busi-
survey evaluation. ness excellence.
Five elements are present in the preparation phase, although (ii) The self-assessment process allows the organization
not all systems apply all processes: to discern clearly its strengths and areas in which
improvements can be made and culminates in
• questionnaire;
planned improvement actions that are then mon-
• self-assessment by the institution being evaluated; itored for progress.
• composition of the evaluation team; When applying to EFQM, the health care service is asked
to provide data about the activities generally derived from
• agenda or audit plan; their self-assessment. This information must be closely aligned
• pre-survey activities. with EFQM’s nine award assessment criteria.
Clinical departments participating in a peer review (visitatie)
Questionnaire. The voluntary nature of the evaluation pro- scheme are also asked to complete a self-assessment ques-
cess is common to all four models, so the onus is on the tionnaire prior to the visit. This questionnaire addresses the
health care institution to request an evaluation. The evaluator’s organizational aspects of professional performance, giving
initial response often takes the form of a questionnaire aimed the ad hoc visitation committee an opportunity to select and
at identifying the institution’s eligibility. This questionnaire is discuss key quality issues with the practitioners and other
sometimes considered a contract between the institution and staff members before the evaluation visit.
the evaluator, and includes notification of the evaluation fees
and other legal regulations, such as confidentiality of the Composition of the evaluation team. Also common to the
data. The demographic and biographic data obtained through evaluation process of all four approaches is the use of an
the questionnaire are usually the basis for planning the size auditor or team of reviewers. This team is composed of
of the evaluation team, the number of days, etc. professionals trained to evaluate the quality of service pro-
vided by the institution. The size of the evaluation team
Self-assessment by the institution under evaluation. All depends on the size, the nature and complexity of the health
evaluators suggest that the institution undertake a self-assess- care organization being surveyed.
ment. How this self-assessment is used varies not only In the accreditation field, the core survey team consists of
between the four models but also between programmes a senior executive physician, nurse and administrator, usually
within the models. led by one surveyor, chosen on the basis of seniority. As
Some accreditation programmes use the institution’s self- organizations increase in size so the survey team gets larger;
evaluation to guide the surveyors in their assessment. The for example, a small community hospital may be reviewed
institution may be asked to state or grade its compliance to by a team of two surveyors, a larger acute hospital by a team
a set of explicit standards; the surveyors then verify the of four. The composition varies according to the accrediting
institution’s results and comment on the differences. body; some prefer nurse–manager teams; others doctor–nurse

233
L. Bohigas and C. Heaton

teams. Most accrediting bodies prefer surveyors who are body in collaboration with their client is to determine the
generalists (rather than specialists) with specific experience scope of the audit and make an initial review, to ensure that
and expertise in defined settings such as acute care, community the client is sufficiently prepared for the audit. This usually
care, ambulatory care. takes the form of a visit to the ExPeRT by the lead surveyor.
Some organizations, for example Lloyds Register, divide
Agenda/audit plan. External evaluators usually define the these responsibilities between the business manager, who
general structure of the on-site evaluation using a timetable, determines the size and agrees to the scope of the audit, and
agenda or audit plan. The generally accepted philosophy is the lead auditor, who ensures the facility is ready and discusses
that the institution should know what is being evaluated and the audit schedule. ISO specifies that the final decision about
when, rather than receiving a surprise visit. Within limits, the which functions and locations are to be audited rests with
evaluators usually have the autonomy to ask about or to visit the client, but should be reached with the assistance of the
other areas besides those planned. lead auditor.
A crucial aspect is the duration of the survey, since this is The length of time taken for an ISO audit depends on
the key variable to the cost of the evaluation process. The the number of employees, location/sites and complexity of
duration of the survey in the accreditation field varies ac- the service. Based on these factors, ISO Guide 62 outlines
cording to the size, nature and complexity of the health care the time requirement for each audit. Certification bodies are
organization being surveyed. This may range between 1 day free to allocate the time requirements between as many or
for a small community health centre to 10 days for a large as few auditors as they feel necessary. For example, SGS in
regional health organization with multiple sites. The number The Netherlands stipulates that a minimum of two auditors
of beds is a common measure of size, although some ac- must audit a health care facility. Others bodies may use only
creditors look at the institution’s activities (such as visits, a single auditor.
length of stay) to determine the complexity and consequently
the length of the evaluation. Survey/audit
ISO’s audit plan, produced by the certification body and The purpose of the survey/audit is to gather information in
agreed by the client, identifies the objectives and scope of the order to write the report. The procedures and methods used
audit; audit team members; personnel with key responsibilities by the evaluators during the survey/audit are central to the
within the facility; reference documentation; the language of operations of the evaluating body. Documentation of these
the audit; date; time; duration; meeting schedule; con- methods enables evaluations to achieve a level of consistency.
fidentiality requirements; and report deadline and distribution. Elements used in accreditation surveys usually include:
In visitatie, the practitioner being visited is responsible for the
agenda of the survey. He or she decides whom the visitation • review of documentation;
team will interview. The visit usually lasts 1 day, but it can
be between 0.5 and 2 days depending on the number of • interviews;
locations and/or practitioners being visited. • sample of medical records and other types of records;
Pre-survey activities. Pre-survey activities vary considerable • visits-observations.
between the four models. Some programmes offer visits prior
to the formal evaluation, others allocate personnel to guide All accreditors offer feedback at the end of the survey,
the institution through the process, whilst others engage in to rehearse the formal written report and to verify their
discussions between the evaluating body and the health care observations. The Health Quality Service survey includes a
service. The completion of a self-assessment and audit plan night visit, in order to find out information about what
is considered sufficient by some programmes. happens in an important and usually neglected part of the
Some accreditation programmes allocate a client or service hospital activity.
manager to the institution to facilitate the complexities of Consistency among surveyors is a crucial item of the
pre-survey activities. How important this person is to the accreditor’s performance. A programme’s reputation depends
accreditation process varies from programme to programme. upon objectivity rather than the subjective individual judge-
The Health Quality Service (UK) uses this position to instruct ment of each survey team member. Protocols and training
the institution about the preparation phase, to be present are the main tools to ensure surveyors’ consistency. During
during the survey and to write the report. Other programmes the survey some accreditors require surveyors to reach con-
do not use survey managers. The Hospital Accreditation sensus in grading the standards. Another way to gain this
Programme (UK) offers facilities the opportunity to par- consistency is to allocate the task of editing and compiling
ticipate in a pre-survey which takes place 3 months before the report to one person, the lead surveyor or the client
the full survey. This involves a 1-day survey by a single manager, together with the central administration of the
surveyor, who assesses the facility against the standards, and accreditation programme to maintain internal consistency and
offers verbal (rather than written) recommendations and compliance with defined rules for reports.
guidance in preparation for the full survey. The surveyor The ISO audit begins with an opening meeting to review
who undertakes the pre-survey is not included in the full and clarify the scope and the objectives of the audit; introduce
accreditation survey team. the audit team to senior management; and summarize the
The primary activity undertaken by the ISO certification procedures to be used in the audit. During the audit, evidence

234
 External evaluation of health care

is collected through interviews and the examination of docu- EFQM’s feedback report is prepared by a team of in-
mentation and activities, as well as observation. The closing dependent assessors, who are senior managers and experts
meeting enables the audit team to present their observations from across Europe. The report provides a list of strengths
and conclusions prior to writing the report. and areas for improvement under each criterion addressed
In visitatie, during the collegial review, peers evaluate the in the application. The assessor’s scoring profile is also given
circumstances under which clinical practice takes place by: together with comparative scoring of other applicants for the
award. After submission of the report to the institution, the
(i) Documentation: prior to the survey the physicians senior assessor attends a meeting to discuss the content of
are asked to inform the ad hoc visitation team about the the feedback report.
availability and status of guidelines, patient medical The visitatie report consists of a description of the clinical
records etc. department, positive and negative findings and re-
(ii) Observations: part of the survey is to check whether commendations with suggestions for improvement.
guidelines are available, readable and consistent;
(iii) Structured and non-structured interviews: intra- and Grading of standards. Most programmes for the external
inter-disciplinary collaboration, evaluation of patient evaluation of health care services grade their review findings.
satisfaction and treatment outcomes. Accrediting bodies do not simply record compliance/non-
compliance but expect the surveyors to grade the degree
Collecting information by interview is not limited to the of compliance. Grading systems vary from accreditor to
physicians under review. Uniformity in medical performance, accreditor and may be numerical (e.g. 1–5) or descriptive, (e.g.
or the use of guidelines in medical practice, is also discussed minimal, partial, substantial or non-compliance). Institutions
with other professionals working closely with the physicians may even be asked to grade themselves, the surveyors check-
being visited (i.e. the nursing staff, the chairman of the ing these grades during the survey and commenting in cases
medical staff, assistants, residents, colleagues from other of divergence.
specialities, etc.). There is a feedback session at the end of An ISO non-compliance is graded major or minor, ac-
each survey by the ad hoc visitation team. Through feedback cording to its impact on the quality of the final product or
on their performance and by formulating suggestions for process.
improvement, visitatie supports physicians in their efforts to EFQM assessors use a defined scoring process to allocate
improve patient care. points to each assessment criteria. The factors used to evaluate
enablers and results are presented. Each of the parts of
Report the enabler criteria is evaluated according to approach and
Each model emphasizes the importance of a report. The deployment. Approach is concerned with the methods the
evaluation report will be analysed in three parts: report organization uses to address the criterion parts.
content; grading of standards; report writing. The score given takes account of the achievement of these
items:
Report content. Accrediting bodies produce reports dem-
onstrating compliance and non-compliance with explicit • the appropriateness of the methods, tools and tech-
standards. Although some accreditors produce negative re- niques used;
ports, including only areas of non-compliance, the favoured • the degree to which the approach is systematic and
method is to balance recommendations for improvement with prevention based;
positive comments on the good practices found. Accreditation
reports usually contain a text summary as well as numerical • the use of review cycles;
grading with appropriate comment, measured against the • the implementation of improvements resulting from
standards. review cycles;
The report resulting from an ISO 9000 audit contains the
details included in the audit plan; documentation against • the degree to which the approach has been integrated
which the assessment was made; ‘observations of non-con- into normal operations.
formities’ or areas which did not comply with the agreed Deployment is concerned with the extent to which the
quality system standards, protocols and procedures; and the approach has been implemented to its full potential. The
audit team’s judgement of the level of compliance. As well score given will take account of appropriate and effective
as simply stating non-compliance, as ISO standards require, application of the approach:
most certification bodies include a balance of positive and
negative findings. In The Netherlands, non-conformities re- • vertically through all relevant levels;
ported are based on the nine aspects of the HKZ (the • horizontally through all relevant areas and activities;
Dutch harmonization model). The lead auditor also has the
• in all relevant processes to all relevant products and
responsibility for submitting the report to the client and
services.
ExPeRT who may then agree to distribute it more widely.
‘The audit is complete upon submission of the audit report Each of the parts of the results criteria is evaluated
to the client’. Any corrective action or re-audit is agreed according to the degree of excellence and the scope of the
between the ExPeRT and the auditing organization. results presented.

235
L. Bohigas and C. Heaton

The degree of excellence of results takes account of: The evaluation committee
The evaluation committee is the group of people within or
• positive trends and/or sustained good performance;
affiliated to the evaluating body that evaluates the report and
• comparison with own targets; makes the decision whether or not to award accreditation/
certification to the institution evaluated.
• comparison with external organizations (including Accrediting bodies call this committee the assessment
competitors and ‘best in class’ organizations wherever panel, accreditation committee, or independent, professional
possible); board. The status and composition of this group also varies:
• evidence that results are caused by appropriate ap- the committee may include any combination of the following:
proaches. senior, independent health care professionals; surveyors; client
managers; representatives of the accrediting body; the chief
The scope of results takes account of: executive or other representatives of the institution itself. In
• the extent to which the results cover all relevant areas cases of non-agreement or no accreditation, the decision may
of the organization; be referred to a higher body. The relationship of the surveyors
to the board is a distinction between many accreditation
• the extent to which a full range of results, relevant to programmes, as some keep the survey and final evaluation
the criterion part, is presented; entirely separate, whereas others link the two activities closely.
• the extent to which the relative importance of the The length of the evaluation process may indicate the
results is understood and presented. quality of the accreditor’s performance. Most accrediting
bodies designate a maximum period of time between survey
An overall percentage score is then derived and converted and accreditation decision.
into points according to the values shown in the EFQM ISO guidelines state that the decision should not be made
model. by any person participating in the audit, but that it should
In visitatie consensus within the team is the first criterion be taken by the personnel within the certification body
for each formal conclusion or recommendation made. The responsible for auditing the facility, rather than delegated to
second criterion is that judgement must be based on more any outside person or body. There is therefore no independent
that one argument mentioned somewhere in the report. committee involved in the decision. ISO does not stipulate
Thirdly, a balance in positive conclusions and re- whether an individual or group should take the final decision,
commendations for improvement is necessary. nor does it give guidelines on the composition of the decision
making team. Exactly who within the certification body is
Report writing. Most accreditation programmes require each
responsible for the decision varies from organization to
surveyor to write a section of the report. Surveyors may be
organization. Some have appointed assessment committees
given a time limit within which to do this, ranging from
elected from the governing board of the organization, others
completion during or on the day of the survey to completion
allocate the responsibility for all decisions to a chief auditor
2 weeks after the visit. Several accrediting bodies use client
(not involved in the audit in question) employed by the
managers or analysts employed by the accreditor to write the
certificating body.
report, based on the surveyors’ notes and observations.
In ISO, the audit or assessment report is prepared by the Rules for making decisions
team under the direction of the lead auditor. In three of the four models (accreditation, ISO and EFQM),
The findings of the visitatie team are written up in a report, the report with graded compliance is converted into a final
which is completed within 2 or 3 weeks of the visit by decision, whether that be accreditation, certification or an
someone from CBO. award. This process can be divided into two distinct activities:
the first is to aggregate the many scorings into one single
Evaluation score or a limited list of scores, based on the aggregation
The evaluation of reports is an area of significant divergence rules; the second activity follows this and involves a decision
among the four types of external health service review on whether this score merits the award or not, depending
programmes. The report may be examined by an individual or on the decision rules. These rules may be implicit in the
a committee that applies explicit rules and implicit judgement minds of the members of the evaluation committee or explicit,
when making the decision of whether or not to accredit/ documented in a series of rules or algorithms forming part
certify the institution. of the policy of the evaluating body.
The following elements may be involved: In the accreditation field the aggregation and decision rules
may be published as part of the standards manual, surveyor
• an evaluation committee; handbook or award recognition guidelines, and used when
• rules for making decisions; reviewing reports. These may include additional elements
such as key safety/business risk criterion. The Hospital
• a certificate or award; Accreditation Programme Board uses a numerical scoring
mechanism to assess the level of accreditation through ag-
• an appeal;
gregated scoring by the surveyors. This is monitored closely
• publication of results. against the surveyors’ recommendation for accreditation.

236
 External evaluation of health care

The rules for deciding whether to award ISO certification of accreditation. The board examines the survey report, the
are developed individually by each certification body and rules applied by the accreditation committee and the evidence
audited by an external organization such as United Kingdom submitted by the hospital in order to reach the final decision.
Accreditation Service (UKAS). Most base their decision cri-
teria on the level of compliance with the standards and on Publication of results
the severity of non-compliant aspects of the quality system. Most accreditors publicize a list of accredited institutions
HKZ in The Netherlands states that the audittee must comply (not including the grade of award or the names of those
with all norms within the certification scheme in order to institutions that have failed to achieve accreditation). In
gain certification. The auditors’ recommendations may be response to public demand, the JCAHO has a policy of
graded and the decision made depends on the grades given. publicising, under request, the main findings of the ac-
creditation survey by comparing the institution with a range
The certificate of similar ones.
A crucial aspect of accreditation and ISO that differentiates The results of visitatie are strictly confidential. The report
them from other external evaluations is the certificate. This is sent only to the physicians visited, who decide whether to
certificate is a public announcement by the evaluating body discuss the survey results with their employers, the hospital
that the institution has been reviewed against a set of standards management and other parties involved in the visitation
and is significantly compliant with them. report.
The accreditation certificate has two main features, the
time span that it covers (usually up to 3 years) and the degree Between surveys
of compliance. The 3-year accreditation award is common
among accreditors, although some use a range of time periods Most accrediting bodies survey each participating institution
between 1 and 5 years. every 3 years. To avoid institutions dropping their standard
The levels of accreditation awarded by the Hospital Ac- of performance between surveys, accreditors have devised
creditation Programme (UK) are as follows: several mechanisms. An agreed quality action plan on how
the institution will implement recommendations may be
• accreditation for 3 years; followed, involving an interim visit to review progress. Un-
• accreditation for 2 years; announced surveys may be made, to a sample of institutions
in the years in between surveys, and facilities may be required
• accreditation for 1 year; to submit standard performance data. The Hospital Ac-
creditation Programme requires facilities to indicate the action
• focus survey (pending award);
taken on the previous recommendations in advance of further
• non-accreditation. survey visits.
Most organizations using the ISO model conduct annual
When the ISO certifier reaches a positive decision, the or biannual audits on previously agreed aspects of the quality
audittee is provided with a certificate for 3 years, stating the system. If the audittee has not achieved certification, a number
name, address, dates, terms and ISO standards against which of types of re-audit may be offered. Facilities may receive a
the health care facility’s quality system have been assessed. follow up visit 3 months after the initial audit, to ensure that
Detail on the scope of certification is included and the status the one or two serious problems identified have been rectified.
of the certification body, as well as the expiry date of the There may also be a partial re-assessment 6 months after the
certificate. If the facility has undertaken the audit process original audit, or for a number of serious non-compliances,
through an accredited certification body, their certificate another full audit.
should recognize this. In the UK, accredited certification is
recognized by the tick and crown symbol, in the Netherlands
by the HKZ and RvA logos.
For most scientific associations the visitatie process has a Conclusions
5-year cycle. After this time, the facility is reviewed by
another team of visitors to establish the degree to which Though the terminology of the four main models of external
recommendations have been followed. The return visit must review of health services may differ (for example evaluators
demonstrate compliance with previous suggestions for im- have different names according to the model: accreditation-
provement. Some scientific societies have introduced return based models use the title surveyors; ISO calls its reviewers
visits after 1 or 2 years, particularly in cases of widespread auditors and lead assessors; EFQM uses the term assessors;
non-compliance with the quality norms. and visitatie the terms visitors) the evaluation processes of
the four models share a similar evaluation methodology.
Appeal Common features include:
In a case where the accreditation committee recommends
• voluntary initiation by the institution;
non-accreditation, the hospital may choose to follow an
appeals procedure, in order to have its case reviewed again. • self-assessment;
The appeal is usually dealt with by the board or committee,
who ask the hospital to submit data or information in favour • agenda or audit plan;

237
L. Bohigas and C. Heaton

• evaluation visit; care, EFQM has no need of activity between visits as its
purpose is not continuous evaluation of an individual in-
• trained reviewer or evaluation team; stitution. The emphasis on self-assessment is important in
• written or verbal report; determining whether an institution may be eligible to compete
for an award. EFQM may promote improvement, but through
• evaluation of findings. competition rather than collaborative organizational de-
velopment. In contrast, visitatie has no need of an appeal
This suggests that these elements are essential to a struc- process, as its main purpose is to offer advice and support
tured approach to the external evaluation of health care for personal and organizational development between peers.
facilities. It does not necessarily follow that the differences This paper covers some of the common criteria and
between the programmes are superfluous, rather, the areas distinctive features of the four models’ evaluation processes.
where the approaches diverge result from the different em- In order to reach a generic set of core criteria for objective,
phases or purposes of the four models. consistent, external evaluation programmes across Europe,
One of the common purposes of accreditation and ISO these elements should be discussed both in association with
is to provide evidence that the quality of health service the other components of evaluation programmes and within
provision reaches a specified standard. This evidence is in the context of each of the four models.
the form of a certificate awarded to the evaluated institution
on successful completion of the evaluation process. The
corresponding purpose of EFQM is to select and give an
award to institutions achieving excellence in their business References
or service category. Only the winners receive an award
1. Shaw CD. External quality mechanisms for health care: summary
confirming high quality health care provision. Visitatie offers of the ExPeRT project on visitatie, accreditation, EFQM and
no corresponding award or certificate in its medical peer ISO assessment in European Union countries. Int J Qual Health
review process. Care 2000; 12: 169–175.
Other distinctions may be drawn between the different
2. Bohigas L, Smith D, Brooks T et al. Accreditation programs
focal points of each model. Accreditation looks at the entire
for hospitals: funding and operation. Int J Qual Healthcare 1996;
organization, either department by department, or with a 8: 583–589.
patient/client focus following the functions surrounding the
patient across all relevant departments. ISO 9000 examines 3. Bohigas L, Brooks T, Donahue T et al. A comparative analysis
designated quality systems, focusing on how the institution’s of surveyors from six hospital accreditation programs and a
consideration of the related management issues. Int J Qual
stated objectives are achieved, rather than how the institution
Healthcare 1998; 10: 7–13.
as a whole meets the needs of its patients. Visitatie directs
its attention to the appropriateness of service delivery pro- 4. Bohigas L. Accreditation across borders: the introduction of
vided by the medical practitioner. Joint Commission accreditation in Spain. J Qual Improv 1998;
These different purposes make some phases of the method- 24: 226–231.
ology more necessary than others. Where accreditation uses
focused surveys, workshops and training events between
surveys to facilitate continual improvement in the provision Accepted for publication 3 February 2000

238