Clinica Chimica Acta 346 (2004) 37 43 www.elsevier.

com/locate/clinchim

Has compliance with CLIA requirements really improved quality in US clinical laboratories?
Sharon S. Ehrmeyer a,*, Ronald H. Laessig a,b
a

Departments of Pathology and Laboratory Medicine, University of Wisconsin Medical School, Room 6175, 1300 University Avenue, Madison, WI 53706 USA b Population Health Sciences, University of Wisconsin Medical School, Madison, WI 53706, USA Received 18 December 2003; accepted 23 December 2003

Abstract Background: The Clinical Laboratory Improvement Amendments of 1988 (CLIA88) mandate universal requirements for all U.S. clinical laboratory-testing sites. The intent of CLIA88 is to ensure quality testing through a combination of minimum quality practices that incorporate total quality management concepts. These regulations do not contain established, objective indicators or measures to assess quality. However, there is an implicit assumption that compliance with traditionally accepted good laboratory practicesfollowing manufacturers directions, routinely analysing quality control materials, applying quality assurance principles, employing and assessing competent testing personnel, and participating in external quality assessment or proficiency testing (PT)will result in improved test quality. Methods: The CLIA88 regulations do include PT performance standards, which intentionally or unintentionally, define intra-laboratory performance. Passing PT has become a prime motivation for improving laboratory performance; it can also be used as an objective indicator to assess whether compliance to CLIA has improved intra-laboratory quality. Results: Data from 1994 through 2002 indicate that the percentage of laboratories passing PT has increased. In addition to PT performance, subjective indicators of improved qualityfrequency of inspection deficiencies, the number of government sanctions for non-compliance, and customer satisfactionwere evaluated. Conclusions: The results from these subjective indicators are more difficult to interpret but also seem to show improved quality in US clinical laboratories eleven years post-CLIA88. D 2004 Elsevier B.V. All rights reserved.
Keywords: Clinical Laboratory Improvement Amendments (CLIA88); Laboratory compliance; Quality laboratory results; Testing deficiencies; Indicators of laboratory quality; Good laboratory practices

1. Introduction To assess improvements resulting from the Clinical Laboratory Improvement Amendments of 1988

* Corresponding author. Tel.: +1-608-262-0859; fax: +1-608262-9520. E-mail address: [email protected] (S.S. Ehrmeyer). 0009-8981/$ - see front matter D 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.cccn.2003.12.033

and the most recent revisions in 2003, one must first understand their underlying philosophy [1 3]. The CLIA88 regulations like the ISO 9000 documents codify, for all testing sites, a collection of good laboratory practices associated with quality laboratories [4]. By incorporating a total quality management-based approach into the regulatory process, the laboratory director is made responsible for guaranteeing that appropriate quality practices

38

S.S. Ehrmeyer, R.H. Laessig / Clinica Chimica Acta 346 (2004) 3743

are followed. CLIA88 specifically states that the laboratory director must ensure that the . . . testing systems. . .used. . .provide quality laboratory services [covering] all aspects of test performance, which includes the pre-analytic, analytic, and post-analytic phases of testing [2]. To achieve this goal, CLIA88 specifies a universal (applicable to all test sites) set of minimum, good laboratory practices: (1) having trained and competent testing personnel; (2) following manufacturers procedural directions (calibration, maintenance, etc.); (3) routinely performing and evaluating daily quality control, responding to outof-control results and correcting problems; (4) applying total quality management and continuous quality improvement principles and practices; (5) participating in external quality assessments; (6) and documenting all activities. The only quantitative performance requirement in CLIA88 is to successfully analyze three sets of PT challenges per year. The mandated, minimum good laboratory practices provide no absolute performance standards. Instead, they take a quality management approach requiring the director to implement and set limits for the quality control parameters appropriate for the intended use of the laboratory results. To assess compliance with these practices, CLIA88 requires that testing sites be inspected biannually. Test sites found out of compliance (either with PT or inspection requirements) are subject to a series of progressive sanctions ranging from requiring plans of correction, to monetary fines, to testing limitations, to revocation of the sites CLIA certificate, i.e., prohibiting testing entirely. The regulation of US laboratories has a brief history. From 1967 to 1988, 12,000 large clinical and reference laboratories were required to follow the quality standards embodied in two federal regulations, the Clinical Laboratory Improvement Act of 1967 (CLIA67) and Medicare 1968 [5,6]. The purpose of these regulations was to improve the quality of laboratory tests in federally funded programs and to ensure that federal healthcare dollars were spent on high quality laboratory tests. In the mid-1980s, a series of newspaper articles highlighted laboratory errors associated with Pap smear testing. The U.S. congress generalized these errors and enacted the CLIA88 amendments designed to ensure quality in all laboratories including physicians offices. For the

first time, all clinical laboratory-testing sites were required to meet uniform standards. CLIA88 as implemented by the February 28, 1992 and subsequent regulations recognize three major categories of testing: waived, moderate complexity and high complexity. Only laboratories registering as moderate and high complexity are required to participate in PT and to undergo bi-annual inspections for the mandated, good laboratory practices. Waived tests were originally limited to a total of nine analytes using methodologies described as so simple to perform that there is essentially no possibility of a wrong test result or, if erroneous, pose no risk to the patient. Most of these were already available, over-the-counter, without prescription to the general public.

2. Indicators of quality Before determining whether the quality of test results has improved as a result of compliance with CLIA88, one must first attempt to do what CLIA does notthat is to define quality and, to a lesser degree, what level of quality is needed to meet good laboratory practice standards. While adherence to government-mandated good laboratory practices is intuitively assumed to improve the quality of test results, linking quality to compliance, as measured by inspection, is difficult because of the lack of a direct relationship. The only objective measurement of quality in CLIA88 regulations is performance in external quality assessment or PT programs. The Centers for Medicare and Medicaid Services (CMS) website claims that PT is a measure of laboratory quality and that PT performance is an indicator of a laboratorys ability to provide quality patient test results [7]. This official, government-sponsored, web site also declares that test accuracy, as assessed by PT: (1) reduces repetitive testing, (2) aids in rapid and appropriate patient diagnoses, (3) contributes to effective treatment and (4) reduces overall costs of medical care. The CMS concludes that the review of a laboratorys PT performance data educates the laboratory staff. This educational purpose closely parallels and further validates the premise first discussed by Belk and Sunderman, who beginning in 1946, demonstrated that mere PT participation motivated laboratories to improve performance [8]. Most

S.S. Ehrmeyer, R.H. Laessig / Clinica Chimica Acta 346 (2004) 3743

39

importantly, now as then, a laboratory, which fails PT by reporting wrong results, is better able to recognize and presumably correct a performance problem. A fundamental assumption, then and embodied in CLIA88, is that a performance problem identified by PT is also present in patient test results. Belk and Sundermans findings motivated the College of American Pathologists to offer, for educational purposes, PT surveys beginning in 1947. Twenty years later, the US government mandated PT participation as part of the CLIA67 requirements governing 12,000 large hospital and reference laboratories [5,9]. CLIA88 extended mandatory PT to all non-waived testing sites. The CLIA 88 mandated, analyte specific, PT tolerance limits for selected chemistry analytes are shown in Table 1 along with examples of acceptable performance ranges for stated target values. These tolerance limits are total error standards that incorporate both inaccuracy and imprecision and are defined as either fixed limits (target value F an absolute percentage of the target value) or target value F a multiple (usually 3.0) of the PT group standard deviation. For example if the target value for sodium is 140 mmol/l, using the CLIA88 tolerance limit of 4.0 mmol/l, a laboratorys PT result to be acceptable must fall within the range of 136 144 mmol/l. Each PT event includes five samples for an analyte. Acceptable performance in a single event is achieving at least four out of five correct results (>80% correct performance). More than one incorrect result ( < 80% correct) constitutes a failed analyte in that PT event.
Table 1 CLIA88 proficiency testing, tolerance limits and acceptable performance (AP) ranges Test or analyte Albumin Calcium, total Cholesterol, total Glucose Sodium Urea nitrogen CLIA88 tolerance limit (TL) AP = V F TL TL = 10%, If TV = 40 g/l; AP = 36 44 g/l TL = 0.25 mmol/l, If TV = 2.20 mmol/l; AP = 1.95 2.45 mmol/l TL = 10%, If TV = 5.0 mmol/l; AP = 4.5 5.5 mmol/l TL = 10%, If TV = 6.7 mmol/l; AP = 6.03 7.37 mmol/l TL = 4.0 mmol/l, If TV = 140.0 mmol/l; AP = 136 144 mmol/l TL = 9%, If TV = 6.0 mmol Urea/l; AP = 5.46 5.54 mmol Urea/l

Because CLIA88 only identifies one objective measurement of quality, subjective indicatorsthe frequency of inspection deficiencies and government sanctionsalso were evaluated. CLIA88 requires laboratories to be inspected every 2 years. The inspectors look for compliance with the mandated, good laboratory practices. Inspectors also determine the degree to which total quality management concepts, such as regular assessment of the total testing process by the laboratory director, are incorporated into the laboratory operation. Conceptually, the inspection process is viewed as a quality improvement opportunity. Laboratories cited for deficiencies during inspection must correct them. Most laboratories readily comply, but a small percentage chooses not to rectify deficiencies. These laboratories are subject to sanctions on re-inspection. The most severe sanction is revocation of the CLIA-certificate. One of the most subjective of the CLIA88 requirements is that the laboratory director must assure that the quality (accuracy, precision, long term stability, sensitivity, specificity, etc.) of test results meets the physicians needs. This makes the laboratory an active participant in patient care. A recent study suggests that US medical care, including the laboratory components, may be falling short of its quality goal.

3. Results and discussion Tables 2a 2c summarize PT participant performance data for three events in 1994, one event in 1997 through 1999, and three PT events in 2002. The Wisconsin State Laboratory of Hygiene PT Program is one of 14 CLIA-approved programs [7]. Participants are primarily from smaller hospital and physician office laboratories. Data for six chemistry analytes were selected because of the highest incidence of PT failures. The PT data, spanning 8 years, show that laboratory performance, evidenced by a decreased percentage of failures, has improved and indicates that laboratories can and do correct problems. The mean failure rate for the six analytes over this 8-year period has decreased roughly by a factor of 3. Our studies provide some insight into intra-laboratory performance requirements necessary to pass PT [10 12]. Through computer modeling techniques and direct statistical calculations, we demonstrated that the

40

S.S. Ehrmeyer, R.H. Laessig / Clinica Chimica Acta 346 (2004) 3743 Table 2c Participant failures (%) in Wisconsin PT program2002 PT event #1 % Albumin Calcium Cholesterol Glucose Sodium Urea nitrogen 2 1 4 8 3 3 N 301 333 436 502 375 415 PT event #2 % 1 1 4 5 4 3 N 302 222 428 500 372 413 PT event #3 % 5 2 3 4 5 4 N 304 337 417 497 374 413

Table 2a Participant failures (%) in Wisconsin PT program1994 PT event #1 % Albumin Calcium Cholesterol Glucose Sodium Urea nitrogen 9 10 15 12 17 14 N 223 314 1094 1099 479 851 PT event #2 % 7 8 12 7 18 4 N 223 314 1083 1080 476 768 PT event #3 % 3 5 11 8 16 5 N 232 323 1057 1060 481 738

Mean failure rate for these six analytes over 3 PT events = 10.1 F 4.6.

Mean failure rate for these six analytes over 3 PT events = 3.4 F 1.8.

PT criteria could be translated directly into total allowable error, or intra-laboratory performance specifications. These consist of maximum allowable coefficient of variation (CV) and bias combinations. Statistically, a laboratory with no bias (inaccuracy) and a day-to-day CV (imprecision) equal to one-third of the specified PT criterion has virtually a 100% chance of passing a 5-sample PT event for that analyte. For example, the PT criterion for both glucose and cholesterol is target value F 10%. A laboratory having a methodology with no bias and a CV that is 100% of the specified PT limit (i.e., an internal CV of 10% of the target value for either glucose or cholesterol) would fail PT (achieve < 80% correct results in one PT event) 51% of the time. This is tantamount to failing every other PT event. If the laboratory reduces its internal CV for glucose or cholesterol to 5% of the target value, or to one-half of the PT limit, the chance of a single event failure is dramatically reduced to about 2%, or one in 50 PT events. Most importantly, if the CV is reduced to one-third of the PT limit, the chance of achieving < 80% correct results in one PT event is virtually zero. This assumes that the laboraTable 2b Participant failures (%) in Wisconsin PT program1997 1999 PT event #1 (97) % Albumin Calcium Cholesterol Glucose Sodium Urea nitrogen 3 5 6 6 7 9 N 232 260 926 827 374 508 PT event #1 (98) % 2 4 5 4 8 7 N 219 245 690 679 356 469 PT event #1 (99) % 0 1 2 3 4 1 N 224 244 548 602 352 428

Mean failure rate for these six analytes over 3 PT events = 4.3 F 2.6.

tory is in statistical control, i.e., the method is functioning properly. It clearly delineates the apparent CLIA88 mandated view of what constitutes adequate performance. For regulatory purposes, CLIA88 defines passing PT as being successful in two out of three consecutive PT events. Our studies show that a laboratory having no bias will pass PT, long term, with a CV of approximately 40% of the specified CLIA88 limits. However, passing PT does not necessarily indicate that a laboratorys results are good enough to meet the needs of the physician. There are no data to support the assumption that a laboratory successfully passing PT is providing physicians with clinically relevant data. The PT criteria defining successful performance (Table 1) are open to question, since they are primarily based on 1985 1990 state-of-theart interlaboratory performance data. A the critical level of 5 mmol/l for cholesterol, we can assume that among laboratories passing PT, some are doing so with day-to-day CVs at or near 40% of the CLIA88 tolerance limit or 0.2 mmol/l. This also means that a cholesterol result of 5.0 mmol/l would be reported, 5% of the time, as >5.4 or < 4.6 mmol/l. The question is will data with this amount of inherent variation permit adequate diagnosis and treatment such as prescribing a cholesterol-lowering drug. As seen in Tables 2a 2c, the number of participants from 1994 to 2002 has significantly decreased. This reflects the fact that some test sites have changed PT providers while others have shifted from moderate and high complexity test methods to waived testing methodologies, which do not require participation in PT. Likewise, some poorer performing laboratories, those failing PT, have discontinued laboratory testing entirely. The latter case

S.S. Ehrmeyer, R.H. Laessig / Clinica Chimica Acta 346 (2004) 3743

41

could be regarded as a CLIA-induced improvement in the quality of testing. The percentages of test sites with CLIA88 inspection deficiencies are shown in Table 3. There is a decline in the percentage of laboratories cited for deficiencies over the last four inspection cycles (approximately 8 years). Before declaring the CLIA88 approach to ensuring quality through enforced compliance to be evidence of success, it must be noted that the number of laboratories inspected for CLIA88 compliance has dropped over this period from 37,000 in 1995 to 22,000 in 2002. The reason for the decline of 15,000 laboratories is uncertain. Some test sites undoubtedly stopped testing entirely while others have chosen to be inspected by other government-approved professional programs such as the Joint Commission on Accreditation of Healthcare Organizations, the College of American Pathologists, or COLA (formerly the Commission of Office Laboratory Testing). Still others, particularly physician office laboratories, have abandoned CLIA-classified moderate and high complexity test methodologies and now perform only CLIA-waived testing that does not require CLIA88 inspections. Today, the waived list of tests includes 72 analytes and 1410 test systems, which provide testing in most disciplines of laboratory medicine [7,13]. As a result, analytes such as hemoglobin Alc, HDL cholesterol and H. pylori
Table 3 Most frequently cited CLIA inspection deficiencies (%) [7] Deficiencies f 1995 1996a f 1997 1998 27% 12% f 1999 2000 24% 15% f 2001 2002b 18% 6%

Table 4 Percent (%) of CLIA-inspected laboratories issued sanctions in 1996 and 2002 for failure to correct deficiencies [7] Sanctions CLIA certificate suspensions, limitations or revocations Test sites fined or given directed plans of corrections
a b

1996a 0.19%

2002b 0.36%

0.01%

0.13%

Approximately 37,000 CLIA inspected test sites. Approximately 22,000 CLIA inspected test sites.

Failure to 30% run QC Failure to 13% follow manufacturers directions No quality 7% assurance program 6% Failure of laboratory director to meet responsibilities
a b

7%

9%

3%

8%

9%

2%

Approximately 37,000 CLIA inspected test sites. Approximately 22,000 CLIA inspected test sites.

are available to test sites that want to provide testing services, but do not wish to be CLIA inspected or participate in mandatory PT. Government sanctions imposed on laboratories not correcting identified CLIA88 deficiencies are summarized in Table 4. These data are difficult to interpret because of the very small percentage of laboratories sanctioned for repeated violations. The 1996 2002 increase in the frequency of citations is possibly due to fewer laboratories surveyed (37,000 versus 22,000) and to inspectors becoming more willing to issue sanctions. In the inspectors view most test sites have 11 years experience with the CLIA88 requirements and should be in compliance. The last indicator considered was determining whether the quality of test results, as a result of universal adoption of CLIA88 requirements, better meet customers (physicians and patients) needs. According to the 2000 U.S. Institute of Medicine (IOM) report, To Err is Human: Building a Safer Health System, between 44,000 and 98,000 hospitalized Americans die each year from a variety of preventable medical errors [14]. Some portion of these serious medical errors must be attributed, in full or in part, to failures in the laboratory testing process. Beginning with Belk and Sunderman, there is overwhelming data to support the idea that problems in laboratory testing are not due to a failure to appreciate the need for quality test results. Rather, the critical concern is how to better manage the testing process to ensure that the appropriate level of quality is achieved. Simply mandating, by regulation, specific quality practices such as those in CLIA88 does not appear to be the entire answer. The IOM study recommends several general solutions for building a better and safer health care

42

S.S. Ehrmeyer, R.H. Laessig / Clinica Chimica Acta 346 (2004) 3743

system. It specifically recommends incorporating safeguards that mandate the use of quality processes to ensure quality and reduce the likelihood of human err. This can apply fully to the testing process governed by the CLIA88 regulations. Conceptually, CLIA88 envisioned a unique, total quality management partnership among regulators, manufacturers, and laboratorians to achieve quality laboratory testing [15]. In the 11-years post CLIA, we have witnessed the evolution of test systems by which, to a lesser or greater extent, manufacturers have taken the testing process out of the hands of testing personnel and have produced instruments capable of delivering assured quality. Examples exist of systems with internal liquid calibrators and controls that use sophisticated analytical algorithms. These systems perform the calibration, but also run the controls, assess performance, rectify problems and, then and only then, release patient test results [16]. Certainly this new generation of instruments is in part due to the manufacturers desire to sell more testing systems by helping customers meet CLIA88 quality requirements.

Other laboratories do benefit from the guidance provided by the CLIA88 regulations. At a minimum, there is strong suggestive evidence that bad laboratories are selectively pushed out of the testing business or, as an alternate hypothesis, have become better performers. The small numbers of government imposed sanctions, one-tenth of one percent; speak to the laboratories overwhelming commitment to do good quality laboratory work. While our overall assessment is that compliance to CLIA88, has improved US clinical laboratory performance, we as scientists must distinguish what we know from what we think (assume) we know. References
[1] Public Law 100 578, Section 353 Public Health Service Act (42 U.S.C. 263a) October 31, 1988. U.S. Department of Health and Human Services. [2] Medicare. Medicaid and CLIA programs: regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA). Final rule. Fed Reg 1992;57:7002 186. [3] U.S. Department of health and Human Services. Medicare, medicaid and CLIA programs; laboratory requirements relating to quality systems and certain personnel qualifications; final rule. Fed Reg 2003;68:3640 714. [4] URL: http://www.iso.ch/. [5] U.S. Department of Health. Education and Welfare, Public Health Service, Center for Disease Control, 1967: Clinical Laboratories Improvement Act of 1967. Part F of Title III of the Public Health Service Act. Sec. 353. 68. Atlanta: CDC; 1967. [6] U.S. Department of Health, Education and Welfare, Social Security Administration: Federal Health Insurance for the Aged Regulations: Conditions for Coverage of Service of Independent Laboratories. Publ. No. HIR-13. 68. Washington: DHEW; 1968. [7] http://www.cms.gov/clia/. [8] Belk W, Sunderman W. A survey of the accuracy of chemical analyses in clinical laboratories. Am J Clin Pathol 1947;17: 853 61. [9] Ross J. Evolution of evaluation criteria in the College of American Pathologists Surveys. Arch Pathol Lab Med 1988;112: 334 9. [10] Laessig R, Ehrmeyer S. Use of computer modeling to predict the magnitude of intralaboratory error tolerated by proposed CDC interlaboratory proficiency testing performance criteria. Clin Chem 1988;34:1849 53. [11] Ehrmeyer S, Laessig R, Leinweber J, Oryall J. 1990 Medicare/ CLIA final rules for proficiency testing: minimum intralaboratory performance characteristics (CV and bias) needed to pass. Clin Chem 1990;36:1736 40. [12] Laessig R, Ehrmeyer S, Leinweber J. Health care financing administrations new proficiency testing rules: characterizing

4. Conclusions In the last analysis, the fundamental question remains, Do fewer PT failures, fewer inspection deficiencies, and a very limited number of government sanctions mean that the quality of patient test results has improved? Intuitively we all believe that adhering to the CLIA88-mandated, good laboratory practices and passing PT improves laboratory test quality. The indicators assessed in this study generally support this conclusion. However, the direct link between regulatory compliance and quality test results is difficult to prove. Clearly, the decision to make PT the cornerstone of CLIA88 compliance has been a good one. The success of PT as a quality tool has been demonstrated by over 50 years of experience. A laboratorys motivation may not reflect a true quality goal in the total quality management or ISO spirit though the outcome is desirable. Adoption of good laboratory practices and successful PT performance, if only to achieve and retain CLIAcertification, most certainly improves test quality.

S.S. Ehrmeyer, R.H. Laessig / Clinica Chimica Acta 346 (2004) 3743 long term performance as successful, probation and suspended. Arch Pathol Lab Med 1992;116:770 6. [13] Pontius C. Committee tackles agenda. Med Lab Obs 2003; 35:26. [14] Committee on Quality of Health Care in America. Institute of Medicine. In: Kohn L, Corrigan J, Donaldson M, editors. To Err is Human: Building a Safer Health system. Washington, DC: National Academy Press; 2000, p. 1 287.

43

[15] Laessig R, Ehrmeyer S. Quality: the next six months. Clin Chem 1997;43(5):903 7. [16] Fallon K, Ehrmeyer S, Laessig R, Mansouri S, Ancy J. From quality control and quality assurance to assured quality. Point Care 2003;2(3):188 94.