A Self-instructional Course for

Laboratory Professionals.
P.A.C.E. Approved
Contact Hours: 3

Advanced Applications in
Clinical Laboratory Quality Control
Skill Level: Advanced
P.A.C.E. Approved Workbook

Strategies for Appropriate Laboratory QC Design

Written by Sten Westgard MS

 Dear Reader,

This is a digital document. For ease of navigation links are provided from the contents
page. To return to the content page from anywhere in the document, simply click on a page
number.

© 2021 Techno-path Manufacturing Limited. All rights reserved. No part of this publication may be reproduced or transmitted in
any form or by any means, electronic or mechanical, including photocopying, recording, scanning or otherwise, or through any
information browsing, storage or retrieval system, without permission in writing from Techno-path Manufacturing Limited. All
content correct at time of publishing. E &O.E.
 INTRODUCTION 3

Contents
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Goals for this workbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Selecting the appropriate control material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Multi-analyte controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Number of levels of QC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Medically-relevant levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Analytical measuring range (AMR) vs. QC levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Shelf life of control materials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Open vial stability (OVS). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

QC Frequency – how often should you run controls. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Commutability, matrix, and matrix effects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Starting your QC: Quality Control Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

How to set ranges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Longer term: switching to a new lot of QC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Multiple instruments: do we need multiple means or a single mean?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Frequent Questions about QC Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Westgard Rules: When to use them and why . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Checking on your QC: Beyond the dot-to-dot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Even more advanced statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Analytical Sigma-metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Allowable Total Error (TEa). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Crunch the numbers! An actual analytical Sigma-metric example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

How often should you calculate the Sigma-metric? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Reference Change Value (RCV). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Measurement Uncertainty (MU). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Moving Averages and Exponentially Weighed Moving Average (EWMA). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Wrapping up. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Self Assessment Quiz - Answer Key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Glossary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

P.A.C.E Continuing Education Assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Evaluation of Advanced QC Workbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

4 INTRODUCTION

Introduction
Congratulations! You’ve advanced to the next As you are becoming more acquainted with your
level of quality control. Perhaps you didn’t realize professional environment, this advanced practice
that there was an advanced level, or even a basic may actually be easier for you to learn.
level, but this workbook is going to help you
elevate your QC game. If, at any time, the material in this workbook
seems too advanced or too overwhelming,
The good news is that using the more advanced remember there is a Basic QC workbook that is
quality control techniques will reduce your work, also available. You can always refer to that basic
not increase it. You can invest a bit of time workbook whenever a concept or technique
assessing your performance, and then customize seems a little difficult. In this Advanced QC
your QC practice so you aren’t running all rules workbook, we won’t redefine the key concepts of
on all tests and running around to troubleshoot Quality Control like mean, SD, CV, etc. We will
every test. Instead, you focus on the right rules assume you know them and move on with the
and the right number of controls for each test best way of leveraging those concepts into better
– in support of delivering the right result to the QC practices.
clinician and patient.
The purpose of this book is to CHANGE how
If you’ve been working in the laboratory a long you do QC, so you do it more efficiently in your
time, this workbook will challenge some of your laboratory, do it more easily for your staff and
long-held beliefs about QC. We will, in fact, try to colleagues, and do it more effectively so patients
help you “unlearn” your bad habits. If you’re new are safer from the possibility of erroneous results.
to the laboratory, thank goodness you’ve joined
the profession; we need you and all of your friends. So, let’s begin, shall we?

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 GOALS FOR THIS WORKBOOK 5

Goals for this workbook

• Setting appropriate QC design strategies exam to get continuing education/contact hours.

(selecting appropriate QC product/levels, QC
frequency, QC ranges, rules, etc.) Also, throughout this book we will be quoting from
various references, standards, and regulations.
• Select the right Westgard Rules or Westgard These are the parts that will be the most difficult
Sigma Rules for your tests to read, since the implicit aim of such documents is
to be as scientific, precise, and boring as possible.
• Assess your QC on a broader scale by Along with every official definition, we provide a
calculating the Analytical Sigma-metric of “real world” definition to preserve your sanity.
your tests and how to apply to your analytes’
QC design

Throughout the book, we provide you with self-

assessment quizzes, with answers you can check.
At the end of this process, you can take a final

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

6 SELECTING THE APPROPRIATE CONTROL MATERIAL

Selecting the appropriate control material

While the basic QC workbook discussed some and monitor, selecting the right control material
of the elements of good quality control material, is trickier – and more critical – than ever. We
there are more aspects we will discuss here. Given describe below the various attributes of quality
all the tests that a modern laboratory must run control material of which you must be aware.

Multi-analyte controls

Decades ago, in the early days of quality control, run any patient samples!
controls were built one analyte at a time. For each
test, you had a control dedicated to monitor it. It’s more efficient in cost and effort to have a single

Today’s modern laboratories, however, cannot control material that covers as many analytes as

afford to run a single control for every test. We possible.

wouldn’t have enough room on our instrument to

Number of levels of QC

How many levels of QC material are really they should also consider: the clinical need. Think
needed? This is a common question that many of the regulations as the minimum: you MUST run
laboratorians struggle to answer. Labs are caught at least two levels of controls for most tests, and
between the regulatory mandates and their for some select tests, you must run three levels.
financial constraints, but there is a third factor But think also about where along the analytical

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 SELECTING THE APPROPRIATE CONTROL MATERIAL 7

range the important medical decisions get made one and level three of the control, representing
is that in only places or are there areas where the sub therapeutic level at 5 ug/mL and the toxic
decisions happen? level at 60 ug/mL. If the calibration curve drops
out in the middle area of the curve between 8
Another reason to consider additional levels of and 58 ug/mL (see Figure 2 below), it is plausible
control material is when the assay is non-linear. that the level one and level three controls would
When performing a non-linear test, you want to still be “in” range. Since they are not running a
ensure that key portions of the curve are being control at the middle therapeutic level, they are
monitored for errors. For example, the curve below missing the aberration in the calibration curve.
(Figure 1 is the non-linear curve for phenobarbital. This could cause the lab to release patient test
results falsely representing the test result as a
Let us assume in this case, the lab uses only level

Figure 1

With drop in
calibration curve,
cause patient
samples and QC to
recover low or sub
therapeutic.

Figure 2

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

8 SELECTING THE APPROPRIATE CONTROL MATERIAL

sub therapeutic value when, in actuality, it is could subsequently move the patient into the
therapeutic. The physician in seeing the result as toxic range.
sub therapeutic may increase the dosage which

Medically-relevant levels

Equally important as the number of analytes in a relevant values. The utility of this type of control
single control are the analyte values represented is limited at best. You want the control material
in the controls. It’s easy enough to manufacture values to run near the levels where medical
controls with very high and very low values, but decisions are being made.
those don’t necessarily represent medically-

Analytical measuring range (AMR) vs. QC levels

You may get confused about covering your linearity kits, but you don’t use control materials
working (or reportable) range of each test to do that. Target the most important medical
method with your control levels. Your QC values decisions, or cutoffs, or diagnosis guidelines, with
are more important for covering the medical- your control levels. Don’t worry if there aren’t
decision levels, not the entire reportable range. control levels at extreme levels or edges of your
Your reportable range is established – and when range unless there are critical patient values in
necessary, verified – using patient samples or those regions.

Shelf life of control materials

One of the best aspects of modern control Cross-over studies are not only time-consuming,
materials is the long shelf life which allows but they are also expensive. The more studies
laboratories to purchase supplies in scale and you must perform, the more test reagent you are
enjoy efficiencies in storage, logistics, and pricing. consuming. The cost of the control plus the test
For many controls, you can get a shelf life of a reagents can add up to hundreds, thousands,
year or longer. Now, bear in mind a long shelf even tens of thousands of dollars every year.
life isn’t a pure benefit – it’s only good if you will Thus, the longer the shelf life, the fewer cross-
consume all the materials within that time period. over studies are required which ultimately saves a
Make sure the volume you use will allow you to considerable amount of money for the laboratory.
consume all your control material in storage
within their expiration dates. There’s no benefit of
a long shelf life if you end up disposing it because
you don’t use it all. Make sure to base your QC
orders on the expected rate of consumption of
the control materials.

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 SELECTING THE APPROPRIATE CONTROL MATERIAL 9

Open vial stability (OVS)

There is a difference in the “shelf life” and the In other words, with shelf life and open vial
“open vial” life. Both are important. Open vial stability, the best combination involves not just
stability refers to how long after the vial is open the control manufacturer, but also the number
the control material is stable and produces of tests, volume, and patient decision levels of
optimal results before it deteriorates. Again, a the laboratory itself. A careful assessment of the
longer open vial life will give you more time to decision levels, number of tests, and QC frequency
utilize it. You don’t want to waste control material will allow you to optimize your order, so you get
because you can’t consume the contents in the maximum value out of your control materials.
vial before its open vial expiration.

QC Frequency – how often should you run controls?

As with the struggle over the number of levels Should a laboratory running hundreds of tests
of control to run, there are similar forces at play and a laboratory running tens of thousands of
when it comes to the frequency of running QC. tests have the same once-a-day QC frequency?
The regulations typically mandate a minimum of Any objective assessment of quality would argue
running two controls every 24 hours for many for running controls based more on the volume
tests. For some specific tests, the frequency of testing – and the quality of the assays and the
may be as often as running three controls every needs of the patients – rather than an arbitrary
eight hours. But when you read the regulations, frequency over a 24-hour period.
you quickly notice that there isn’t any guidance
on the rationale for once a day or three times a Westgard sigma rules also aid in determining the

day QC. These are simply arbitrary requirements, frequency of QC runs per day. See page 19 of this

representing a very bare minimum for compliance. workbook for more information.

Commutability, matrix, and matrix effects

Commutability is the ideal goal of any type of ISO:

control. That is, the control material mimics as “ability of a material to yield the same numerical
closely as possible a real patient sample. Good relationships between results of measurements
commutability means you can be confident that by a given set of measurement procedures,
a warning result of your control run adequately purporting to measure the same quantity, as those
alerts you that a potential instrument or test between the expectations of the relationships
reagent malfunction could plague the patient obtained when the same procedures are applied
samples too. to other relevant types of material” 1

Commutability, of course, has its own official Simply put, commutability is good, and it’s what
definition, established by the meteorologists of we seek in control materials.

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

10 SELECTING THE APPROPRIATE CONTROL MATERIAL

The opposite of commutability is often referred

to as a Matrix Effect (no, this is not the movie The
Notes
Matrix with Keanu Reeves, this is the really nerdish,
geeky, uncool Matrix). The matrix of a control is
all the extra stabilizers, preservatives, and other
ingredients present to support the analyte itself
but are wholly unrelated to a patient sample.

These additives may help keep the control

material stable, or have a longer shelf life, but
they do not make the control behave as a patient
sample. In the worst case, the matrix of a control
material will make the control behave differently
than a real patient sample. In the worst-case
scenario, this means a control results may be
“out”, but the analytic performance of patient
results are completely fine, and unaffected by
whatever is causing the control to be out. This
defeats the very purpose of a control; it becomes
an unreliable signal of whether patient results are
going to be reliable.

As much as possible, labs need to avoid controls

with heavily artificial matrices and need to have
controls that are as commutable as possible.
Inevitably, as labs desire controls with long shelf
lives and greater stability, the control materials
must be modified with a matrix that will make
it less like a real patient sample. Between our
financial constraints and our quest for best quality,
we must strike a balance.

It is important for the laboratory to seek a control

material that uses fresh, human-based materials.
While it is impossible to find a QC vendor that
produces a control that is 100% fresh human
serum or plasma with optimal shelf life and open
vial stability, the lab should look for a QC product
that mimics the patient samples as much as
possible.

In summary, commutability is a good thing, matrix

effect is a bad thing.

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 STARTING YOUR QC: QUALITY CONTROL DESIGN 11

Starting your QC: Quality Control Design

How to set ranges

Let’s start at the very beginning. You have an [Fun fact: The mean and SD in the package insert
absolutely new control – no previous lots, no are commonly determined by running the control
previous data, nothing cumulative from the past. on a number of instruments or by sending the
Today is the first day you’re running it. How control out to a select group of laboratories to run
do you set up your ranges? [By ranges, we are on their variety of instruments. So, the package
referring to the mean, SD, and the Levey-Jennings insert SD is the standard deviation of that group of
chart, by the way. “Range” is a shorthand for a lot instruments around the mean of that group. Since
of different aspects of the QC chart setup.] it comprises multiple instruments, even multiple
labs, the SD will be larger, possibly MUCH larger,
The best practice is to establish your own mean than any individual laboratory’s SD. Therefore, it
and SD over a period of 20 days for each level is important to switch to your own SD as soon
of control. Clearly, that means during the first 20 as possible after you have derived it from the
days of your new control, you’re waiting for the data you have captured. The longer you use the
mean and SD, but what are you going to use in package insert SD, the longer you are at risk of
the meantime? having your control limits too wide and missing
significant errors. Okay, that’s not so fun.]
This is where the package insert range is most
valuable. If you have an assayed control, there is So, once you have more complete data
a data sheet supplied, either in digital format or representing your control material’s performance
on old-fashioned paper, providing expected or on your method on your instrument in your labs,
target means as well as standard deviations. If start using that information. The best practice is
you have absolutely no other information, start by to set up each method and each control level with
setting up your mean and SD with the information its own mean and its own SD.
from this package insert.
What if you want tighter ranges than offered in

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

12 STARTING YOUR QC: QUALITY CONTROL DESIGN

the package insert but you, don’t have 20 days to measurements), calculate the new SD.
collect data?
You now have the new mean and the new SD for
If you need to establish your mean and SD in a your control charts
shorter amount of time, you can consider running
multiple controls over a shorter number of Continue to update the SD when you have the next

days: run four control runs per day for five days. month of data, and the next, until the expiration

Whatever estimates you get from this sprint, of the control.

should be replaced with updated data as soon as

Caution: Many labs believe that if the shelf life of a
you’ve been running for a month.
control is very short, they do not need to perform

If you have a really short shelf-life control – this cross-over studies. Do not fall into that trap! You

often happens with hematology control material are required by regulation to establish ranges

– you may need to work with even shorter cross- on all lot numbers prior to use. This is not only a

over intervals. good lab practice, it’s essential risk management.

1. Establish a new mean for the new control More caution: If control shipments are delayed

with 8-10 values run over a few days. This is or sit on the loading dock during excessive hot

statistically sound – you can establish a new or cold weather, the only way to prevent that

mean with as few as eight values. damaged control from going into use is the cross-
over study. And from a regulatory perspective,
2. Pair up the new mean with the old CV, and then the laboratory must have proper documentation
calculate new mean x old CV = temporary SD of each new lot number on file for any surveyor
request.
3. As soon as you have enough data (about 20

Longer term: switching to a new lot of QC

If you are starting a new control lot of a control SD. Again, 20-days-worth of data is preferred.
material you’ve used recently, and you have
experience with and historical data for the [Fun fact: different control lots, like different test

previous lot, you can consider using that historical reagent lots, are never truly identical. They will

mean and SD as a bridge to the new lot. Assuming always be slightly different, but everyone hopes

these lots are manufactured with the goal of they are different in such minor ways that there

producing nearly identical performance, you can is no clinical significance. You get to judge what

use the historical mean and SD until you have constitutes a clinically significant difference.]

enough data to calculate the new mean and new

Multiple instruments: do we need multiple means or a single mean?

In the modern laboratory, it’s increasingly manufacturer. There is a temptation to assume

common to have several instruments of the same that all these instruments will have the same

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 STARTING YOUR QC: QUALITY CONTROL DESIGN 13

mean and same SD, and it certainly could be more SD, you will have to constantly monitor that the
convenient to run all the instruments with one set group of instruments maintain its “uniformity”.
of numbers. That means under the convenient unified mean and
SD, you must still maintain a log of the individual
However, we caution your lab to strongly consider mean and SDs of each test and instrument. So,
the risk to this option. The convenience of having you are doing more work than if you would simply
one set of numbers may be outweighed by the maintain the individual mean and SDs of each
risk that you will miss true outliers or that simple instrument. Is the superficial similarity worth the
instrument differences will present as false additional effort? This is ultimately the decision of
outliers. the laboratory professional.

Further, even if you move to a uniform mean and

Frequent Questions about QC Design

Uncomfortable question: What if your range emphasizes the use of patient specimens, not
doesn’t match the package insert range? control materials. If the patient specimens show
there is a significant change from lot-to-lot, that
Remember the range on your package insert is is grounds for concern about the new reagent
meant to be a guide. Your mean should fall within lot. If the patient specimens show no significant
the package insert range, but the range of values change, but the control mean has changed on the
represented by the SD and mean is not required new reagent lot – you should reassign the mean.
to fall within the package insert range. The patient analysis has shown you that the mean
of the control lot has changed, but it’s not a
If your mean is not within the package insert range,
clinically significant change.
you should examine your method, confirm that
you stored and processed the control correctly, Therefore, you want to use quality control material
etc. If you can eliminate any internal causes, then that is as commutable as possible. This will
contact the control manufacturer to ask if there prevent the occurrence of a QC shift vs. patient
are any other similar reports of the new control samples recovering the same when comparing
lot. If the control lot is found to be the source of data from the old test reagent lot to the new lot,
the problem, a replacement control lot should be prior to use.
sought.
Uncomfortable question: Should I use allowable
Uncomfortable question: Should I use my total error (TEa) to set my ranges?
controls to determine the acceptability of new
test reagent lots? Allowable total error is a quantity used in assessing
external quality assurance, proficiency testing,
Controls are often used to handle the evaluation and analytical Sigma-metrics. It is not directly
of new test reagent lots. It’s tempting to apply applied to a Levey-Jennings chart. It is not to be
the same protocol for lot-to-lot variations used as an SD or range setting. Do not even use
among control lots to the lot-to-lot variations a fraction of the allowable total error as an SD on
in reagent lots. But the latest advice from CLSI your Levey-Jennings chart. That is like confusing
EP26 on handling lot-to-lot variation for reagents

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

14 STARTING YOUR QC: QUALITY CONTROL DESIGN

the maximum speed listed on your speedometer need a new mean, possibly a new SD, to match
(the dial) as the recommended actual operating that new state.
speed (the needle). One is a performance
capability; the other is actual performance. You
want the QC chart to reflect how your laboratory Notes
is actually operating. And you don’t want to be
running at full speed – consuming every possible
component of operating error to the maximum.

Uncomfortable question: What do I do if I notice

that it appears that I have significantly different
performance in my historical data?

This may occur within the data collection timeframe

where there was a change in the test system that
caused different results to be generated. This
may be due to a shift with a new test reagent lot,
implementation of a new component to the test
system, operator technique, or some other issue.
Remember that whatever you calculate for your
current mean, it should only reflect the current
performance. If, in the past, there was significantly
different performance that does not reflect the
current state, that data should not be included in
your calculations.

Uncomfortable question: When adjusting our

ranges and mean, how much is too much?

How often should you adjust your mean? As often

as is necessary, but not more. That's easy to say,
but hard to determine when it is truly the right
time to make an adjustment. Let's begin by stating
that every time a peer group mean changes, that
doesn't mean an individual laboratory should
change their mean, too. Of course, when you have
a QC rule failure, the laboratory will troubleshoot
extensively to uncover the root cause of the
issue. If the error is significant (and definitely not
random, but truly systematic), once it is fixed,
the method is now in a new state. If your trouble-
shooting uncovers a serious problem, something
that needs to be replaced, etc., you're definitely
breaking from previous history. Therefore, we will

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 STARTING YOUR QC: QUALITY CONTROL DESIGN 15

Self-Pace Exercise # 1:
You have collected the following data in your 20-day cross-over study and
recorded the following supporting data.
Level 1 Level 1 Package Insert Package insert TEa
Instrument A Instrument B Mean Range +/- 3S

Mean 10.48 11.32 11.5 8.4 -14.6 -

SD 0.15 0.12 - - -
CV 1.43 1.06 - - -
N 20 20 - - -

How would you set your ranges using the following information?

1. Do both instrument values fall within the 3. Using the calculated mean and SD,
package insert range? what ranges would you apply to both
Yes instruments.
a. Instrument A: +/- 2 SD range:
No
2. Previous lot had the following cumulative ____________________________
information:
• Old lot mean for instrument A = 11.1, b. Instrument B: +/- 2 SD range:
Old lot SD = 0.15, old CV = 1.35 ____________________________
• Old lot mean for instrument B = 12.6,
old lot SD = 0.14, old CV = 1.11
a. What would you choose for
instrument A’s new control lot
mean based upon the table above?
b. What is instrument A’s SD for the
new lot using the CV of old lot?
c. What would you choose for
instrument B’s new control lot
mean based upon the table above?
d. What is instrument B’s SD for the
new lot using the CV of old lot?

Answer Key: Page 29

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

16 WESTGARD RULES: WHEN TO USE THEM AND WHY

Westgard Rules: When to use them and why

[Fun Fact: A majority of labs worldwide still use This z-score or SDI will calculate the positive
the 1:2s rule in their QC protocol. Many labs still or negative bias with each level of control as
use the 1:2s as their only QC rule. Despite half compared to the mean and SD. For example, run
a century of experience and all the statistical # 1 shows that level 1 has a z-score of -0.5. That
knowledge that informs us about ruinous false means if we looked at the Levey-Jennings plot for
rejection rates (nearly 10% false rejection rate if this level, the data would be plotted at 0.5 SDs
you run two controls; nearly 15% false rejection rate below the mean value. The level 2 control shows
if you run three levels of control!), this simplistic the same negative bias, but the point is plotted
approach to QC has endured as a tradition at the on its Levey-Jennings chart 0.36 below the mean.
bench level. Again, this is not-so-fun fact is a
practice that creates a lot of extra, unnecessary In the Basic QC workbook, you can find very

effort and frustration in the lab.] detailed discussions of the rationale and evolution
of Westgard Rules.
The Westgard Rules were introduced back in 1981
as a solution to the 1:2s rejection rule, and a way Westgard Rules are a useful tool to maximize

to reduce false rejections without compromising error detection while minimizing false rejection.

error detection. It converted the 1:2s from a Each of the rules can help you determine whether

rejection rule into a warning rule. Modern versions the error you are experiencing is a random or a

of Westgard Rules have eliminated even the 1:2s systematic error.

warning rule. The latest advice is, “Don’t wait for

The more advanced use of Westgard Rules is
a warning to sound. Interpret the rejection rules
to utilize Six Sigma and each assay’s analytical
directly with every control result.”
Sigma-metric to determine that appropriate

The table opposite represents 30 runs of a bi- Westgard rule. This advanced usage is known as

level quality control product. The z-score is Westgard Sigma Rule.

another term for Standard Deviation Index (SDI).

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 WESTGARD RULES: WHEN TO USE THEM AND WHY 17

Self-Pace Exercise # 2: Self-Pace Exercise # 3:

Record any rule interpretation based upon the Below are two Levey-Jennings charts for
Westgard rule violated and whether you would two levels of a control product for a certain
hold patient results. The following Westgard test. The graph from each day’s QC result
Rules should be used for this exercise: 12s is plotted based upon the SDI or z-score
(warning), 22s (reject), 13s (reject), 10x value. Based upon these two charts, please
(warning), and R4s (warning). note what day’s QC results failed any of the
Westgard Rules:

Values
Level 1 Level 2 Rule 1:2s:
Z Score Z Score Interpretation
2:2s:
Run 1 -0.5 -0.36
Run 2 0.33 -3.21
1:3s:
Run 3 -1.8 0.4 10x within run:
Run 4 2.33 -0.2 R:4s:
Run 5 2.16 1
Run 6 -0.33 -0.2
Run 7 -0.1 -0.2
Run 8 2.33 -2.8
Run 9 0 0.2
Run 10 0.67 1.2
Run 11 -1.83 2.2
Run 12 0.67 -1.6
Run 13 -1.1 -1.4
Run 14 -1 -1
Run 15 -0.33 1
Run 16 1.83 -0.4
Run 17 1.33 -0.6
Run 18 -0.67 -0.8
Run 19 0.33 -1
Run 20 0.67 -0.2
Run 21 1.33 -0.8
Run 22 -0.33 -0.2
Run 23 -1.33 -1
Run 24 0.67 0.8
Run 25 -1 -0.8
Run 26 0.4 0.2
Run 27 -0.6 -1
Run 28 0.3 1.2
Run 29 -1.33 -0.4
Run 30 -1 0

Answer Key: Page 29

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

18 WESTGARD RULES: WHEN TO USE THEM AND WHY

A discussion of how to calculate the analytical Westgard Rules, until at Three-Sigma, you need
Sigma-metric of an analyte is included later in this all the Westgard Rules and need to increase the
workbook. number of control measurements you are making.

If you can determine the analytical Sigma-metric What does this mean? Labs with excellent
of your analyte, you can also determine how many performance can reduce the number of rules and
Westgard Rules are necessary to properly monitor controls they use, which will reduce the number of
the test. For a Six-Sigma test, you don’t truly out-of-control events they have to trouble-shoot.
need multiple rules – you can sufficiently monitor This can reduce both test reagent and control
your test with just 1:3s rule and two control levels. costs as well as labor cost.
As your sigma-metric is lower, you need more

Checking on your QC: Beyond the dot-to-dot

While your bench-level specialists and • Review the observed mean, observed SD and
technologists are in charge of reviewing each CV of the entire month.
new control point and thus, each new dot on
the Levey-Jennings chart, there is a higher-level » How does it compare to the package

review which you or someone you designate insert mean and SD? (Hint: Your mean

should conduct periodically. Typically, there is a should still be within the package insert

monthly QC review, where additional actions are range)

evaluated. » How does it compare to peer group mean

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 WESTGARD RULES: WHEN TO USE THEM AND WHY 19

and peer group SD? (Hint: your individual » Has the peer group report shown any
SD should be smaller than the peer group changes in performance?
SD)
» Has the EQA/PT report shown any
• Review Levey-Jennings charts to evaluate changes in performance?
outliers, shifts, trends.
These are just some of the things a QC Review
should consider. For problematic tests that
» Did your staff miss any random or
generate an outlier more than once a week, you
systematic errors?
shouldn’t wait a whole month before conducting
» Are there some errors occurring more this kind of review.
frequently this month?
The main thing you are doing with this review
» Review the bench-level actions.
is trying to look at the bigger picture, trying
» Is everyone running QC when they are to connect other events in the laboratory with
supposed to? any changes you’ve seen in QC, and to identify
» Are some technologists repeating and patterns over a longer time-frame than just the
repeating controls to get them in range? run-to-run, dot-to-dot perspective.

» Are all troubleshooting actions being While this review can be conducted by section
logged? heads or QC specialists, the summary should at
» Are the technologists interpreting rule least be reviewed by the Director and signed each
violations correctly? month.

» Are the technologists troubleshooting

correctly?

• Are multiple instruments of the same

manufacturer still performing similarly?

» Have you run split patient specimens this

month to check this performance?

• Split patient samples are aliquots of a

single patient specimen run on all the
similar instruments.

» Are individual means, SDs and CVs still

similar?

• Review the instrument history.

» Has maintenance been performed in the

last month?

» Have any calibrators, reagents, or parts

been changed?

» Are there any alerts from the manufacturer?

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

20 WESTGARD RULES: WHEN TO USE THEM AND WHY

Self-Pace Exercise # 4: Notes

Using data from example one above, review the actions taken by
the bench technologist for troubleshooting as noted below.
Remember, the following Westgard Rules were used for this
exercise: 12s (warning), 22s (reject), 13s (reject), 10x (warning),
and R4s (warning). Are these actions correct?

Level 1 Level 2 Technologist Action

Values
Z Score Z Score Action Correct?

Run 1 -0.5 -0.36 Accepted

Run 2 0.33 -3.21 Repeat L2
Run 3 -1.8 0.4 Accepted
Run 4 2.33 -0.2 Repeat L1
Run 5 2.16 1 Accept
Run 6 -0.33 -0.2 Accept
Run 7 -0.1 -0.2 Accept
Run 8 2.33 -2.8 Accept
Run 9 0 0.2 Accept
Run 10 0.67 1.2 Accept
Run 11 -1.83 2.2 Accept
Run 12 0.67 -1.6 Repeat L2
Run 13 -1.1 -1.4 Accept
Run 14 -1 -1 Accept
Run 15 -0.33 1 Accept
Run 16 1.83 -0.4 Accept
Run 17 1.33 -0.6 Accept
Run 18 -0.67 -0.8 Accept
Run 19 0.33 -1 Accept
Run 20 0.67 -0.2 Accept
Run 21 1.33 -0.8 Accept
Run 22 -0.33 -0.2 Accept
Run 23 -1.33 -1 Accept
Run 24 0.67 0.8 Accept
Run 25 -1 -0.8 Accept
Run 26 0.4 0.2 Accept
Run 27 -0.6 -1 Accept
Run 28 0.3 1.2 Accept
Run 29 -1.33 -0.4 Accept
Run 30 -1 0 Accept

Answer Key: Page 29

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 EVEN MORE ADVANCED STATISTICS 21

Even more advanced statistics

(Oh no, do we really need to know these things?)

Analytical Sigma-metric

This concept has been mentioned before in both That’s really good. When the creators of Six
the Basic QC workbook and in this Advanced Sigma reached it in their processes, they found
QC workbook. This is an application of the Six they had very happy customers, a very reliable
Sigma theory which has been used successfully product, and a very efficient profitable operation.
in industry, business, and healthcare for decades. Labs implementing Sigma-metrics have found
But we’re going to skip all the traditional Six Sigma the same thing: reduced effort, reduced errors,
theory – we don’t have space to describe it here, reduced expenses, all while delivering better,
and you don’t have the time to learn about 40 more effective, more accurate, and more timely
years of quality management evolution – so don’t results to patients.
worry about learning about green belts, black
belts, and master black belts. We’re skipping Analytical Sigma-metric Equation
the belts and some of the more colorful (read:
extraneous) elements of Six Sigma. We’re going (TEa - Bias)
Sigma-metric =
to focus on the core idea: quantifying, identifying, CV
and eliminating defects in your processes.
Where:
Six Sigma is really just a scale for benchmarking.
TEa = Total Allowable Error
If you have a Six Sigma process, you’re getting
just under four defects-per-million outcomes of Bias = inaccuracy
a process – or less than four false positives, false
negatives, or errors-per-million reportable results. all variables expressed in %

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

22 EVEN MORE ADVANCED STATISTICS

For the laboratory, the analytical Sigma-metric is inaccuracy.

built on three variables:
You may quickly notice that two of the three
• Allowable Total Error (TEa): this is a variables are already routinely collected by
performance specification, or, if you prefer your lab. You run controls at least daily, so
other words – a quality goal. This is how good you’re already determining your imprecision
your test should be, and when it isn’t, it means (your CV). You are participating in an external
you are in danger of producing false positives quality assurance/proficiency testing (EQA/PT)
and false negatives. program, or in a peer group, or you’re comparing
your observed mean against the control package
• CV: We’ve gone over this before. It’s your insert mean, so you have a way to determine your
observed imprecision. bias. Sigma-metrics simply leverages your data
on the observed performance of your method
• Bias: We’ve also gone over this before. It’s
and places it into the context of a universal scale.
your trueness, or if you like older terms, your

Allowable Total Error (TEa)

The TEa is the trickier part, but it’s really not organized the group of scientists who compiled
that hard. For labs in the US, CLIA has directly a global database of biological variation. Dr.
specified the allowable total error for more than Ricos led the group until about 2014. The theory
80 analytes in the PT criteria. The same goals that behind this database is that by assimilating all
you are using to judge whether you are going to the information we know about within-subject
pass or fail your EQA/PT are TEa goals. So, there and within-group biological variation for any
are goals for analytes in the CAP survey, in EQA/ analyte, we can scientifically determine how
PT programs all over the world. All you have to good any measurement of that analyte should
do is look them up – and most of them can be be (simply put, our analytical methods should
found easily online. have only a fraction of the biological variation for
each analyte, otherwise we’re adding more noise
When you have a choice of TEa goal – not all rather than generating a signal). The good news
EQA/PT programs agree about the performance is that the Ricos 2014 database is available for
specifications they set – you want to evaluate free online and covers more than 350 analytes.
the needs of your patients to choose the most A new database organized by the European
appropriate goal. Don’t automatically choose the Federation of Clinical Chemistry and Laboratory
biggest or the smallest – that may not be right Medicine (EFLM) is also available, which covers,
for your patients. Industry leaders have tried at the time of this booklet printing, about 80
to sort through the different resources to find analytes. The bad news about all the biological
challenging, but not impossible goals. variation data, regardless of the source, is that for
some analytes, the performance specifications
You may be familiar with a set of allowable total
are impossible to achieve. No method on the
errors derived from biological variation. These
market currently can hit the electrolyte goals
were informally known as the “Ricos goals” in
for performance as derived from EFLM or Ricos
honor of Dr. Carmen Ricos of Spain who originally
databases. The EFLM has recognized that even

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 EVEN MORE ADVANCED STATISTICS 23

as they update and expand their biological the term Total Error, which is related to but not
variation database, that may only reveal that the same as Allowable total error (TEa). Total
there are analytes where the biologically derived Error was one of the first estimates of combined
performance specifications are impossible to hit imprecision and inaccuracy, first published in the
by any of today’s diagnostic manufacturers. So, 1970s. Approximately, Bias + 2 * SD, it represents
don’t simply choose all biological goals to set for a worst case scenario of test results. Decades
your analytical performance. ago, you compared your Total Error against
your Allowable Total Error, and as long as the
The other good news hidden in there is that former was smaller than the latter, you were fine.
labs do not have to conduct their own biologic However, since the turn of the century, the Six
variation studies to set their performance goals. Sigma approach is more appropriate for today’s
Evaluate the current biologic goals, see if they laboratories and their quality management. Given
are appropriate, and if they aren’t, look to other today’s higher volume and greater accuracy
sources of performance specifications. needs of patient care, using the analytical Sigma-
metric is a better way to manage your QC.
Stepping back a bit, you may also be familiar with

Crunch the numbers! An actual analytical Sigma-metric example.

Let’s take the following ALT data.

Analyte: Alanine Aminotransferase (ALT), U/L

Monthly Lab Stats Peer Stats
Level Mean SD %CV N Peers Mean SD %CV N Peers
Test System
Level 1 28.40 1.199 4.22 335 64 28.23 1.364 4.83 2432 77
Peer

The laboratory’s control at level 1 has a monthly The TEa for this analyte is 25%.
mean of 28.4, compared to a peer group monthly
mean of 28.23. That control is showing a difference Now we have all the elements necessary to

of calculate an analytical Sigma-metric.

(28.4-28.23)/28.23 = 0.17 / 28.23 = 0.006 or Sigma-metric = (TEa – bias%) / CV

0.6% bias. = (25 - 0.6) / 4.22

= 24.4 / 4.22
[Note if the bias had been negative, we would still = 5.78
have converted it into an absolute value. Sigma-
ALT, at this decision level and at this time, is
metrics always plan for the worst-case and does
performing at an excellent level on the Six Sigma
not assume that in some situations a bias will
scale since the calculated sigma is 5.78 and
cancel out some of the imprecision or vice versa.]
therefore rounded to 6.0. A six sigma result is
The laboratory’s monthly imprecision of that considered world-class.
control is 4.22%.

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

24 EVEN MORE ADVANCED STATISTICS

Now that you have your Sigma-metric, what

next?

The Sigma-metric allows you to make design

choices for your QC rules and frequency. The
Westgard Sigma Rules are the easiest tools to
convert a Sigma-metric into actions you can take
in your laboratory.

Self-Pace Exercise # 5:
QC rules: 12s warning, 22s run rejection, 22s run rejection, 13s run rejection, 10x run rejection.
Situation # 1:
Sigma score is 7.2. Based upon the action above, which Westgard rule(s) could you implement?
__________________________
Situation #2:
Now, let’s assume that the sigma score is 4.1. Which Westgard rule(s) could you implement?
__________________________
Sigma scores can be used when comparing like instruments and tests.
Analyte Instrument Level 1 Level 3

Mean SD % Peer % % Sigma Sigma Mean SD % Peer % % Sigma Sigma

CV Avg bias TEa Calc Score CV Avg bias TEa Calc Score
mean mean

ALT 1 28.40 1.20 4.22 28.23 0.6 25 5.78 5 249.8 3.2 1.28 240.92 3.69 10 4.92 4

ALT 2 28.38 1.17 4.12 28.23 0.52 25 5.98 5 249.7 3.3 1.32 240.92 3.64 10 4.81 4

When we compare the performance of Aspartate Aminotransferase (AST) for both instruments,
we see the following data:
Analyte Instrument Level 1 Level 3

Mean SD % Peer % % Sigma Sigma Mean SD % Peer % % Sigma Sigma

CV Avg bias TEa Calc Score CV Avg bias TEa Calc Score
mean mean

AST 1 36.83 0.75 2.04 36.4 1.2 20 9.2 6 273 0.89 0.33 267.46 2.07 20 54.72 6

AST 2 36.23 0.44 1.21 36.4 0.45 20 16.15 6 266 1.41 0.53 267.46 0.54 20 36.59 6

Notice how the two instrument’s ALT results generate similar calculated sigma scores for both
levels of QC. Also, notice, that you can apply different TEa limits for different levels of QC.
Answer Key: Page 29

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 EVEN MORE ADVANCED STATISTICS 25

How often should you calculate the Sigma-metric?

Once you know the (relatively) easy way of even need to change your QC rules for years at a
calculating the Sigma-metric, you may be time if you are so fortunate to have a world -class
tempted to calculate it every day. Your software assay or instrument.
may even be able to update the Sigma-metric
with every new control data point. But don’t That said, if some major event happens – a

chase the metric every day. It’s most useful to replacement of the internal mechanics, a new lot of

re-visit the Sigma-metric about once a month, or reagent, a serious error condition, etc. – common-

even quarterly. Particularly when you have high sense dictates that we should re-establish our

Sigma-metric performance, you will see steady mean, our SD, and our Sigma-metric.

performance and you won’t be changing your

QC rules every day, week, or month. You may not

Reference Change Value (RCV)

This is a statistic that is useful not in the

Reference Change Value
management of QC, but in the interpretation
of patient results. It is sometimes called the
RCV = 2*Z* CVa2 * CVi2
Critical Difference. It represents the smallest
change between two serial patient results that is
nevertheless a clinically important difference. This Where:
formula can be seen immediately below.
Z value of 1.65 to represent 95% probability
When comparing two results on the same patient,
CVa = analytical imprecision
if the difference between them is less than the
RCV, there is no reason to believe anything CVi = within-subject biological variation
has changed clinically with the patient. If the
difference between the two results is greater than RCV may also be used when monitoring delta
the RCV, then the clinician can have confidence checks in your laboratory as a quality assurance
that something clinical has changed in the patient. monitor. Delta checks are useful for detecting pre-
The RCV is useful for helping clinicians focus on analytical (specimen identification or specimen
the most important test results. integrity errors), analytical errors, and post-
analytical errors. For example,
But again, this is a reporting and interpretation
A delta check flag occurred for ALT when
tool, not a tool to determine the acceptability of
comparing the first patient result 12 hours
the method or to monitor the analytical state. It
ago as 19 IU/L to their most recent value of 25
is listed here mainly because many use this as a
IU/L. Is this a true medically-relevant change
quality assurance check on the test system.
in the patient or just the noise of day to day
measurement? Given the within-subject

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

26 EVEN MORE ADVANCED STATISTICS

biological variation of ALT, according to the Interpretation of RCV:

Ricos 2014 database, is 19.4%, and also given
that the ALT method has an 8% imprecision, Percent change of ALT result from

the following calculation for RCV can be

19 to 25 = (25-19)/19 = 6/19 = 31.57%
performed:
RCV = ( 2) x 1.65 x ( CVimprecision2 + With the test change of 31.57% and because
CVindividualbiologicalvariation ) 2 31.57% is less than the RCV of 48.82%, the change
= 1.41 x 1.65 x (82 + 19.42) in values may simply be a result of analytical
= 1.41 x 1.65 x (64 +376.36) imprecision and within-subject biological
= 1.41 x 1.65 x (440.36) variation.
= 1.41 x 1.65 x 20.98
= 48.82 %

Measurement Uncertainty (MU)

This is a much-debated statistic, and paradoxically, At its simplest, MU is your intermediate

it is basically unknown within the US, while imprecision, and your expanded uncertainty is
outside the US it is a mandated requirement. ISO 2*CV.
15189 and all derivative accreditation guidelines,
requirements, and regulations mandate that There are more texts and publications that take

every method should determine MU, and report MU very seriously, but these are beyond the scope

it to clinicians. In practice, labs often comply of this workbook. If you are governed by an ISO

by calculating MU, showing it to the inspector, 15189 standard, you need to calculate MU, but you

and possibly never doing another thing. It’s a only have to satisfy the inspector. If you find the

statistic more honored in the breach than in statistic mystifying or unhelpful, don’t use it any

the observance. Labs generate it, as they are further than compliance.

required to, but mostly ignore it. A few labs

enjoy the additional rigor of calculating and using
measurement uncertainty. There are many ways
to calculate measurement uncertainty, and it
seems more methods to calculate are published
every day.

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 MOVING AVERAGES & EXPONENTIALLY WEIGHED MOVING AVERAGE (EWMA) 27

Moving Averages and Exponentially

Weighed Moving Average (EWMA)

There are still more patient-based techniques that sophisticated tests run on the car and usually
are useful to employ in concert with traditional obtain a better assessment of the problem. Moving
QC. Moving Averages, Average of Normals, Averages are the dashboard lights whereas the
Patient-based QC, these are all terms that apply QC analysis will usually be a better assessment
to a set of “normal” patient specimens that are of the problem. Labs using Moving Averages
averaged continuously to monitor the stability of can track instrument performance between QC
the method. The trickier part is filtering out all events. But, if the Moving Average indicates a
the abnormal patient values and determine how true change in normal patient results, it is highly
many patient specimens need to be combined to recommended that the lab run the quality control
generate the average. The EWMA uses a weighting material to better ascertain what kind of problem
that values the more recent values more than the is impacting the test result.
older ones. The technical details are beyond the
scope of this workbook, but these techniques, if
you have the informatics architecture to support
the calculations, are a useful complement to
traditional QC. Moving Averages are often
described as “real-time” QC, since the results are
far more immediate than the periodic traditional
controls that are run.

Think of Moving Averages as a light on the

dashboard of your car. Once you see this light,
you take the car to the mechanic to get more

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

28 WRAPPING UP

Wrapping up

Advanced QC means more planning before the complementary statistics. Remember, however,
actual QC material is run and the data captured, that while RCV and Moving Averages can add
and more planning to utilize additional techniques value to basic quality control, they can’t replace it.
after the QC is run, with the goal of reducing how
much effort you expend on all of these processes. The ultimate goal of this workbook is to make
sure you get maximum benefit from your QC
So how do you take your QC to the next level of processes, with no wasted effort. Through the
laboratory operations? Sigma-metric approach, you can optimize and
customize your rules, controls, and frequency, so
Make sure the basics of quality control are known you aren’t “over-controlling” methods that are
and faithfully and correctly implemented. But highly reliable, and you aren’t “under-controlling”
invest time before you run QC – in the selection methods that are more unstable.
of the control material and training of the staff, in
the construction and set up of the Levey-Jennings There are more resources available to you on all of
charts. The act of quality control is compromised these subjects online.
if we start out with inferior controls, or the wrong
ranges and limits, or the wrong rules. Thanks for spending time with this workbook and
course materials.
Make sure you also invest in what you do
beyond QC, such as Sigma-metrics, RCV, Moving
Averages, EWMA, etc. You can maximize the
usefulness of your outputs with these additional,

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 WRAPPING UP 29

Self-assessment Quiz - Answer Key

Exercise 1: Exercise 4:
Q1: Do both instrument values fall within the package Using data in example one above, review the actions
insert range? taken by the bench technologist for troubleshooting.
Answer: Yes. Remember, the following Westgard Rules were used
Q2a: What would you choose for instrument A’s new for this exercise: I2s (warning), 22s (reject), 13s (reject),
control lot mean based upon the table above? 10x (warning), and R4s (warning). Are these actions
Answer: 0.48. correct?
Q2b: What is instrument A’s SD for the new lot using Answer:
the CV of old lot? Run 1 Yes
Answer: 0.14. Run 2 Yes
Run 3 Yes
Q2c: What would you choose for instrument B’s new
Run 4 Yes
control lot mean based upon the table above?
Run 5 No- repeat level one
Answer: 11.32. Run 6 Yes
Q2d: What is instrument B’s SD for the new lot using Run 7 Yes
the CV of old lot? Run 8 No- repeat both levels
Answer: 0.13. Run 9 Yes
Q3 Using the calculated mean and SD, what ranges Run 10 Yes
would you apply to both instruments. Run 11 Yes
Run 12 No- no need to repeat
Answer 3a: Instrument A: +/- 2 SD range: 10.20
Run 13 Yes
to 10.76.
Run 14 Yes
Answer 3b: Instrument B: +/- 2 SD range: 11.06 to Run 15 Yes
11.58 Run 16 Yes
Run 17 Yes
Exercise 2: Run 18 Yes
Record any rule interpretation based upon the Run 19 Yes
Westgard rule violated and whether you would hold Run 20 Yes
patient results. The following Westgard Rules should Run 21 Yes
be used for this exercise: I2s (warning), 22s (reject), 13s Run 22 Yes
Run 23 Yes
(reject), 10x (warning), and R4s (warning).
Run 24 Yes
Answer:
Run 25 Yes
Run 2: 13s Reject Run 26 Yes
Run 4: 12s Warning Run 27 Yes
Run 5: 22s Reject Run 28 Yes
Run 8: 22s Reject Run 29 Yes
Run 30 Yes
Run 11: 12s Warning

Exercise 3: Exercise 5:
Below are two Levey-Jennings charts for two levels of QC rules: 12s warning, 22s run rejection, 22s run rejection,
a control product for a certain test. The graph from 13s run rejection, 10x run rejection.
each day’s QC result is plotted based upon the SDI Situation # 1: Sigma score is 7.2. Based upon the
or z-score value. Based upon these two charts, please action above, which Westgard rule(s) could you
note what day’s QC results failed any of the Westgard implement?
Rules: Answer: 13s
Answer:
12s: 4, 11 Situation #2: Now, let’s assume that the sigma
22s: 5, 8 score is 4.1. Which Westgard rule(s) could you
13s: 2 implement?
10x within run: R4s: 4, 12 Answer: 13s, 22s, R4s, 41s

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

30 GLOSSARY

Glossary
Allowable Total Error (TEa) encompasses the – that the control material is as close as possible
imprecision and bias of a single test measurement. to a real patient sample. This attribute provides
It is a quantity used in assessing external quality confidence that when the device produces a
assurance, proficiency testing, and analytical control value out of range - thus indicating that
Sigma-metrics. the test system has a problem, you can be certain
that the patient sample results would be incorrect
Analytical measuring range (AMR): this is the as well.
same as your working or reportable range for
each assay in your laboratory. This is the range Exponentially Weighed Moving Average (EWMA)
of values the method is able to report patient or sometimes called Moving Averages, Average
and quality control samples. Your controls should of Normals, Patient-based QC; these are all terms
recover close to key medical decision levels or that apply to a set of “normal” patient specimens
cutoff levels. that are averaged continuously to monitor the
stability of the method.
Analytical Sigma-metric: leverages the observed
performance of your method and places it into Matrix Effect: The matrix of a control is all the extra
the context of a universal scale. This is frequently stabilizers, preservatives, and other ingredients
referred to as Six Sigma or Sigma-metric. that are present that are wholly unrelated to a
patient sample. These additives may help keep
Calibrators or Calibration Materials: solutions the control material stable, or have a longer shelf
or devices of known quantitative/qualitative life, but they do not make the control behave
characteristics (e.g., concentration, activity, similarly to a patient sample
intensity, reactivity) used to calibrate, graduate, or
adjust a measurement procedure or to compare Mean: Also referred to as the average.
the response obtained with the response of a test
specimen/sample. Measurement Uncertainty (MU): is your
intermediate imprecision, and your expanded
Coefficient of Variation (CV): a calculation that uncertainty is 2*CV. ISO 15189 and all derivative
allows you to monitor the imprecision across accreditation guidelines, requirements, and
multiple control levels, even compare imprecision regulations mandate that every method should
between methods and instruments, and compare determine MU, and report it to clinicians.
them against the manufacturer’s expectations.
The lower the CV, the lower the test system’s Multi-analyte controls: These are control products

imprecision. with more than one analyte. The goal of the

laboratory is to utilize control materials that cover
Coefficient of Variation Ratio (CVR): This as many analytes as possible. This will allow for
calculation will allow you to assess if the CV of the reduction of the single analyte controls and
your test system is comparable to other systems therefore saving the lab time in processing many
exactly like yours. This is usually provided with vials of controls, managing all the lot numbers
QC and QA peer group programs. and expiration dates, and performing multiple
cross-over studies throughout the year.
Commutability is the goal of any type of control

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 GLOSSARY 31

Open vial stability: Refers to how long the control Standard Deviation Index (SDI): This is a
material, once opened, is stable and produces measurement of the difference between the
optimal results before it deteriorates. A longer laboratory’s mean from the peer group mean as
open vial life will give you more time to utilize it. measured by the peer group standard deviation.
While it’s a discussion of accuracy and trueness
QC design: This is planning and designing (bias), it’s expressed in units of standard deviation
QC procedures for maximum error detection, or imprecision (random error).
minimum false rejection, and optimum efficiency.
The design should take into effect the appropriate Systematic Errors: See trends and shift definitions.
mean, ranges, appropriate rules, frequency of
control testing, consideration on patient testing Total Error (TE) is related to but not the same as

volume, quality of the assay itself, etc. Allowable total error (TEa). Total Error was one
of the first estimates of combined imprecision
Random Errors: these errors are much more and inaccuracy, first published in the 1970s. The
difficult to identify and resolve, mostly due to the net or combined effect of random and systematic
nature of the error which cannot be predicted or errors.
quantified as can systematic error. Some describe
these as “flukes”. Trend: This is usually observed with the QC
values gradually increase or decrease over time
Reference Change Value (RCV): smallest on the Levey-Jennings chart. This is indicative of
change between two serial patient results that is a systematic error.
nevertheless a clinically important difference

Shelf life: This is the length of time that the

product remains stable. The product should not
remain in use in the laboratory past the expiration
date unless approved by the manufacturer. Labs
must maintain a copy of the manufacturer’s
extension letter for future inspection needs.

Shift: This is an abrupt change in the QC results

and is caused by a distinct and, in some cases,
dramatic change in a component of the test
system. This is a systematic error.

Standard Deviation: a measurement of how

closely the control values cluster around the
mean, how tightly packed they are around the
mean, or how widely dispersed they are away
from the mean.

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

32 REFERENCES

References
CMS, CDC, HSS. Clinical Laboratory Improvement https://www.cms.gov/Outreach-and-Education/
Amendments of 1988 (CLIA) Proficiency Testing Medicare-Learning-Network-MLN/MLNProducts/
Regulations Related to Analytes and Acceptable Downloads/CLIABrochure.pdf
Performance. Fed Reg 2019; 84:1536-1567.
2019 Comprehensive Accreditation Manual for
US Department of Health and Social Services. Laboratory and Point-of-Care Testing. Joint
Medicare, Medicaid, and CLIA Programs: Commission, Oakbrook Terrace, IL 60523.
Regulations implementing the Clinical Laboratory https://www.jointcommission.org
Improvement Amendments of 1988 (CLIA). Final
Rule. Fed Regist `992 (Feb 28);57:7002-7186. 2018 CAP Checklists. Northfield, IL 60093.
https://www.cap.org
US Centers for Medicare & Medicaid Services
(CMS). Medicare, Medicaid, and CLIA Programs: 2019 COLA Laboratory Accreditation Manual.

Laboratory Requirements Relating to Quality Columbia, MD 21046.

Systems and Certain Personnel Qualifications. https://www.cola.org

Final Rule. Fed Regist Jan 24 2003;16:365-3714.

ISO 15189: Medical laboratories — Requirements

Current CLIA Regulations (most up to date for quality and competence, ISO, Geneva,

electronic version) available at: Switzerland

https://www.ecfr.gov/cgi-bin/
CLSI Harmonized Terminology Database.
retrieveECFR?gp=1&SID=4e8cf4d97d3ae
https://htd.clsi.org/listterms.asp?searchd
4925e669e74bb825785&ty=HT
Last accessed 3/4/2020.
ML&h=L&mc=true&r=PART&n= pt42.5.493
ISO 15194:2002. Description of reference
CMS State Operations Manual (Appendix C),
materials. Geneva, Switzerland: ISO; 2002. In
Regulations and Interpretive Guidelines for
vitro diagnostic medical devices – Measurement
Laboratories and Laboratory Services, available
of quantities in samples of biological origin
at:
https://www.cms.gov/Regulations-and- Allowable Total Error Limits- CLIA: https://www.
Guidance/Legislation/CLIA/index?redirect=/clia/ westgard.com/clia.htm

Clinical Laboratory Improvement Amendments Allowable Total Error Limits Data Innovations
(CLIA) homepage: database: https://datainnovations.com/
https://www.cms.gov/Regulations-and- allowable-total-error-table
Guidance/Legislation/CLIA/CLIA_Regulations_
CLSI EP26 User Evaluation of Between-Reagent
and_Federal_Register_Documents.html
Lot Variation, 1st Edition document:
MLN Fact Sheet. CLIA Program and Medicare https://clsi.org/standards/products/method-
MLN Fact Sheet. CLIA Program and Medicare evaluation/documents/ep26/
Laboratory Services. Centers for Medicare and
Medicaid Services. 2018. Available at:

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 P.A.C.E CONTINUING EDUCATION ASSESSMENT 33

P.A.C.E Continuing Education Assessment

Each person may receive three (3) P.A.C.E. contact hours of continuing education credit by studying
this workbook, performing the study questions, and returning the completed final assessment to:

Mailed Scanned and Faxed Scanned and Emailed

Technopath USA
ATTN: Terri Wolek
99 Lafayette Drive [email protected]
516-864-0166
Syosset, NY 11791

Please note, only one original copy of the final assessment will be awarded the contact hours. The
participant must pass the final assessment test by at minimum of 70% score.

Technopath is approved as a provider of continuing education program in the clinical laboratory

sciences by ASCLS P.A.C.E.® Program. This advanced self-instructional course is approved for 3
contact hours.

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

34 P.A.C.E CONTINUING EDUCATION ASSESSMENT

Basic QC Workbook PACE Continuing Education Test

Contact Hours: 3
1. What’s the downside of having controls at extreme levels but not at medically relevant decision
levels?

a. Controls might be out, but patient samples might be perfectly fine

b. Controls might be fine, but patient samples are experiencing a significant error

c. All of the above

2. Do you need controls running at the very lowest and highest levels of the range of the test?

a. Only if those levels are medically relevant

b. Only if those levels are medically irrelevant

c. Only if those levels are measurement uncertainty

d. Only if those levels are moderate complexity

3. When aligning your control material usage to your shelf life, you want to your use of controls to

a. Exceed the shelf life of the controls

b. Match the shelf life of the controls

c. Fall short of the shelf life of the controls.

d. All of the above

4. What aspect of the control material characterizes how long you can use the material once you
have opened it?

a. Open Source

b. Open Book

c. Open Vial

d. Open Sesame

5. Commutability means…

a. Technologists can easily commute to work

b. Technologists gain a superpower that allows them to change controls into patient samples

c. Controls mimic patient samples

d. Technologists gain the power to commute prison sentences

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 P.A.C.E CONTINUING EDUCATION ASSESSMENT 35

6. The Matrix of a control is…

a. Stabilizers, preservatives, and other additives that make the control Less similar to a patient
sample.

b. Aspects of artificial reality that make it Less likely people will wake up and fight their robot
overlords

c. Aspects of a control material that make the control More similar to patient samples

7. When is the best time to use the package insert range of a control?

a. When the control has expired

b. When you have accumulated 3 to 6 months of performance data

c. When you have accumulated 20 days of performance data

d. When you have no other information

8. Once you have calculated the mean and SD over the first 20 days, when should you update or
change the mean and SD?

a. As soon as the control lot expires

b. As soon as you have a new control lot

c. As soon as you have peer group means and SDs to use instead

d. As soon as you have more data, an additional month, or an additional 2 months, etc.

9. What is the danger of having one common mean and a common SD for a group of instruments?

a. Small SD may generate more outliers

b. Large SD may miss errors

c. Mean too high may generate more outliers

d. Mean too low may generate more outliers

e. All of the Above (and possibly more)

10. If your control material has a matrix issue, what is the problem that could emerge during a reagent
lot change?

a. Control will indicate a shift, but patients will not change.

b. Control will not change even when patient values are shifted

c. Either of the above

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

36 P.A.C.E CONTINUING EDUCATION ASSESSMENT

11. When the control material has a short shelf life, you should not perform:

a. Eliminate the practices of performing cross over study.

b. Notate the date when the control vial when put in use

c. Track performance in the control in the QC Data Management program/record.

12. After completing the new lot’s crossover study, you notice that your calculated mean is not within
QC vendor’s package insert range. What are appropriate action(s)?

a. Review is the control was stored properly in the laboratory

b. Review performance of the current QC data for any issues

c. Verify the QC vendor’s units and method compared to the package insert data

d. All of the above

13. How often should I adjust my test’s mean?

a. Every time I get my peer group reports

b. When it is a significant change in the test system that has caused a truly systematic change
resulting in the test’s new state.

c. With every reagent lot change

d. Just prior to the inspection so the surveyor knows I am paying attention

14. What should you use to set your range?

a. Allowable Total Error (TEa)

b. Peer group SD

c. PT/EQA peer group SD

d. Your individual laboratory SD, ideally

15. Why use “Westgard Rules”?

a. They balance high error detection with low false rejection

b. They balance high cost with low quality

c. They balance high complexity with low comprehension

d. They balance high uncertainty with low measurement

16. What is the difference between “Westgard Rules” and Westgard Sigma Rules?

a. Westgard Sigma Rules is just 6 times more Westgard Rules

b. Westgard Sigma Rules is one sixth of the Westgard Rules

c. Westgard Sigma Rules is just a way to optimize Westgard Rules

d. Westgard Sigma Rules is just a way to maximize Westgard Rules

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 P.A.C.E CONTINUING EDUCATION ASSESSMENT 37

17. During monthly QC review you should NOT…

a. Review bench level actions

b. Review instrument history

c. Review observed mean, SD, CV of the method

d. Rerun all control outliers including any warnings or errors.

18. What 3 variables do you use to calculate the analytical Sigma-metric?

a. Measurement uncertainty, expanded measurement uncertainty, and permissible measurement

uncertainty

b. TAT, throughput, and calibration time

c. TEa, CV, Bias

d. insert mean, peer group mean, reference mean

19. Where can you find TEa performance specifications?

a. CLIA

b. EQA PT

c. EFLM biological database

d. All of the above

20. If you use TE and compared to TEa, what should you monitor?

a. You should ensure TE < TEa

b. You should ensure TE > TEa

c. You should ensure TE < 1/6 TEa

d. You should ensure TE < 6* TEa

21. Where can you find TEa goals?

a. CLIA PT criteria

b. CAP PT criteria

c. Ricos and colleagues from Spain

d. All of the above and more

22. When given a choice between possible TEa goals, you should always choose

a. The smallest goal

b. The largest goal

c. The average of the goals

d. The goal that best matches patient needs

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL
38 P.A.C.E CONTINUING EDUCATION ASSESSMENT

23. What is the difference between TE and TEa?

a. One is allowable and the other is actual

b. One is uncertain and the other is unknown

c. One is compliant and the other is complacent

d. One is total and the other is terrifying

24. Given a 5 Sigma method, what would the Westgard Sigma Rules recommendation be?

a. 1:3s

b. 1:3s/2:2s/R:4s

c. 1:3s/2:2s/R:4s/4:1s

d. 1:3s/2:2s/R:4s/4:1s/8:x

25. Given a TEa of 15%, Bias of 3%, CV of 2%, what is the analytical Sigma-metric?

a. 3 Sigma

b. 4 Sigma

c. 5 Sigma

d. 6 Sigma

26. Given a TEa of 13%, Bias of 3%, CV of 2.5%, what is the analytical Sigma-metric?

a. 3 Sigma

b. 4 Sigma

c. 5 Sigma

d. 18.5 Sigma

27. Given a 4 Sigma method, how many Westgard Rules are recommended?

a. 1:3s

b. 1:3s/2:2s/R:4s

c. 1:3s/2:2s/R:4s/4:1s

d. 1:3s/2:2s/R:4s/4:1s/8:x

28. If the difference between 2 serial patient results is greater than RCV, that means...

a. The run is out of control.

b. The run is uncertain.

c. You can’t be sure there is anything changed in the patient’s status.

d. A clinically significant change is likely to have occurred in patient replicates

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

 P.A.C.E CONTINUING EDUCATION ASSESSMENT 39

29. Within the US, measurement uncertainty is…

a. Basically unknown

b. Completely uncertain

c. Mandated by CLIA.

d. Cornerstone of compliance

30. Moving Averages and EWMA are a

a. Replacement for QC

b. Complement to QC

c. Complication of QC

d. Reduction of QC

Please record your laboratory details below to receive P.A.C.E. continuing education contact hours:

Laboratory Name:

Laboratory email:

Facility Name:

Facility Address:

Facility Phone number:

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

40 P.A.C.E CONTINUING EDUCATION ASSESSMENT

Evaluation of Advanced QC Workbook

This section must be completed with the answer key in order to process your quiz for P.A.C.E. content
hour credits.

How well did this book meet the objectives/goals of this advanced workbook?

Did not
Objective Met Exceeded
meet

Illustrate the purpose and practice of statistical QC

Outline the setup, implementation and interpretation of

single statistical rules as well as “Westgard Rules”

Reveal useful troubleshooting techniques

Grade the following statements:

1 = Disagree Completely
2 = Somewhat Disagree
3 = Agree
4 = Completely Agree

The Basic QC Workbook. . .

1. ...helped me with understanding advanced QC practices and applications.. . 1 2 3 4

2. ...was user friendly and well formatted. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 3 4

3. ...is not a reference source for future use. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 3 4

ADVANCED APPLICATIONS IN CLINICAL LABORATORY QUALITY CONTROL

About the Author
Sten Westgard MS

Sten Westgard, MS, is the

Director of Client Services
and Technology for Westgard
Quality Control.

For nearly 25 years, Sten has managed the

Westgard website, course portal, and blog,
creating and administering online training, as well
as editing and writing hundreds of reports, essays,
and applications on quality control, method
validation, Six Sigma, Risk Management and other
laboratory management topics.

He has edited and contributed to numerous books

on quality, including Basic QC Practices, Basic
Method Validation, Basic Quality Management
Systems, Six Sigma QC Design and Control, Six
Sigma Risk Analysis, CLIA Final Rules, Assuring
the Right Quality Right, The Poor Lab’s Guide to
the Regulations and Nothing but the Truth about
Quality. He has co-edited two special issues of
Clinics in Laboratory Medicine (2015 and 2017),
as well as a special issue of Biochemica Medica
(2018)

Sten is also an adjunct faculty member of

the Mayo Clinic School of Health Sciences in
Rochester, Minnesota; an adjunct faculty member
of the University of Alexandria, Egypt; an adjunct
visiting faculty member of Manipal University
in Mangalore, India; and an honorary visiting
professor in 2017 at Jiao Tong University, Shanghai.

www.westgard.com
www.westgard.org
james.westgard.com
www.linkedin.com/in/sten-westgard-683770/
 www.technopathcd.com

[email protected] | Tel: +353 61 525700

Technopath Life Sciences Park, Fort Henry, Ballina, Co. Tipperary, V94 FF1P, Ireland.

USA
[email protected] | Tel: 1.888.235.3597
99 Lafayette Drive, Suite 179, Syosset, NY 11791