Clinica Chimica Acta 490 (2019) 28–33

Contents lists available at ScienceDirect

Clinica Chimica Acta

journal homepage: www.elsevier.com/locate/cca

Is your assay stable? Using process stability and capability to evaluate assay T
performance
⁎
Robert L. Schmidt, Lauren N. Pearson
Department of Pathology and ARUP Laboratories, University of Utah, Salt Lake City, UT, United States

A R T I C LE I N FO A B S T R A C T

Keywords: Background: Many laboratories are involved in efforts to increase precision and improve process capability.
Quality Control Metrics to evaluate performance, assess potential for improvement, and prioritize improvement projects would
Stability facilitate these efforts. We show how stability analysis can provide metrics to evaluate assay performance and
Capability indicate potential for improvement. We also show how stability analysis along with capability analysis can be
Process Improvement
used to prioritize assays for improvement. Stability reflects the degree to which a process is free of extrinsic
Statistical Process Control
sources of variation. Capability reflects the ability of a process to meet external requirements.
Methods: We used the SR test and analysis of variance to compare the short-term and long-term stability of two
assays. We illustrate the analysis with detailed calculations for two assays.
Results: One assay (thyroid stimulating hormone) was stable and the other, (methotrexate) was unstable.
Conclusion: Stability analysis provides metrics that can be used evaluate assay performance and to prioritize
process improvement efforts.

1. Introduction over a period of time. The analysis is performed by examining patterns

in a process behavior chart (e.g., Levey Jennings chart) in which suc-
Healthcare organizations are under increasing pressure to reduce cessive quality control (QC) results are plotted in sequence over time.
costs, improve patient safety, and increase overall efficiency. Clinical Such charts show the variation in the process output (e.g., QC results)
laboratories contribute to these goals by producing accurate, reliable, over time. Stability analysis attempts to determine whether the varia-
and timely results for patient care. The quality management system of a tion is abnormal. Thus, this analysis requires an understanding of the
clinical laboratory assesses all phases of the testing process (pre-ana- factors that give rise to variation.
lytic, analytic, and post-analytic phases) to help maintain patient safety The variation in QC results can be understood by taking a systems
and ensure reliable results. Many laboratories are engaged in con- view of the overall measurement process. The measurement system has
tinuous improvement activities that span all phases of the testing pro- multiple inputs (e.g., temperatures, concentrations, mixing times, vol-
cess. tages, etc.) that contribute to produce the final result. Variation in these
Continuous improvement projects consume resources and labora- inputs cause variation in the output. Usually, these variations are small
tories that are engaged in continuous improvement need to prioritize and it is impossible to relate the variation in the output to the variation
their efforts. To do so, they need metrics to summarize the historical in any single input. This variation is called common cause variation and
performance of an assay and indicate the potential for improvement. represents the natural variation of the measurement process [1]. Be-
Without such metrics, resources may be used inefficiently. The objec- cause it is based on the combination of many varying inputs, common
tive of this article is to introduce stability analysis, a tool that can be cause variation contains no patterns (shifts, trends, etc.) [2]. A process
used to assess the potential for assay improvement. This tool, used in that exhibits a relatively predictable pattern of random variation
conjunction with capability analysis, can help managers prioritize im- (common cause variation) is said to be stable or in statistical control
provement activities. We also describe a two-dimensional matrix, the [3–5].
stability-capability matrix that can be used to evaluate assay perfor- At times, a process can become unstable and produces results that
mance. are unusual or contain a pattern. For example, QC results may show an
Stability analysis is a retrospective analysis of process performance extreme outlier, shifts or trends. When such results are observed, it is

⁎
Corresponding author at: Department of Pathology, University of Utah, 15 N Medical Drive East, Salt Lake City, UT 84112, United States.
E-mail address: [email protected] (L.N. Pearson).

https://doi.org/10.1016/j.cca.2018.12.015
Received 14 November 2018; Received in revised form 11 December 2018; Accepted 12 December 2018
Available online 13 December 2018
0009-8981/ © 2018 Elsevier B.V. All rights reserved.
R.L. Schmidt, L.N. Pearson Clinica Chimica Acta 490 (2019) 28–33

Fig. 1. Assignable cause variation. Assignable cause variation is variation in the output that can be linked, in theory, to a change in an input. Assignable cause
variation is due to some event that is extrinsic to the process acting on the process so that the process does not behave as designed. Assignable cause variation is
distinguished from common cause variation. Common cause variation is the natural variation that is intrinsic to the process and arises from the net effect of the
normal variation in many inputs. In contrast to assignable cause variation, common cause variation contains no patterns or unusual results and cannot be linked to a
particular input.

likely that the output no longer represents common cause variation 2. Methods
(i.e., the combination of small fluctuations in many inputs). Rather, it is
reasonable to presume that the change in output was caused by a large 2.1. Evaluating stability
change in one or more inputs (Fig. 1). Such variation is called assign-
able cause variation because the change in output can be linked (in There are several methods to evaluate stability [9,19]. We will focus
theory) to a particular input, or assignable cause [3,6,7]. Assignable the SR test and ANOVA test because they are based on comparisons of
cause variation is extrinsic to the process and reflects a change that is short-term and long-term variation which has a simple managerial in-
outside the normal operation of the process [7]. When assignable cause terpretation.
variation is present, the process is said to be unstable or out of statistical The standard deviation of the long-term variation, sL, is estimated
control. using the standard formula for the standard deviation:
Common cause and assignable cause variation provide a useful way
N
to categorize variation. However, variation can also be viewed from a 1
short-term and long-term perspective. The short and long-term per-
sL =
N−1
∑ (Xi − X )2
i=1 (1)
spective can be linked to common cause and assignable cause variation
[8,9]. Over the long term, the total variation will be a combination of where Xi is observation i (i = 1…N) and X is the sample average. N
common cause variation and assignable cause variation [10]. The would be selected so that the data covered a long period of time (e.g.
common cause variation is always present and represents the “back- 100 runs).
ground noise” of variation in the process. Episodes of instability (due to The short-term variation is usually estimated using rational sub-
assignable cause variation) add to this background to produce the total groups. A rational subgroup is a group of QC measurements performed
variation. When such episodes are infrequent, assignable cause varia- under homogeneous conditions. In clinical chemistry, such measure-
tion will contribute very little to short-term variation and short-term ments are called repeats (i.e., more than one QC sample measured at
variation will reflect the underlying common cause variation. For ex- one level at a single point in time). The variation within one rational
ample, consider a process with a shift in the mean due to assignable subgroup (within group variation) provides a single estimate of the
cause variation. Before and after the shift, the variation is relatively short-term variation. The average short-term variation is determined by
small. Variation measured over short periods would not incorporate the averaging the within-group variation over all rational subgroups.
shift and would reflect the common cause variation. Variation mea- The standard deviation can also be estimated using the range. The
sured over longer periods would incorporate the shifts due to assignable range of subgroup, Ri, is defined as:
cause variation. Thus, differences between long-term variation and
short-term variation are due assignable cause variation [8,11–18]. The Ri = max(Si ) − min(Si ) (2)
long-term and short-term variation are similar when process is stable
the output contains little assignable cause variation. The more unstable where Si = {Xi1, Xi2, …Xim } is the set of m samples taken at time point i.
the process, the greater the difference between the short-term and long- The short-term standard deviation, ss is estimated from the average
term variation. range, R :

29
R.L. Schmidt, L.N. Pearson Clinica Chimica Acta 490 (2019) 28–33

Table 1 where TEa is the total allowable error, b is the bias and s is the standard
Example calculations. df = degrees of freedom, SS = sum of squares. deviation. Capability is a quantitative index that relates the observed
MTX = methyltrexate (L2), TSH = thyroid stimulating hormone (L3). variation in QC results to the allowable variation, TEA − b. When a
TEST Statistic MTX TSH process is capable, the variation in the QC results is much lower (say 3,
to 4 times) than the allowable variation. Capability is often expressed as
Stability SR TEST Number of observations 465 826 “sigma” units.
mean 0.43 25.8
Capability reflects the current observed capability or the capability
Average moving range (R ) 0.01 1.26
Short-term (ST) standard 0.013 (288) 1.12 (512)
in a potential future state, depending on the value used for the standard
deviation (df) deviation. The observed capability is based on the long-term variation,
Long-term (LT) standard 0.03 (464) 1.16 (825) sL:
deviation (df)
Ratio LT/ST 2.3 1.03 Σobs = [TEa − b]/ sL
F statistic (SR statistic) 4.7 1.1
P value < 0.0001 0.18
The observed capability reflects the current state of the process.
Conclusion Unstable Stable Processes are stabilized and improved by identifying and controlling
ANOVA Total SS (df) 0.312 (464) 1106 (825) assignable cause error. The short-term variation provides an estimate of
Model SS (df) 0.266 (232) 596 (412) the variation that would be present if assignable cause variation were
Residual SS (df) 0.046 (232) 511 (413)
removed. The potential or future capability is based on the short-term
F statistic 5.7 1.2
P value < 0.0001 0.06 variation:
Conclusion Unstable Stable
[TEA − b]
Capability Total allowable error 25% 30% Σpot =
Bias 5% 5% sS
Observed capability (LT sd) 2.7 5.8
Potential capability (ST sd) 6.3 6.0
Conclusion Not Capable Capable 3. Results

3.1. Example calculations

N
R = ∑ Ri We illustrate the use of stability analysis by providing detailed
i=1 (3)
calculations for two assays: methotrexate (MTX) and thyroid stimu-
lating hormone (TSH).
ss = R d2 (4) For MTX, the ratio of the long-term standard deviation to the short-
where d2 is a constant that depends on the sample size, m. Tables for d2 term standard deviation was 2.3 (Table 1). The SR statistic (F statistic)
are available in standard textbooks on quality control and on many was 4.7. There was a statistically significant difference between the
websites [3,4,20]. In many applications, including clinical chemistry, long-term and short-term variation (p < .0001). The ANOVA F test
the sample size is typically one (m=1). In that case, it is impossible to also found a statistically significant difference (F = 5.7, p < .0001).
obtain ranges using Eq. (2) and subgroups are formed from successive For TSH, the ratio of the long-term standard deviation to the short-
measurements (i.e. moving range). The reasoning is that conditions in term standard deviation was 1.30 (Table 1). The SR statistic (F statistic)
successive measurements are likely to be similar and form a reasonable was 1.1. Based on the SR test, the difference between the long-term and
approximation of a rational subgroup (with size m=2) [21]. The short-term variation was not statistically significant (p = .18). The
moving range, mRi, is then defined as: ANOVA test also found no statistically significant difference (F = 1.2,
p = .06).
mRi = |Xi − Xi − 1| (5) The expected distributions of results for the current state (long-term
The standard deviation is then estimated using Eqs. (3) and (4) variation) and potential future state (short-term variation) for MTX and
except that the moving range, mRi, is used in place of the range of a TSH are presented in Fig. 2.
subgroup, Ri.
The SR ratio is the ratio of the long-term variation to the short-term 3.2. The stability-capability matrix
variation:
Stability and capability can be graphically represented on a grid that
SR = sL/ sS (6)
we call the stability-capability matrix (Fig. 3). The SR index (stability
The SR test is performed by comparing the short and long-term index) and capability index provide two metrics that can be used to
estimates of the variance. This is done using the F test. help managers prioritize process improvement efforts and guide man-
agerial action. The stability-capability provides information on two key
F = sL2/ sS2 (7) performance measures for each assay and provides a useful summary of
The numerator has N-1 degrees of freedom. The degrees of freedom laboratory performance. It is a tool that can help managers prioritize
of the denominator is a subject of research but the current re- assays for improvement and provides guidance for managerial action
commended value is 0.62 N [9]. (assay improvement vs redevelopment). We suggest cutoffs that
Using the ANOVA approach, subgroups of successive observations roughly divide good and poor performance. For capability, a sigma
are created to compare the within-group (short-term) and between- value of three is generally considered a minimum value for a capable
group (long-term) variation [9]. process. For stability, we find that processes with an SR index of ≥1.2
generally have statistically significant instability. These rules of thumb
divide the matrix into 4 quadrants.
2.2. Evaluating process capability
Assays that fall into the stable/capable quadrant are operating at
full potential and are not in need of improvement. Laboratories should
Many laboratories are familiar with process capability; however, we
continue to monitor these processes but resources should not be spent
provide a short review. Process capability (sigma) is defined as:
on improving these assays.
[TEa − b] Assays that are incapable but stable are problematic. Because such
Σ=
s assays are stable, there is little opportunity to reduce variation with

30
R.L. Schmidt, L.N. Pearson Clinica Chimica Acta 490 (2019) 28–33
.6

TSH
.4
.2
0

−4 −2 0 2 4
.6

MTX

Fig. 3. Stability-Capability Matrix. Each circle represents an assay. The size of

.4

the circle corresponds to the annual volume of the assay. Stability is the ratio of
the short-term (ST) to long-term (LT) variation. Assays with a stability
index > 0.8 are considered stable. Those with a stability index < 0.8 are
considered unstable. Assays with capability > 3.0 are considered capable.
.2

assignable cause variation. In such cases, the source of assignable cause

variation should be identified and removed because the magnitude of
the uncontrolled variation could easily change. However, one could
observe sources of assignable cause variation (e.g. lot changes) that
0

never affect capability. Such sources of variation are assignable causes

−4 −2 0 2 4 but may not be manageable or a cause for concern. Thus, we believe
Fig. 2. Comparison of long-term and short-term variation of example assays.
that there are circumstances where a process can be “unstable” (as-
Long-term variation is shown by the solid gray curve. Short-term variation is signable cause variation is identified) but capable. In such cases, one
shown by the dashed red curve. Long-term variation is composed of both needs to determine whether the variation is intrinsic to the measure-
common cause variation and assignable cause variation. The long-term varia- ment process (common cause variation) or should be regarded as ex-
tion would become similar to the short-term variation if assignable cause var- trinsic (assignable cause variation).
iation were removed by process improvement activities. The difference between
the curves shows the potential for improvement. The thyroid stimulating hor-
mone (TSH) assay is stable and has little opportunity for improvement. The 4. Discussion
methotrexate (MTX) assay is unstable and has opportunity for improvement.
(For interpretation of the references to colour in this figure legend, the reader is This study shows how stability analysis can be used to assess the
referred to the web version of this article.) performance of assays in a clinical laboratory. We provided two prac-
tical examples in which we outlined the statistical analysis for stability.
process improvement programs that identify and remove assignable We also showed how stability analysis provides a useful metric that can
cause variation. It is possible that the poor capability could be due to be used to compare assay performance.
bias or to a mistaken TEa value. If so, these should be addressed. In Stable processes differ from unstable processes due to the presence
general, a stable but incapable process needs to be replaced. of assignable cause variation. The variation of a stable process consists
Assays that are incapable and unstable have the potential to be only of common cause variation. Thus, for a stable process, the long-
salvaged. The key is to identify and remove sources of assignable cause term variation and short-term variation with be similar and the ratio of
variation. This will reduce variation and may move the assay to the long-term to short-term variation will be close to one. The long-term
stable/capable quadrant. It is possible that process improvements will variation is usually greater than the short-term variation when assign-
move the assay to the stable/incapable quadrant which suggests that it able cause variation is present in an unstable process.
needs to be replaced because there is little opportunity for improve- Process improvement (i.e. variance reduction) is achieved by iden-
ment. tifying and controlling sources of assignable cause error [22]. Im-
The last quadrant contains assays that are unstable but capable. The provement can be achieved when there is a difference between the
quality literature suggests that a process can't be capable unless it is long-term and short-term variation. There is little opportunity to reduce
stable. That is because an unstable process is unpredictable. It may be the variation of a stable process because stable processes have no as-
capable today but become incapable tomorrow. This depends some- signable cause variation to identify and control. In contrast, unstable
what on the capability of the process. A process may meet customer processes can be improved by identifying and controlling sources of
specifications (TEa) but also display patterns that are associated with assignable cause variation.
The stability-capability matrix compares the performance of assays

31
R.L. Schmidt, L.N. Pearson Clinica Chimica Acta 490 (2019) 28–33

using two important measures of performance. Capability shows whe- Several different statistical tests have been proposed for stability
ther an assay is meeting requirements. The stability index shows the analysis. We prefer the SR test because it is relatively easy to interpret
potential for improvement. An unstable process is influenced by as- and to implement For example, it indicates the change in CV or process
signable cause variation that, in theory, could be identified and re- capability (sigma) that could be obtained through process improvement
moved to stabilize the process. Thus, the matrix can help managers efforts. Also, the SR test can be easily implemented in Microsoft Excel.
improvement efforts by identifying assays that are performing poorly Other tests provide measures that are more difficult to interpret or re-
and, in addition, suggesting appropriate responses. Stable processes quire statistical software. In our laboratory, we perform all the stability
have little opportunity for improvement. A stable process should be tests. Each statistical test provides a different perspective on stability. In
replaced if it is incapable because process improvement activities are general, the statistical tests are correlated - if one test indicates in-
unlikely to be helpful. In contrast, an unstable process has potential for stability, the others usually do as well.
improvement because one can identify and remove assignable sources QC activities in clinical laboratories have traditionally focused on
of variation. Thus, the stability-capability matrix can help manages compliance. From a compliance perspective, the key question is whe-
determine the appropriate response for an underperforming assay: ther results are reliable and can be released. Thus, traditional QC fo-
continuous improvement vs replacement. cuses on the current time. Quality is achieved by inspection: unreliable
The stability-capability matrix can also direct attention to the most results are identified and removed; reliable results are retained. This
urgent assays. The matrix shows which assays are furthest from the approach achieves high quality but can be very inefficient. More re-
capable/stable quadrant. The relative position of assays should provoke cently, laboratories have begun to use approaches such as 6 sigma
a conversation as to what would be required to move an assay from one which focus on continuous improvement (variance reduction). These
of the underperforming quadrants to the capable/stable quadrant. If the approaches seek to insure quality by building high quality into the
issue is capability, the laboratory needs to assess whether the issue is process rather than inspecting poor quality out. The continuous im-
bias or imprecision and determine the costs associated with improving provement approach takes a retrospective view of QC data and analyzes
each of these parameters. These determinations are not easy but the long-term performance. The stability metrics that we propose are de-
stability-capability matrix provides a framework to discuss and prior- signed to support the continuous improvement approach.
itize improvements. Stability analysis was first introduced in 2007 [9]. We are aware
The capability-stability matrix provides a number of advantages that stability analysis is used by companies in the food, biopharma-
over traditional analyses. First, it provides a simple visual summary of ceutical, chemical and semiconductor manufacturing industries. Stabi-
two important performance metrics: stability and capability. Second, lity analysis has also been incorporated into a popular statistical pro-
the matrix suggests the appropriate managerial action (monitoring, gram, JMP. Thus, it appears that stability analysis is being adopted in a
improvement, redevelopment) depending upon the quadrant in which wide range of industries. Although stability analysis has not been ap-
an assay resides: capable and stable assays should be monitored, un- plied in laboratory medicine, we believe it is a potentially useful too
stable assays should be stabilized, and stable incapable assays should be that deserves consideration.
replaced or redeveloped. Finally, stability analysis provides a quanti- We showed 2 examples (MTX and TSH) of stability analysis. Both of
tative metric. This enables management to prioritize assays for im- these assays are immunoassays. We have also conducted stability ana-
provement and to evaluate improvement. The combination of stability lysis on a approximately 35 different assays performed by liquid
and capability on a single chart facilitates this type of analysis. Levy- chromatography/mass spectrophotometry (LC-MS/MS). We found a
Jennings charts provide qualitative information on stability but quali- wide range of stability results among this group of assays [18].
tative analyses are less informative than quantitative metrics. Our study is limited because stability metrics are relatively new and
Improvement efforts should not be based on the results of a statis- have not achieved widespread adoption. We are currently experi-
tical test. A statistical test such as the SR test can only show whether a menting with these statistics in our laboratory. At present they appear
result is statistically significant. Managerial judgement is required to to provide useful information regarding assay performance and we use
determine whether a statistically significant result is practically sig- these metrics to prioritize assays for quality improvement. In summary,
nificant. Retrospective review of QC results can provide large data sets. we have described simple metrics that can be used to assess assay
Large data sets are likely to produce statistically significant results. For performance and to help identify assays in need of potential improve-
example, the TSH assay showed a statistically borderline s t difference ment.
between long-term and short-term variation (p = .06); however, the
difference in variation was only 3%. Although the difference is statis- References
tically significant, it is not clear that a 3% reduction in variance would
be clinically meaningful. On the other hand, MTX showed differences [1] D.J. Wheeler, Understanding Variation: The Key to Managing Chaos, SPC Press,
between short-term and long-term variation that are statistically sig- 2000.
[2] H.V. Roberts, R.S. Tsay, Making control charts more effective by time series ana-
nificant and, most likely, clinically meaningful. Further, given clinically lysis: three illustrative applications, Commun. Stat. Theor. Meth. 25 (11) (1996)
meaningful differences, managerial judgement is required to prioritize 2767–2796.
improvement efforts. Some assays are easier to improve than others. [3] D.C. Montgomery, Introduction to Statistical Quality Control, John Wiley & Sons,
New York, 2009.
This type of assessment requires sophisticated knowledge of the assay. [4] D.J. Wheeler, D.S. Chambers, Understanding Statistical Process Control, SPC Press,
Overall, stability analysis can be a useful tool for prioritizing efforts, but 2010.
is not a substitute for managerial judgement and consideration of other [5] J.S. Oakland, Statistical Process Control, Routledge, 2007.
[6] M.G. Leitnaker, R.D. Sanders, C. Hild, The Power of Statistical Thinking: Improving
important factors such as patient impact. Industrial Processes, Addison-Wesley Publishing Company, 1996.
Laboratories are increasingly using capability as a metric to assess [7] T.P. Ryan, Statistical Methods for Quality Improvement, John Wiley & Sons, 2011.
assay performance. Stability metrics are related to capability because [8] E.N. Cruthis, S.E. Rigdon, Comparing two estimates of the variance to determine the
stability of a process, Qual. Eng. 5 (1) (1992) 67–74.
an increase in stability (variance reduction) will improve capability.
[9] B. Ramirez, G. Runger, Quantitative techniques to evaluate process stability, Qual.
Stability analysis is not limited to QC results. Stability analysis can Eng. 18 (1) (2006) 53–68.
be applied to any process with a quantitative result. In general, a [10] R.A. Boyles, Estimating common-cause sigma in the presence of special causes, J.
quality improvement program should look upstream to identify and Qual. Technol. 29 (4) (1997) 381–395.
[11] D.J. Wheeler, Advanced Topics in Statistical Process Control: The Power of
control sources of variation. Critical inputs to an assay (preanalytical Shewhart's Charts, SPC Press, 2004.
variables) should be charted and analyzed for stability. This type of [12] R.W. Amin, R.A. Ethridge, A note on individual and moving range control charts, J.
analysis can provide data to identify causes of QC failures. Stability Qual. Technol. 30 (1) (1998) 70–74.
[13] W.J. Braun, D. Park, Estimation of σ for individuals charts, J. Qual. Technol. 40 (3)
analysis can also be applied to post-analytical processes.

32
R.L. Schmidt, L.N. Pearson Clinica Chimica Acta 490 (2019) 28–33

(2008) 332–344. too wide? Clin. Chim. Acta 486 (2018) 329–334.
[14] S.E. Rigdon, E.N. Cruthis, C.W. Champ, Design strategies for individuals and [19] Y. Wooluru, D. Swamy, P. Nagesh, Approaches for detection of unstable processes: a
moving range control charts, J. Qual. Technol. 26 (4) (1994) 274–287. comparative study, J. Mod. Appl. Stat. Methods 14 (2) (2015) 17.
[15] L.S. Nelson, Control charts for individual measurements, J. Qual. Technol. 14 (3) [20] W. McNeese, The Estimated Standard Deviation and Control Charts, https://www.
(1982) 172–173. spcforexcel.com/knowledge/control-chart-basics/estimated-standard-deviation-
[16] H.M. Wadsworth, K.S. Stephens, A.B. Godfrey, Modern Methods For Quality Control and-control-charts, (2012) (accessed May 25.2018).
and Improvement, Wiley, 2002. [21] P. Qiu, Introduction to Statistical Process Control, CRC Press, 2013.
[17] J.D. Cryer, T.P. Ryan, The estimation of sigma for an X chart: or S/c 4? J. Qual. [22] S.H. Steiner, R.J. MacKay, Statistical Engineering: An Algorithm for Reducing
Technol. 22 (3) (1990) 187–192. Variation in Manufacturing Processes, ASQ Quality Press, 2005.
[18] R.L. Schmidt, B.S. Walker, L.N. Pearson, Quality control limits: are we setting them

33