LoD Estimation Methods FINAL 29Jul2024A
LoD Estimation Methods FINAL 29Jul2024A
ABSTRACT
In assay performance evaluation for quantitative assay studies based on polymerase chain reaction (PCR), the Limit
of Detection (LoD) is defined as the lowest concentration or amount of analyte that is consistently detectable (in our
case, in at least 95% of the samples tested; CLSI EP17-A2). In practice, the estimation of the LoD uses a parametric
curve fit to a set of panel member (PM1, PM2, PM3, etc.) data where the responses are binary (i.e., percent detected
at a certain level). Typically, the parametric curve fit to the percent detection levels takes on the form of a logistic or
probit distribution. Additional methods for estimation of the LoD in PCR-based studies include using the “based on hit
rate” and maximum likelihood estimation (MLE) methods (Singh & Nocerino, 2001). Amongst all methods, the MLE
method is preferred since the MLE is taken to be sufficient (Rice, 1988) given the selected parent probability distribution
assumed to model the data. Moreover, it can be shown that the MLE for a Poisson-distributed random variable has
the minimum variance unbiased estimator (MVUE) property (Cassella and Berger, 2002), thus allowing it to be used
as a reference in comparison with other methods. We performed a comparison of the LoD calculation methods using
various realizations of a real seven-member panel for HIV-1 PCR assay results.
___________________
* HBV=Hepatitis B Virus, HCV=Hepatitis C Virus, HIV=Human Immunodeficiency Virus.
** MRSA=Methicillin-resistant Staphylococcus aureus, MAP=mycobacterium avium subspecies paratuberculosis.
In the experiment, the levels are chosen to include at least one panel member at 100% detection, another to anchor
the parametric curve at the bottom end (not including zero) with the remaining three or more levels targeting the
1
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
region where one believes the LoD might be. Table 1 shows an example of six panel levels (not including zero) with
the top end anchored by an HCV RNA assay level at 50 IU/mL and one anchor at the bottom at 2.5 IU/mL. There are
four levels in between both the top and bottom ends: 25, 15, 10 and 5 IU/mL.
Table 1. LoD in EDTA Plasma from “Empower change in HCV” for COBAS® AmpliPrep/COBAS® TaqMan®
HCV Qualitative Test, v2.0
An Aside: A related but natural question then becomes, “How many samples shall one test at each level?” The
answer, of course, usually depends on the purpose. If one is performing first-time development work for determining
positioning of the LoD for future targeting, one needs only a minimum amount of testing for each level (say, ~20 samples
per level with possibly more at the lower end if we expect “dropouts” or samples that do not all quantitate at each level).
On the other hand, if the regulatory agency requires testing with material that is in short supply (e.g., World Health
Organization (WHO) material), or is expensive, the researcher may, by default, be forced to use the same minimum
number of samples per level as in the development case. Furthermore, considerations for level sample sizes are also
made in order to more fully characterize the true LoD. That is, the more sample size one uses at each level, the tighter
the confidence interval about the estimated LoD. Table 1 shows that the experiment used about 252 samples for each
level. However, typical experiments will have 40 samples or more per level. Note that the mention of “sample” in this
discussion can be taken to be, for example, a contrived sample that is diluted to its specific level from a common pool
of inactivated analyte.
Figure 1 shows the results of a parametric curve fit to the six-member panel “hit rate” data using the probit distribution
with panel concentrations transformed using the logarithm base 10 (i.e., log10) scale. The estimated LoD is 11.7 IU/mL
which rounds to 12 IU/mL as indicated at the bottom of the published Table 1. The goodness-of-fit (GOF) statistics
show statistical non-significance thereby indicating a good fitting model (Pearson chi-square=2.9, p-value=0.58 and LR
chi-square=3.1, p=0.55). Note that the panel member with zero IU/mL concentration is not used in the calculations as
the log10 of zero is undefined. Moreover, in PCR-type studies, we can momentarily define the target response as binary
(i.e., detected or not detected) so that there is no background “noise” that needs to be measured in estimating the LoD.
Therefore, the Limit of Blank (LoB; CLSI EP17-A2) has no intrinsic meaning in PCR testing so we exclude any further
discussion about the LoB.
Of course, the parametric probit curve fit method is one of several that will be discussed in this paper. We include the
parametric logistic curve fit, the method of percent detection and the method of maximum likelihood using a Poisson
probability distribution to model the observations in the vicinity of the LoD. The reader should note that these methods
are not exhaustive.
2
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
data LoD ;
input Target $3. Titer Hits TotalTests @ ;
Log10_Titer = log10(Titer) ; Obs_HitRate = Hits / TotalTests ;
Obs_HitPcnt = Obs_HitRate * 100 ;
datalines ;
HCV 50 252 252
HCV 25 251 252
HCV 15 246 251
HCV 10 233 252
HCV 5 183 252
HCV 2.5 124 251
; run ;
Next, we load our LoD5 SAS macro and specify the macro call (for details, please see our SAS paper introducing the
LoD5 SAS macro):
%include "<<your path>>/LoD5.sas" ;
%LOD5 (
datain = LoD
,study = CTM_1
,target = &target
,subset = target = "HCV"
,units = IU/mL
,distn = Probit
,alpha = 0.05
,sided = 2
,TableNo = 1
,FigureNo = 1
,ODStype = RTF
,RTFout = <<your path>>/LoD_HCV_CTM_Probit.RTF
,GraphMin = -0.5
,GraphMax = 3.0
,GraphByInc = 0.5
,GraphTitle1 = Figure 1 Limit of Detection Estimation for HCV Target
,GraphTitle2 = Probit Curve Fit
,YaxisTitle = Hit Rate
,XaxisTitle = Titer (log10 &Units) ) ;
This SAS code produces the graph in Figure 1 (without the additional annotation) where the LoD is estimated to be
11.7 IU/mL, with 95% CI from 10.3 to 13.6 IU/mL, using the probit curve fit. In addition, the SAS code generates a
percent detection table (aka “hit rate” table) as shown in Table 2 with associated 95% (Clopper-Pearson) exact
confidence intervals for the panel level hit rates. From Table 2, one can see that the estimated LoD should be between
10 and 15 IU/mL (second column) since the hit rate at those levels is between 92.4% and 98.0%, respectively (column
6).
Table 2. Percent Detection Output Results using the LoD5 SAS Macro for Data in Table 1 with Associated
95% Exact (Clopper-Pearson) Confidence Intervals for Panel Level Hit Rates.
Titer Titer No. Total Percent
Target (IU/mL) (log10 IU/mL) Detected Tests Detection 95% Exact LL 95% Exact UL
50 1.6990 252 252 100 98.5 100
25 1.3979 251 252 99.6 97.8 100
15 1.1761 246 251 98.0 95.4 99.3
HCV
10 1.0000 233 252 92.4 88.4 95.3
5.0 0.6990 183 252 72.6 66.6 78.0
2.5 0.3979 124 251 49.4 43.0 55.7
HCV=Hepatitis C Virus, IU=International Units, LL=Lower Limit, UL=Upper Limit
3
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
To obtain a logistic curve fit to the hit rate data, one needs only to change the distribution (viz., distn) portion of the
LoD5 SAS macro as shown above to “distn = Logistic”. The results of which are shown in Figure 2.
Again, we see that the LoD estimate is 11.7 IU/mL – the same as when
using the probit fit. However, the 95% CI for the LoD is slightly different:
10.2 to 13.9 IU/mL for the logistic fit as compared to 10.3 to 13.6 IU/mL
using the probit fit (see Figure 1), although the GOF statistics are better
for the probit than for the logistic curve fit (viz., Pearson chi-square=7.8,
p-value=0.1 and LR chi-square=8.8, p-value=0.07 (see Figure 2).
This SAS code produces the estimated LoD as 11.5 IU/mL with 95% CI from 10.5 to 12.5 IU/mL.
4
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
Figure 4 shows the graphical results for panel levels with n=40 at each level. For example, the graph shows 35 blue
open-circles representing the LoD estimates for all three level combinations out of the seven total levels.
Mathematically, there should be (73) (i.e., 7 choose 3) or 35 possible combinations [e.g., (1,2,3), (1,2,4),
(1,2,5),…,(5,6,7)] where the LoD is estimated using Probit for each of the 35 combinations of size 3. Similarly, the red
crosses represent the LoD estimates for (74) or all 35 possible combinations of size 4. Next, the green Xs represent the
(75) or 21 possible panel level combinations of 5 from 7 levels. It is easy to see that the brown triangles and purple
square represent the remaining panel level combinations of (76)=7 and (77)=1, respectively. Finally, the ML estimate is
calculated using the full 7-member panel and is shown on Figure 4 as a violet line at 10.5 IU/mL with associated 95%
CI at 8.2 cp/mL (green line at the lower limit) and 12.7 cp/mL (red line at the upper limit).
5
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
DISCUSSION
TABLE 1 DATA
Table 3 shows the summary of the various LoD estimation methods shown above when using the data in Table 1. If
we take the MLE of 11.5 IU/mL to be the reference or “true” value, then the other methods can be compared in
performance using the difference of the estimate for that method with that of the MLE (i.e., the bias). The last column
in Table 3 indicates that the least bias comes from the Probit and Logistic methods (bias: 0.2 IU/mL) followed by the
Extreme Value and Gompertz methods (bias: 0.3 IU/mL). The Hit Rate method produces the largest bias (bias: 3.5
IU/mL).
The second to the last column of Table 3 shows the 95% confidence interval length. The narrowest interval comes
from the ML method (95% CI Length: 2 IU/mL) whilst the largest comes from the Logistic method (3.7 IU/mL).
Table 3. Limit of Detection and associated 95% CI using Six Methods.
LoD 95% LL 95% UL Interval Bias
Method From MLE
(IU/mL) (IU/mL) (IU/mL) Length
Probit 11.7 10.3 13.6 3.3 0.2
Logistic 11.7 10.2 13.9 3.7 0.2
Extreme
11.8 10.6 13.5 2.9
Value 0.3
Gompertz 11.8 10.6 13.5 2.9 0.3
Hit Rate 15.0 --- --- --- 3.5
ML 11.5 10.5 12.5 2 ---
The Table 4 LoD standard deviations (SD) buttress the equivalent Figure 4 graph in terms of the dispersion of
individual LoD estimated values for each combination of levels (viz., 3, 4, 5, 6 and 7). Interestingly enough, the
average bias levels seem to be optimal (for this example) when n=50 per level. Of course, this determination
6
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
assumes that the MLE is the “true estimate” of the LoD. Moreover, within each of the levels (for any sample size), the
least bias occurs when choosing four levels to estimate the LoD.
CONCLUSION
The results using different methodologies for estimating the Limit of Detection for one example of hit rate data in
PCR-type studies showed that, using the Poisson distribution to model observations in the vicinity of the LoD where
the distribution is not likely to be Normal, and given that the MLE is MVUE, one can use the MLE as the reference to
compare with parametric methods with respect to the bias and confidence interval lengths. We showed that the LoD
methods presented were biased with the least biased LoD estimation methods of those presented were those using
the Probit and Logistic model fit to the hit-rate data. When consideration is for how many levels to use in the LoD
experiment at hand, a small performance evaluation study using the Probit model with average bias from the MLE
showed the average LoD/bias combined optimal number to be 6 levels (SD=0.4, bias= -0.5 at n=50 per level).
7
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
REFERENCES
1. Canchola JA, Hemyari P. 2016. “Limit of Detection (LoD) Estimation Using Parametric Curve
Fitting to (Hit) Rate Data: The LoD_Est SAS Macro.” Proceedings of the SAS Global 2016
Conference held in Las Vegas, NV USA. Cary, NC: The SAS Institute.
2. Canchola JA, Vaks JE, Tang S (2019). “Limit of Detection (LoD) Estimation Using Maximum
Likelihood from (Hit) Rate Data: The LoD_MLE SAS Macro.” Proceedings of the Western Users
of SAS Software Annual Conference, September 2019, Seattle, WA.
3. Casella G, Berger RL (2002). Statistical Inference. 2Ed. pp. 334-339. Duxbury Thomson
Learning. Pacific Grove, CA: Wadsworth Group.
4. Clinical and Laboratory Standards Institute (CLSI) document EP05-A2. “Evaluation of Precision
Performance of Quantitative Measurement Methods; Approved Guideline--Second edition.”
Wayne, PA. 2004.
5. Clinical and Laboratory Standards Institute (CLSI) document EP06-A. “Evaluation of the Linearity
of Quantitative Measurement Procedures: A Statistical Approach; Approved Guideline.” Wayne,
PA. 2003.
6. Clinical and Laboratory Standards Institute (CLSI) document EP17-A2. “Evaluation of Detection
Capability for Clinical Laboratory Measurement Procedures; Approved Guideline--Second
edition.” Wayne, PA. 2012.
7. Hogg RV, McKean JW & Craig, AT (2013). Introduction to mathematical statistics. Pearson
Education India.
8. Johnson NL, Kemp AW & Kotz S (2005). Univariate discrete distributions (Vol. 444). John Wiley &
Sons.
9. Lehmann EL & Casella G (2006). Theory of point estimation. Springer Science & Business
Media.
10. Mood AM, Graybill FA and Boes DC (1974). Introduction to the Theory of Statistics. McGraw-Hill.
11. Pawitan Y (2013). In All Likelihood: Statistical Modelling and Inference Using Likelihood.
Oxford, United Kingdom: Oxford University Press. 528 pp.
12. Purcell S (2007). “Maximum Likelihood Estimation”. Accessed 05Jun2015. Available at:
http://statgen.iop.kcl.ac.uk/bgim/mle/sslike_3.html
13. Rice JA (2007). Mathematical statistics and data analysis (Vol. 371). Belmont, CA:
Thomson/Brooks/Cole.
14. Roche Molecular Systems, Inc. 2011. “Empower change in HCV” for COBAS®
AmpliPrep/COBAS® TaqMan® HCV Qualitative Test, v2.06. Accessed on 05Jun2015. Available
at: http://www.roche-diagnostics.ch/content/dam/corporate/roche-
dia_ch/documents/broschueren/molecular_diagnostics/virology/06611656001_EN_EA_COBAS-
AmpliPrep_COBAS-TaqMan-HCV-Qualitative-Test-v2.0.pdf
15. Singh A & Nocerino J. 2001. “Robust Estimation of Mean and Variance Using Environmental
Data Sets with Below Detection Limit Observations”, U. S. Environmental Protection Agency
internal article. Clearance Number 01-062. Washington, DC: U. S. EPA.
CONTACT INFORMATION
Your comments and questions are valued and encouraged. Contact the first author at:
Name: Jesse A. Canchola
Company: Roche Molecular Systems, Inc
Address: 4300 Hacienda Drive
8
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS
Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are
trademarks of their respective companies.
APPENDIX A
N=40 at each level.
9
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
10
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
11
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
12
Limit of Detection Calculation Methods Comparison for PCR-based Quantitative Studies using the SAS® System, continued
13