Method Evaluation Workshop

10th June 2024

Scenario

Your analyser manufacturer is changing method for gentamicin

You have been asked to lead on the verification of the new method

• This is intended to replace an existing method

• Both methods are run on the same random-access automated platform

Questions:

1. Discuss and decide in your group which experiments are required for
this verification.
2. Consider which samples or other material are required, how you will
obtain appropriate samples for this verification and any issues you
anticipate.
3. Decide what resources you would need for this verification.
4. Making use of the background information provided, decide on
appropriate acceptance criteria for precision and method comparison.
5. Calculate Grubbs’ limits for the imprecision data and determine if any
replicate is an outlier.
6. Determine the upper verification limit for repeatability and within-
laboratory imprecision. Decide if imprecision is acceptable.
a. If accepted, consider if mitigations are required and what these
would be.
b. If not accepted, consider what potential actions or further
experiments are required.
7. Calculate the expanded uncertainty from within Laboratory
Imprecision (ignoring any calibration uncertainty) and compare this to
the Royal College of Pathologists of Australasia Acceptable
Performance Limits.
8. Review raw data, charts and summary statistics for the method
comparison study. Compare the data to your acceptance criteria and
decide whether this is acceptable.
a. If accepted, consider if mitigations are required and what these
would be.
b. If not accepted, consider what potential actions or further
experiments are required.
9. Use the raw data to determine concordance (PPA, PNA, POA) between
the methods at a cut-off of 5 mg/L.
10. Decide an overall conclusion including potential clinical significance of
changing to the new method.
Background Information:

Gentamicin once daily dosing interval nomogram:

From: https://www.antimicrobialcompanion.scot/nhs-lothian/vancomycin-gentamicin-
other-antibiotics-dosing-guidance/gentamicin-adult-standard-dosing/
Information from the new assay Manufacturer’s Instructions for Use (IFU):
Current assay QC performance:
EQA Method Comparison:

From: UKNEQAS Birmingham Quality Antibiotics Scheme

Measurement uncertainty targets from different organisations:

Toxicology CLIA 2024

Analyte or Test MU Target (TV = True Value)
Acetaminophen TV ± 15% or or ± 3 mcg/mL (greater)
Alcohol, blood TV ± 20%
Blood lead TV ± 10% or 2mcg/dL (greater)
Carbamazepine TV ± 20% or ± 1.0 mcg/mL (greater)
Digoxin TV ± 15% or ± 0.2 ng/mL (greater)
Gentamicin TV ± 25%
Lithium TV ± 15% or ± 0.3 mmol/L (greater)
Phenobarbital TV ± 15% or ± 2 mcg/mL (greater)
Phenytoin TV ± 15% or ± 2 mcg/mL (greater)
Salicylate TV ± 15% or ± 2 mcg/mL (greater)
Theophylline TV ± 20%
Tobramycin TV ± 20%
Valproic acid TV ± 20%
Vancomycin TV ± 15% or ± 2 mcg/mL (greater)
From: https://westgard.com/clia-a-quality/quality-requirements/1002-2024-clia-
requirements.html
From: https://rcpaqap.com.au/resources/chemical-pathology-analytical-performance-specifications/

Lower limit applies to values equal to or below the quoted concentration and upper limit applies to
concentrations above the quoted concentration
Imprecision Data:

Run
1 2 3 4 5
Replicate

1 7.3 8.5 9.2 8.0 7.5

2 6.9 8.8 8.7 7.5 7.9
3 7.8 8.3 9.0 7.7 7.7
4 7.2 8.9 9.0 7.5 7.6
5 7.0 8.4 9.3 7.5 7.8

Mean 7.2 8.6 9.0 7.6 7.7

SD 0.4 0.3 0.2 0.2 0.2
% CV 5.6 3.5 2.2 2.6 2.6

Overall Mean 8.0

Overall SD 0.7
Overall CV 8.8
G 3.125
Grubbs' limits = Mean ± (SD*G)
Instructions for Upper verification limit (UVL):
UVL = manufacturer’s CV*F

First determine degrees of freedom for repeatability and within-laboratory imprecision:

Repeatability:

degrees of freedom (dfR) = N – k

N = total number of results and k = number of runs

Within-laboratory imprecision:

Claims ratio (ρ) = Manufacturer’s within-Lab CV (or SD)/repeatability CV (or SD) – see kit
insert

Use this to find degrees of freedom (dfWL) in table 1

Second, determine F:

Use the dfR and dfWL to find F in table 2 (assume only 1 sample)

Finally, determine separate UVL values for repeatability and within-laboratory imprecision:

UVL = manufacturer’s CV*F

TABLE 1: dfWL as a function of the claim’s ratio

From CLSI EP15-A3 User Verification of Precision and Estimation of Bias; Approved Guidelines – Third
edition.
TABLE 2: UVL factors (F) as a function of DF and number of samples

From CLSI EP15-A3 User Verification of Precision and Estimation of Bias; Approved Guidelines – Third
edition.
Measurement Uncertainty (MU):

Calculation (assuming imprecision is the only major contributor to

uncertainty):

Expanded uncertainty = k * within-laboratory imprecision

K is 1.96 for a 95% confidence level

Method Comparison
Sample Current Method New Method Difference % Difference
Rank (mg/L) (mg/L) (mg/L) from current
1 0.08 0.4 0.32 400.0
2 0.15 0.7 0.55 366.7
3 0.53 1.1 0.57 107.5
4 0.65 1.3 0.65 100.0
5 0.67 1.3 0.63 94.0
6 0.74 1.2 0.46 62.2
7 0.78 1.3 0.52 66.7
8 1.1 1.8 0.7 63.6
9 1.13 1.9 0.77 68.1
10 1.36 2 0.64 47.1
11 1.65 2.3 0.65 39.4
12 1.72 2.6 0.88 51.2
13 2.66 3.6 0.94 35.3
14 3 3.8 0.8 26.7
15 3.4 4.6 1.2 35.3
16 4.21 5.3 1.09 25.9
17 4.42 5.7 1.28 29.0
18 4.56 6.1 1.54 33.8
19 4.63 6 1.37 29.6
20 5.03 6.5 1.47 29.2
21 5.2 6.6 1.4 26.9
22 5.25 7.3 2.05 39.0
23 5.44 7 1.56 28.7
24 5.46 7 1.54 28.2
25 5.58 7 1.42 25.4
26 5.93 7.7 1.77 29.8
27 6.5 8.9 2.4 36.9
28 6.99 8.9 1.91 27.3
29 7.03 9.3 2.27 32.3
30 7.62 10.4 2.78 36.5
31 7.81 10.9 3.09 39.6
32 7.84 9.6 1.76 22.4
33 8.92 11.9 2.98 33.4
34 9.52 13.3 3.78 39.7
35 9.87 14.2 4.33 43.9
36 10.56 14 3.44 32.6
37 13.42 18.1 4.68 34.9
38 15.27 19.3 4.03 26.4
39 17.99 23.6 5.61 31.2
40 19.21 24 4.79 24.9
41 24.51 29.4 4.89 20.0
 40

35
y = 1.2457x + 0.4506
30 R² = 0.9948
New Method (mg/L)

25

20

15

10

0
0 10 20 30 40
Current Method (mg/L)

4
difference (mg/L)

0
0 5 10 15 20 25 30
-1
Gentamicin Concentration
 450.0
400.0
350.0
300.0
Difference (%)

250.0
200.0
150.0
100.0
50.0
0.0
0 5 10 15 20 25 30
-50.0
Gentamicin Concentration

4
Difference (mg/L)

0
0 5 10 15 20 25 30 35 40 45
-1
Sample rank
 450.0
400.0
350.0
300.0
Difference (%)

250.0
200.0
150.0
100.0
50.0
0.0
0 5 10 15 20 25 30 35 40 45
-50.0
Sample rank

mg/L %
Mean Difference 1.9 57.8
Mean difference ≤5 mg/L 0.9 85.6
Mean difference >5 mg/L 3.0 31.4

OLR equation y = 1.2457x + 0.4506

Concordance

Concordance is very similar to diagnostic accuracy but does not assume that
the comparator method is a gold standard or true diagnosis. It can be used to
determine clinical agreement between two tests. This is often used for
qualitative tests but can be used for a quantitative test at a set cut-off.

Comparative Method
Candidate Positive Negative
Method Positive a b
Negative c d

Percent Positive Agreement (PPA) = [a/(a+c)]*100

Percent Negative Agreement (PNA) = [d/b+d))]*100

Percent Overall Agreement (POA) = [(a+d)/(a+b+c+d)]*100