Method Validation and

Verification

Lauren N. Pearson, DO, MPH

Laboratory Director, University of Utah Health Sciences Center Clinical Laboratories AUGUST 2020
Objectives
Identify the difference between method validation and method
verification

Describe the studies required to document method

performance

Interpret method performance data and statistical data

outcomes

2
Outline

Context and definitions

Regulatory requirements
Studies required for analytical verification and analytical validation

3
Context and Definitions

4
Validation versus Verification
Validation
» Establishing the performance specifications of a new diagnostic tool such as a new test, laboratory
developed test or modified method

Verification
» A one-time process to determine performance characteristics of a test before use in patient testing

5
Why Evaluate a Method?
• Document initial performance:

» Reference when troubleshooting problems

» Quality assurance - to ensure results

» Helpful for clinical consultations

» Meet regulatory requirements

6
Laboratory Regulations
• General and open to some interpretation

• Direct what must be done, not “how” it is accomplished

7
U.S. Test Categorization
• Determined during FDA pre-market approval

• Waived testing
» Approved for home and point-of-care use
» “Low risk of patient mismanagement if performed incorrectly”
• Non-waived testing
» Moderate Complexity
» High Complexity e.g. LDTs
» Modified Tests
https://wwwn.cdc.gov/clia/resources/testcomplexities.aspx. Accessed February 12, 2018.

8
Regulatory Requirements

9
Waived Tests
Labs have only 3 requirements!
• Pay biennial fee (every 2 years) for CLIA certificate renewal
• Follow manufacturers instructions for use
• Allow the laboratory to be inspected
̶ Generally, for cause (patient complaint)
̶ Random state survey
̶ Periodic inspections not required!

Note: No method evaluation required

10
Nonwaived Tests

11
 Method “Validation” to CLIA
Moderate Complexity High Complexity
» Precision » Precision
» Accuracy » Accuracy
» Reportable Range » Reportable Range
» Reference Range(s)
» Reference Range
» Analytical Sensitivity (LOD)
» Analytical Specificity
• Mnemonic: PARR
» Establish calibration and
control procedures
» Other performance criteria
Halling KC, Schrijver I, Persons DL. “Test Verification and Validation for Molecular Diagnostic Assays. Arch Pathol Lab Med. 2012;136:11-13.
Nichols JH. “Verification of Method Performance for Clinical Laboratories”. Advances in Clinical Chemistry. 2009;47:121-138.

12
Test Modifications
Any change in the intended use or change to an assay that could affect
performance:
• Different sample matrix (urine in a serum assay)
• Promoting different use (screen vs diagnosis)
• Type of analysis (qualitative vs quantitative)
• Incubation times and temperatures
• Sample or reagent dilution
• Using different calibration material or set-point
• Change or eliminating a procedural step

13
Analytical Verification/Validation
• Laboratories are required to perform analytical verification or validation of each nonwaived test,
method, or instrument system before use in patient testing
̶ Regardless of when it was first introduced by the laboratory
̶ Includes instruments of the same make and model and temporary replacement (loaner) instruments

• There is no exception for analytical verification or validation of tests introduced prior to a specific date

• The laboratory must retain records as long as the method is in use and for at least two years after
discontinuation

14
How to Meet the Regulations
• There is no one right way
• Consensus - CLSI protocols
• Literature - do what others have done
• Manufacturer’s recommendations
• Balance between cost and what is reasonable

15
Validation Studies

16
Accuracy
Bias to a “reference” method
̶ Absolute
̶ Relative

17
Accuracy Studies
Method comparison
» Carefully select “reference” method
» Curate high quality samples with a range of analyte concentration
» Analyze >40 specimens by both test and reference method
 Best to analyze in duplicate over a period of many days

Data analysis
̶ Scatter plot of data
̶ Calculate regression statistics and estimate bias
̶ Compare results with claims or internal criteria to judge acceptability

CLSI EP-09

18
 192 specimens
2 lots of reagent and calibrator

19
Special Considerations
• Medical decision points
• Clinically relevant cutoffs

20
21
22
Precision
• Within-run (Intra-assay)
• Between-run
• Day-to-day (Inter-assay or total)

23
Precision Studies
• Selection of appropriate material

• Verification study
̶ 5 x 5 study design

• Full precision study

 Within run
̶ 20 consecutive replicates/single run
 Total
̶ 2 replicates/concentration level/run
̶ 2 runs/day x 20 days
• Data analysis
̶ Calculate mean, SD, and CV
̶ Compare results with claims or internal criteria to judge acceptability

24
3 controls run twice per day in duplicate x for 20 days

25
Utility of Precision Data
• Future troubleshooting

• Clinical queries about significant change

• Setting QC ranges

26
Reportable Range
Includes:
• Analytical measurement range (AMR)
» Range of values that an instrument can report directly without alteration
or pretreatment of the sample

• Clinically reportable range (CRR)

» Range of values that can be reported with alteration of the sample
 Medical director discretion

27
Range for Reporting Patient Samples
• May use the AMR
• May modify AMR to create expanded range (CRR)
» Must document that modifications to the sample and method produce
reliable results
 Verify every 6 months
» Must be verified or established before patient testing begins
» Must establish reportable limits (undiluted) and maximum dilution

28
Reportable Range Studies
• Linearity Study
• 5+ concentrations of analyte throughout range
• Spike low sample with known amount of analyte
• Dilute high sample with a blank
• Mix high and low sample to create a curve
• Standard reference materials
• Commercial linearity products
• Two replicates at each level
• Data analysis
o Evaluate linear fit with XY plot
o Calculate slope and intercept
CLSI EP-6

29
Pooled patient serum sample and calibrator A mixed to get 6 sample concentrations

Run in triplicate on each Architect instrument

Both instruments demonstrated linearity of the assay consistent with manufacturer’s claims

30
Reference Intervals (RI)
• Labs are not required to establish their own
» Good practice is to verify that RI is appropriate for patient population
• Can use previously established RI or create a new one
» Discretion of the medical director
• Transfer of a RI is acceptable if test subject population and
methodology are the same or comparable
» Verified by testing N ≥ 20 samples
» If ≤ 2 outside limits, then accept

31
Establishing a RI
• Typically the central 95% of the values for the study population
• Considerations
• Exclusion criteria
• Partitioning
• Pre-analytical considerations
• Specimen handling and storage
• Special or unique patient populations

32
Protocol for Full RI Study
• Establish selection criteria for individuals
• Establish a list of interferences or sources of biological variability
• Decide on appropriate number of individuals based on desired
confidence limits (e.g. n=120)
• Collect and analyze specimens
• Evaluate data using histogram to evaluate distribution

CLSI EP28A3C

33
Samples from 20 apparently healthy donors into PST and SST tubes

Donor exclusion criteria

Samples had PCT concentrations of 0.01 to 0.03 ng/mL confirming the manufacturer’s claims

34
Analytical Sensitivity
• Establishes the analytical sensitivity (lower detection limit) of the
assay

• For modified FDA-cleared/approved tests or laboratory-

developed tests (LDTs)

35
Analytical Sensitivity Studies
• Acquire measurements from multiple, independent blank and low-
level samples or pools of samples
» At least four samples of each type
» Can dilute or spike samples to provide low level samples at desired
analyte levels
» Low level sample around assumed LoD
» Obtain a series of replicate results
• Data analysis
» Parametric or nonparametric statistical methods

36
Limit of Blank determination:
Calibrator A (concentration 0 ng/mL) analyzed 10 times on each instrument
Calibrator B (concentration 0.1 ng/mL) analyzed 3 times on each instrument

Limit of Quantitation determination:

8 calibrator samples including 4 low level concentrations analyzed over 10 days

Results were equivalent to the manufacturer’s claims

37
Analytical Specificity (Interferences)
• Refers to the ability of a test or procedure to correctly identify or
quantify an entity in the presence of interfering or cross-reactive
substances

• For modified FDA-cleared/approved tests or laboratory-

developed tests (LDTs)

38
Interfering Substances (IFS)
• Interference- a significant difference in test result because of another
component of the sample

• Interfering substance- a substance that causes the measurement to be

inaccurate

• Can cause a concentration dependent difference in the test

• Manufacturers screen for IFS during method development

39
Identifying Error from IFS
Quantify effects by performing paired difference study:
• Pairs of test samples
• One with potential IFS, the other without
• Measure analyte of interest
• Calculate differences
• May be performed with patient samples

CLSI EP07

40
Summary
• Regulations require performance verification prior to patient
testing
• Precision, accuracy, reportable range and reference interval
must be evaluated, at a minimum, for all nonwaived tests
before patient use
• No “one size fits all” approach to validation/verification

41
References
• Burtis, CA, Ashwood ER. (editors) Tietz Textbook of Clinical
Chemistry, Third Edition. 1999, WB Saunders Co. Philadelphia PA.
• Clinical and Laboratory Standards Institute. Evaluation Protocols.
Various publication dates. Wayne, PA.
• HCFA 42 CFR; Final Rule; Medicare, Medicaid and CLIA Programs;
Regulations implementing Clinical Laboratories Improvement
Amendments of 1988 (CLIA. Fed. Regist. 57;7001-288 (28 February
1992)
• College of American Pathologists Commission on Laboratory
Accreditation. Inspection Checklist. 2016 CAP, Northfield, IL.
• Gruzdys V et al. 2019. Method Verification Shows a Negative Bias
between 2 Procalcitonin Methods at Medical Decision
Concentrations. JALM; 4(1):69-77.

42
© 2020 ARUP LABORATORIES