NCSL International Workshop & Symposium | Measurements of Tomorrow
August 27-30, 2018 | Portland Oregon
Risk-Based Thinking in the Calibration Laboratory: Practical
Examples
Speaker/Author:
Helga A. Alexander
International Accreditation Service, Inc.
3060 Saturn Street, Suite 100
Brea, CA 92821
Tel: 866-427-4422
[email protected]
Abstract
Many calibration laboratory managers, who plan to get their laboratories accredited or re-
accredited to the new ISO/IEC 17025:2017 standard, worry about how to address a new
requirement of the standard: considering the risks and opportunities associated with laboratory
activities. They may not realize that a well-run calibration laboratory practices risk-based-
thinking routinely. The purpose of this presentation is to remind laboratories that good laboratory
management practices already incorporate the consideration of risk and opportunities in their
laboratory activities. Practical examples of consideration of risks and opportunities in routine
laboratory activities such as calibration interval determination, out-of-tolerance investigations,
root cause analysis of nonconformity incidences, etc., will be discussed, and suggestions on how
to enhance, monitor and document such activities are addressed. Laboratory personnel will find
that they do not have to start from scratch in order to comply with this new aspect of ISO/IEC
17025, but they can leverage and improve upon best practices.
Learning Objectives
Familiarization with the new ISO/IEC 17025:2017 requirement to consider risks and
opportunities.
Understanding of calibration laboratory practices which already consider risks and
opportunities
Understanding of how to provide objective evidence to assessors to show that risks and
opportunities are being considered in laboratory activities
1. Introduction
The latest version of ISO/IEC 17025:2017, General requirements for testing and calibration
laboratories [1], was released on November 30, 2017. Similar to ISO 9001:2015, Quality
management systems—requirements [2], and other standards in the ISO/IEC 17000 and ISO
14000 series of standards, the new version introduces the concept of risk-based thinking. The
standard now requires that risks associated with laboratory activities be identified and addressed.
But the standard does not require that formal methods of risk management be employed, such as
1
NCSL International Workshop & Symposium | Measurements of Tomorrow
August 27-30, 2018 | Portland Oregon
those outlined in ISO 31000, Risk Management---Principles and guidelines [3]. But naturally, a
laboratory can consult ISO 31000 in order to learn more about the principles of risk
management.
The consideration of risks and opportunities is not a new concept to a well-run calibration
laboratory. In the following sections, we will explore how most laboratory managers already
practice risk-based thinking, although they may not be explicitly aware of it.
2. The new requirements
While risk is mentioned in almost every section of the newly structured standard, Clause 8.5 in
the management system requirements section is dedicated to actions addressing risks and
opportunities. This clause requires the laboratory to consider the risks and opportunities
associated with their activities. These activities are described throughout the standard and
include risks related to
impartiality (4.1.4),
statements of conformity (7.8.6),
nonconforming work (7.10.1) and
corrective actions (8.7.1).
3. Examples of risk-based thinking in the calibration laboratory
Let us explore typical calibration laboratory activities where we should or already do employ
risk-based thinking. While risk-based thinking applies to opportunities as well as risks, this paper
concentrates on the latter.
3.1 Risks with respect to impartiality
What do we mean by “impartiality” in the laboratory context? In clause 3.1, ISO/IEC
17025:2017 defines impartiality as “presence of objectivity.” The notes to this clause provide us
with examples of what the authors of the standard meant to convey, such as “freedom from
conflict of interest”, “freedom from bias”, “lack of prejudice”, “neutrality”, “fairness”, “open-
mindedness”, “even-handedness”, detachment”, “balance.” Let us try to think of some real-world
examples where impartiality could be a concern.
For example, let us say you are a calibration laboratory associated with a manufacturing facility,
and you serve both internal and external customers. It is important that you have policies in place
that ensure that both the interests of your external and internal customers are considered fairly.
Your policies, which need to be communicated and enforced by laboratory staff, need to ensure
that your external customers’ equipment does not languish indefinitely on your shelves waiting
to be calibrated, while you have an influx of internal work. Your policy may be “first in, first
out,” regardless of whose equipment it is, or you may be working toward a particular turnaround
time (TAT) which you declare to your customers during the contract review process. For
example, your policy may promise a TAT of 5 days to external customers for routine
2
NCSL International Workshop & Symposium | Measurements of Tomorrow
August 27-30, 2018 | Portland Oregon
calibrations, and 2 days to internal customers. This is perfectly acceptable, but it would not be
acceptable for one of your technicians to decide to delay an external customer’s calibration
because his friend on the production floor promised to buy him a drink for putting his piece of
equipment first on bench.
Another way to help ensure impartiality of laboratory activities is the organizational structure
itself. In the case of an internal calibration laboratory, it may be useful if laboratory management
did not report directly to the Director of Manufacturing, if their mission is to provide calibration
support to that section of the organization. Otherwise the laboratory manager may feel pressure,
real or imaginary, to consider product shipment targets rather than calibration history and
technical information when making calibration interval extension decisions, or when considering
what actions to take when a piece of equipment used on the production floor is found out of
tolerance.
Most important when considering risks to impartiality is to require periodic training of your
laboratory staff, both technical and administrative, on your policies on impartiality. To make the
concept less abstract, you should use real-world, relevant examples of how everyday actions of
staff could pose a risk to the laboratory’s impartiality. Do not assume that employees will
automatically make the connection between the clauses of the standard and their daily work
encounters.
3.2 Risks with respect to statements of conformity
The new version of the standard requires laboratories to document their decision rule if they
make statements of conformity, and to take into account the level of risk associated with the rule.
Section 3.7 of the standard defines “decision rule” as “rule that describes how measurement
uncertainty is accounted for when stating conformity with a specified requirement.” This means
that when a laboratory states that an instrument is in or out of tolerance, it has to consider how
their chosen decision rule affects the likelihood of calling a calibration results a “pass” when it is
really a “fail” (false accept), or calling something a “fail” when it is really a “pass” (false reject).
While both can have undesirable consequences for the end user of the instrument, this discussion
focuses mainly on the false accept risk.
While statements of conformity on calibration reports are not necessarily made routinely in every
part of the world, most calibration laboratories in the US provide such statements. When their
customers pay good money for a calibration, they expect to be able to see at a glance whether
their freshly calibrated instrument met all manufacturer’s specifications, or some other type of
specification laid out by an industry standard. However, in the past, the decision rule applied by
the laboratory may have been chosen for convenience to the laboratory, without any
consideration of the level of risk associated with the rule. The new version of the standard
requires laboratories to declare their decision rule to the customer up front. The reason for this is
that different decision rules have different false accept and false reject probabilities associated
with them, and customers need to be able to decide prior to purchasing a calibration service
whether the level of risk this presents is acceptable to their needs.
3
NCSL International Workshop & Symposium | Measurements of Tomorrow
August 27-30, 2018 | Portland Oregon
False accept risk levels associated with different kinds of decision rules are discussed in detail in
various publications; see, for example, references [4-11]. Decision rules oftentimes involve
guard banding based on measurement uncertainty as a means to limit the risk of making false
accept decisions. Guard banding is a method whereby acceptance limits for the calibration result
are more restrictive than the actual tolerance limit of the calibration item, thus changing the in-
or out-of-tolerance decision point. Different guard banding methods result in different levels of
false accept risks, which is why the standard requires that customers be made aware of which
method is employed. To illustrate this, we review a few guard-banding techniques.
3.2.1 No guard banding applied
False accept probabilities without guard banding have been published by previous authors, for
example [4-7]. The graphs published in these papers show that the false accept probability may
still be at an acceptably low level depending on the ratio of the instrument tolerance to the
expanded measurement uncertainty at 95% confidence level, commonly known as the test
uncertainty ratio (TUR) [12]. For example, assuming an in-tolerance probability of 80% for the
calibration item, a TUR of 4:1 provides a probability of false accept (PFA) of approximately 2%,
but for a TUR of 1.5:1, the PFA more than doubles.
3.2.2 Managed Risk Guard Band [4]
This method applies a guard band that is smaller than the expanded measurement uncertainty.
The author of this method derived an empirical equation to determine a measurement risk
multiplier for guard banding purposes for a given TUR and a desired PFA. A higher TUR leads
to a decrease in guard band for a particular PFA. In general, the false accept risk can be kept
between 1% and 2% for most in-tolerance probabilities and TURs. This guard banding method is
similar in nature to a method published in 1993 by Deaver [5].
3.2.1 Guard band equals expanded measurement uncertainty
In this method, the acceptance limit is determined by reducing the tolerance by the full amount
of the expanded measurement uncertainty. While this reduces the PFA to practically zero, it
increases the probability of false reject (PFR) to more than 10% [4]. While the calibration
laboratory employing this method protects the customer from assuming an instrument is good
when it actually is bad, making an incorrect out-of-tolerance call may also have undesirable
consequences for the customer.
There are numerous other published methods for guard banding that a calibration laboratory can
consider when deciding on a decision rule, and all are associated with various levels of risk.
Unlike the standard ANSI/NCSL Z540.3 [12], which many US laboratories have to meet,
ISO/IEC 17025:2017 does not prescribe that the probability of false accept be kept below a
certain level, nor that a specific decision rule be adopted. However, it does require that the
calibration laboratory and the customer agree on a decision rule that is acceptable to them, after
considering the risk levels involved. ISO/IEC 17025:2017 thus allows for greater flexibility
while ensuring that both parties, laboratory and customer, can agree on a level of risk that is
acceptable for their needs.
4
NCSL International Workshop & Symposium | Measurements of Tomorrow
August 27-30, 2018 | Portland Oregon
3.3 Risks with respect to nonconforming work and corrective actions
Finding a reference standard out of tolerance (OOT), a calibration performed incorrectly, an
incorrect calibration report issued to a customer—all are examples of nonconforming work in the
calibration laboratory. When dealing with these occurrences and deciding what action to take, we
consider the following risks: what is the likelihood of a calibration item having been
miscalibrated, what is the likelihood of recurrence if we do not take some kind of preventive
action, what is the likelihood of a similar issue having occurred elsewhere in the laboratory?
These are questions asked routinely in any well-run calibration laboratory addressing
nonconforming work. The actions taken depend on the root cause of a nonconformance, and the
actions need to fix the immediate problem as well as prevent the problem from recurring.
When a calibration laboratory finds any piece of its measuring and test equipment (MTE) pool
out of tolerance (OOT) during one of its periodic calibrations, the laboratory must perform an
immediate investigation to determine whether the instrument was used as a reference standard in
a calibration event. The next question to be addressed is to determine whether the OOT condition
could have affected the results of previously calibrated items. It is every calibration laboratory
manager’s worst nightmare to have to recall calibrated items, and it is not in the customers’ nor
the laboratory’s interests to do so without a thorough analysis of the likelihood and extent of a
miscalibration. If a recall is deemed necessary, the more information the laboratory can provide
to the customer regarding the nature of the problem, the easier it is for the customer to assess the
risks as to whether their own processes were negatively affected. Offering customers a free re-
calibration is oftentimes less important to a customer than a detailed analysis of the nature of the
miscalibration of their item.
Next, the laboratory has to determine what actions, if any, need to be taken to prevent the
recurrence of the out-of-tolerance condition. For example, the laboratory may decide to reduce
the calibration interval of the instrument found OOT. In doing so, it practices risk mitigation by
aiming to reduce the likelihood of the reference standard being found out of tolerance again in
the future. Presumably, the laboratory will have evidence that the instrument’s OOT condition
was likely caused by instrument drift, not by some other catastrophic event. The shorter the time
period until the next calibration, the smaller the risk of the OOT condition recurring. This is
because the “before” or “as found” results of the calibration should alert the laboratory to the fact
that an adjustment back to nominal is necessary. If such an adjustment is not possible, the
reference standard’s performance specification may have to be downgraded, and its suitability
for use in the calibrations has to be re-examined. However, mitigating actions are associated with
costs, so the laboratory has to weigh carefully the cost of more frequent calibrations of the
reference standard against the risk of potentially having to recall miscalibrated items.
A laboratory may want to mitigate its risks of finding a reference standard out of tolerance by
performing additional intermediate checks in between calibrations. In this case, a decision rule
similar to what we discussed earlier will have to be deployed to determine whether a full
calibration is necessary before the end of its assigned calibration interval. The risk here is that if
5
NCSL International Workshop & Symposium | Measurements of Tomorrow
August 27-30, 2018 | Portland Oregon
these intermediate checks are performed with equipment that is not sufficiently accurate,
resulting in large measurement uncertainties, many false alarms may occur due to the increased
likelihood of “false rejects” due to large measurement uncertainties. Or the laboratory may get a
false sense of confidence in the equipment’s performance due the intermediate checks providing
“false accept” results. So the effectiveness of intermediate checks has associated risks, which the
laboratory should be monitoring to assure that resources spent on intermediate checks have the
desired effect.
Depending on the root cause of a nonconforming work issue, other forms of preventive
recurrence may be to add more detail to calibration procedures, to improve technician training or
to re-train staff at more frequent intervals. In doing so, laboratories can mitigate the risk of
different technicians using different techniques or parameters that may produce differences in
calibration results significant enough to effect the performance of the instrument.
Whatever the nonconforming work issue, evaluation of both the likelihood of recurrence and the
severity of the consequences will determine the level of effort and resources expended to
mitigate the risk of recurrence.
3.4 Other laboratory activities involving risk-based thinking
3.4.1 Setting calibration intervals
We have discussed shortening calibration intervals as a preventive action after an OOT condition
of a piece of laboratory measurement and test equipment (MTE). However, the setting of
periodic calibration intervals for all of the laboratory’s MTE is another activity where risk-based
thinking must be applied. The main purpose of periodic calibration is to ensure the reliability of
the instrument, i.e. that the MTE, when employed, performs as designed. Thus, most
laboratories’ quality objectives include a set of performance or measurement reliability goals for
their pool of MTE. For example, a laboratory may want a minimum of, say, 95% of MTE to be
found in tolerance during their periodic calibrations. Measurement reliability goals may be set to
a higher or lower level for certain categories of equipment, depending on the consequences
associated with an OOT event.
The setting of optimal calibration intervals strikes a balance between the risk of finding a piece
of MTE out of tolerance and the cost pressures. If a laboratory is too conservative in setting its
calibration intervals in order to maintain extremely high measurement reliability goals, it may
calibrate itself right out of business. Hence, setting optimal calibration intervals is not only a
means of mitigating OOT risk, but also an opportunity to control calibration costs. Guidance on
setting calibration intervals can be found in NCSLI’s RP-1, Establishment and Adjustment of
Calibration Intervals [8].
3.4.2 Adjustment back to nominal
Guard banding was discussed earlier in this paper in conjunction with the decision rule at the
time of calibration in order to mitigate the risk of a false-accept or false-reject decision. Setting
an adjustment threshold that is lower than the tolerance limits could also be considered a form of
6
NCSL International Workshop & Symposium | Measurements of Tomorrow
August 27-30, 2018 | Portland Oregon
guard banding to mitigate a future OOT risk by adjusting an instrument back to nominal prior to
its drifting out of tolerance.
If the calibration result shows that a measured value is very close to its specification limit, but
the calibration laboratory’s adjustment threshold is equal to the specification limit and
adjustment to nominal does not take place, the instrument is likely to be out of tolerance before
the end of its calibration interval. On the other hand, running an adjustment procedure on every
instrument every time it comes in for calibration, without regard to how far it has drifted off
nominal, is very costly and oftentimes does not help improve the overall performance of a piece
of equipment. For example, the adjustment process for a complex electronic measurement
instrument may bring most measurement attributes closer to nominal, but it may move others
further away. As a result, a calibration laboratory may have a policy to run an adjustment
procedure only when the calibration results fall outside a certain adjustment threshold limit, and
this limit may vary from laboratory to laboratory.
It is important for the MTE owner performing calibration interval analysis in pursuit of optimal
calibration intervals to take this kind of information into consideration. Whether an instrument is
routinely adjusted back to nominal by the calibration provider or whether it is already close to its
specification limits at the beginning of its calibration interval will influence the laboratory’s
measurement reliability performance and the risk of finding MTE out of tolerance.
4. Providing evidence of risk-based thinking
This paper has listed many examples of calibration laboratory activities where risk is identified
and mitigating action is taken. It is important for the laboratory to document these risks and
individual actions, monitor whether the objective of mitigating the risk is met, and take further
actions if it is not. Keeping good records of the identified risks, associated mitigating actions
associated trends is not only necessary for good laboratory management, it is also a requirement
of the ISO/IEC 17025:2017 standard. It may be helpful to review any existing templates to
support laboratory activities, and to explicitly build risk consideration requirements into them.
The standard also requires the laboratory to review identified risks during the management
review. If careful records are kept on the type of activities discussed in this paper, the
information can easily be summarized and provided as management review input for review and
discussion.
5. Conclusion
The purpose of this paper was to provide examples of risk-based thinking being performed
routinely by laboratory management. The examples provided are not new, rather they are
activities carried out in today’s laboratories, accredited and non-accredited, on a regular basis.
Laboratories that monitor and review the risks involved in these activities on a regular basis
should have little difficulty in meeting this aspect of ISO/IEC 17025:2017 requirements.
References
7
NCSL International Workshop & Symposium | Measurements of Tomorrow
August 27-30, 2018 | Portland Oregon
1. ISO/IEC 17025:2017, General requirements for the competence of testing and
calibration laboratories
2. ISO 9001:2015, Quality management systems—Requirements
3. ISO 31000:2009, Risk Management---Principles and guidelines
4. Michael Dobbert, A Guard-Band Strategy for Managing False-Accept Risk, 2008 NCSL
International Workshop and Symposium
5. David Deaver, How to Maintain Your Confidence, 1993 NCSL International Workshop
and Symposium
6. Michael Dobbert, Understanding Measurement Risk, 2007 NCSL International
Workshop and Symposium
7. David Deaver and Jack Somppi, A Study of and Recommendations for Applying the False
Aceptance Risk Specification of Z540.3, Measurement Science Conference Proceedings,
2010
8. RP-1, Establishment and Adjustment of Calibration Intervals, NCSLI, Boulder, CO, 2010
9. RP-18, Estimation and Evaluation of Decision Risk, NCSLI, Boulder, CO, 2014
10. Handbook for the Application of ANSI/NCSL Z540.3-2006 – Requirements for the
Calibration of Measuring and Test Equipment, NCSLI, Boulder, CO, 2009
11. Michael Dobbert and Robert Stern, A Pragmatic Method for Pass/Fail Conformance
Reporting that Complies with ANSI Z540.3, ISO 17025, and ILAC-G8, 2009 NCSLI
Workshop and Symposium
12. ANSI/NCSL Z540.3-2006, Requirements for the Calibration of Measuring and Test
Equipment, NCSLI, Boulder, CO, 2006
