Introduction and Motivation to

Measurement Uncertainty Concepts

Steve Phillips
[email protected]

National Institute of Standards and Technology

www.NIST.gov

Kim Summerhays
[email protected]

MetroSage LLC
www.MetroSage.com

1
National Institute of Standards and Technology

Providing high accuracy measurements with rigorous

metrological traceability since 1901

2
 Workshop Outline

„ Introduction & Economic Issues

„ Basic Terminology & Uncertainty Concepts
„ ASME B89.7 Documentary Standards
„ Measurement Uncertainty Evaluation

S.D. Phillips PQI / IIGD&T 8/11/2011

3
 Motivation & Economics of
Measurement Uncertainty

„ Advanced economies, like the US, produce high value

products ⇒ dimensional control of features (GD&T) ⇒
measurement uncertainty must be known and
significantly less than the tolerance.

„ To understand or control a maufacturing process,

measurements with known uncertainty are needed.

„ Uncertain measurements led to uncertain decisions and

increased costs

S.D. Phillips PQI / IIGD&T 8/11/2011

4
 Decisions Generally Have
Economic Consequences

„ Measurements :
„ Provide Information to Make Better Decisions
„ Cost Money and Time

„ Manage the Risk of Making Decisions Through Uncertainty

Evaluation
„ The risks can be managed by measurement uncertainty analysis –
a technical issue
„ A decision is based on cost analysis, and hence is a matter of
business economics
„ Bad decisions cost money
„ An economically optimal decision balances costs vs uncertainty

S.D. Phillips PQI / IIGD&T 8/11/2011

5
 General Concepts of Uncertainty
„ What is Measurement Uncertainty?
Uncertainty of a measurement means doubt
about the validity of the result of the
measurement (GUM 2.2.1)

„ Measurement uncertainty represents our state

of knowledge regarding a measurement result.
„ When new information is available our knowledge
changes and the uncertainty statement needs to be
updated to reflect this new information.

S.D. Phillips PQI / IIGD&T 8/11/2011

6
 Benefits of Uncertainty Evaluation

„ Measurement Traceability
„ Required for calibration labs per ISO 17025
„ Select right tool for the job
„ e.g. meeting 4:1 Tolerance to Uncertainty ratio
„ Economics of workpiece accept/reject decisions
„ Optimizing decision rules
„ Effective use of $ for improved accuracy
„ Identify and address the largest uncertainty sources

S.D. Phillips PQI / IIGD&T 8/11/2011

7
 Uncertainty Standardization
„ The formulation of Uncertainty presented today
is based on the Guide to the Expression of
Uncertainty in Measurement, the “GUM”
„ This provides a internationally accepted and well
defined methodology for all measurements
„ Used by all NMIs, e.g. NIST, PTB, NPL…
„ Used by all ISO 17235 accredited calibration labs,
e.g. A2LA, LAB, NVLAP, accredited.
„ Used in all modern ASME & ISO standards
„ Is a US National Standard (NCSL Z540-2)

S.D. Phillips PQI / IIGD&T 8/11/2011

8
 The GUM

ANSI/NCSL Z540-2-1997 (same as GUM) is an American National Standard

http://www.bipm.org/en/publications/guides/gum.html (free download)
http://physics.nist.gov/Pubs/guidelines/TN1297/tn1297s.pdf (free download)

S.D. Phillips PQI / IIGD&T 8/11/2011

9
Mistakes are NOT Uncertainty Sources

Mistakes and Blunders associated with taking,

recording, or analyzing measurement data and
uncertainty are NOT considered an uncertainty
source for purposes of its calculation (3.4.7)
Example: Do Not include an uncertainty source for
data transcription errors, but do review your data
carefully!

“Random results are the consequences

of random procedures”

S.D. Phillips PQI / IIGD&T 8/11/2011

10
 Basic Terminology: The VIM

www.bipm.org/en/publications/guides/vim.html
(ISO Guide 99)

Beware of Jabberwocky:
“The big red sound smelled like sandpaper.”
“A direct uncertainty of – 5 μm resulted from a
type B error associated with the systematic
uncertainty of the resolution of the true value to
yield a 95% confidence interval.”
S.D. Phillips PQI / IIGD&T 8/11/2011
11
Basic Terminology: Speak like a Pro!
„ Calibration & Inspection
„ Measurand
„ Conformance / Non-Conformance
„ Accuracy
„ Uncertainty
„ Error
„ Bias (Systematic Error)

S.D. Phillips PQI / IIGD&T 8/11/2011

12
 Calibration & Inspection
For purposes of this workshop, we define:
„ Calibration: The measurement process of assigning a
value and its (GUM) uncertainty to an artifact, workpiece
or instrument result in a documented manner; e.g. the
length of a gauge block and the associated length
uncertainty in a calibration report
„ Inspection: The measurement and decision process
associated with the acceptance or rejection of an
artifact, workpiece or instrument

S.D. Phillips PQI / IIGD&T 8/11/2011

13
 The Measurand

„ The particular quantity subject to measurement

(GUM B.2.9)

„ A set of specifications (instructions) (GUM D.1)

„ NOT a number or value

„ Specifies the values, i.e. the “conditions”, of all

the potential influence quantities so that
(ideally) ONE “true value” can be realized.

S.D. Phillips PQI / IIGD&T 8/11/2011

14
 The Measurand & True Values

„ Failure to adequately define the measurand IS a

source of measurement uncertainty (GUM 3.3.2) and
can result in Multiple True Values!!! (GUM B.2.3)

„ A True Value is the result of a perfect instrument

measuring an infinite number of points on the
surface while fully complying with the definition of
the measurand
„ GD&T reduces the number of true values…
„ But not reduced to zero…
„ We don’t follow the instructions of the measurand…
S.D. Phillips PQI / IIGD&T 8/11/2011
15
 The Realized Quantity of a
Measurement
„ Typically, the value realized by the measurement system
is not consistent with the definition of the measurand
and a CORRECTION must be applied to yield a result that
does satisfy the measurand. Example, length
measurement at 22 °C must be corrected to 20 °C
„ The uncertainty in the correction is an uncertainty
contributor
„ We frequently choose, or are forced, to realize a quantity
other than the measuand and thus are required to make
corrections.

S.D. Phillips PQI / IIGD&T 8/11/2011

16
 Conformance / Non-Conformance
„ Conforming:
„ Having at least one true value lying within or on the
boundary of a stated tolerance interval.

„ Non-Conforming:
„ Having all true values lying outside the boundary of a
stated tolerance interval.

„ Note:
„ Metrologists do NOT know the true value, and hence
do not know if a workpiece is conforming… we can
only decide acceptance or rejection.

S.D. Phillips PQI / IIGD&T 8/11/2011

17
 Accuracy

„ “The closeness of the agreement between the result

of a measurement and a true value of the measurand.”
(GUM B.2.14)
„ the measurand is the particular quantity subject to measurement
„ the true value is unknowable

„ Accuracy is a qualitative concept

„ Uncertainty is the quantitative statement of accuracy
„ “Precision” is not accuracy
„ (I prefer not to use the word “precision” as a quantitative
metrological quantity)

S.D. Phillips PQI / IIGD&T 8/11/2011

18
 Measurement Uncertainty U

„ (Expanded) Uncertainty of a measurement

„ “A parameter [number] associated with the result of
a measurement, that characterizes the dispersion
[spread] of the values that could reasonably be
attributed to the measurand” (GUM B.2.18)

„ The issue of reasonable values is fundamental

to the GUM; reasonableness allows “expert
judgment” in addition to experimental data

S.D. Phillips PQI / IIGD&T 8/11/2011

19
 Measurement Uncertainty U
(continued)

„ (Expanded) Uncertainty is a positive number, not an

interval or probability distribution
„ Incorrect: U = ± 5 μm Correct: U = 5 μm
„ Stating measurement results: Y = y ± U OR y – U ≤ Y ≤ y + U
„ Note: the uncertainty is centered about the measured value
creating an uncertainty interval of width 2U U U

„ The VIM calls the “uncertainty interval” the “coverage interval”

„ Uncertainty is an attribute of a measurement result

„ Instruments do NOT have uncertainty, measurement results do!

S.D. Phillips PQI / IIGD&T 8/11/2011

20
 Measurement Uncertainty U
(continued)

„ Default (expanded) uncertainty corresponds to

a level of confidence of ≈ 95 %
„ Do not use “confidence level” or “confidence interval”

„ This means you will bet $95 against $5 that the “true value”
of the measurand Y, with a best estimated value y (from
measurement), lies in the uncertainty interval:
y–U≤Y≤y+U

„ Use uncertainty to describe our ignorance

about the “true value” of the measurand

S.D. Phillips PQI / IIGD&T 8/11/2011

21
 Measurement Error

„ “The Measured value minus the True value”

(GUM B.2.19) E=y-T
„ Errors can have either a positive or negative
sign
„ Since the true value is unknown, the error is
unknown -- hence we can only estimate
errors – but often we have very good
estimates!

S.D. Phillips PQI / IIGD&T 8/11/2011

22
 Measurement Error (continued)
„ For the measurement of most workpieces, the true
value is unknown (which is why we are measuring),
hence the error is unknown. In this case we should
speak of uncertainty not error
„ We can ONLY determine errors when we have an
estimate of the true value, e.g. when we measure
CALIBRATED objects!
„ Typically, the only time we estimate errors is during a
calibration.

S.D. Phillips PQI / IIGD&T 8/11/2011

23
 Measurement Error (continued)

„ Two different measurements of a gauge block with a caliper

may yield two different measurement results and hence two
different errors.
„ Repeated measurements will yield a distribution of errors for
the measurand; This is telling us something about the
measurement uncertainty
Measured
Frequency
Value

Calibrated
0.02 Error Value
-0.02 0.0
S.D. Phillips PQI / IIGD&T 8/11/2011
24
 Systematic Measurement Error
(Measurement Bias)

„ “The component of measurement error that in replicate

measurements remains constant or varies in a
predictable manner” (VIM 2.17)

„ The (mathematical) expectation value of the error and

estimated as the arithmetic average error

„ The GUM strongly recommends correcting for all

significant systematic errors in the measurement

S.D. Phillips PQI / IIGD&T 8/11/2011

25
 Repeated measurements with good
reproducibility but with a large uncorrected bias
Error E=y–T

• • • •
• • • •
Bias
0
Measurement Number
Note: Gage repeatability and reproducibility (GR&R) studies
typically use uncalibrated workpieces and hence report only
variations not errors. Systematic error (bias) is not
observable with uncalibrated workpiece GR&R studies.

S.D. Phillips PQI / IIGD&T 8/11/2011

26
 Some Sources of Systematic Error
„ Uncorrected thermal errors
„ Workpiece deflection under probe contact
„ Incorrect algorithm fitting
„ Fixture induced distortions
„ Point sampling strategy
„ Systematic errors in instrument
„ Incorrect compensations
„ Workpiece temperature, CTE, index of refraction…

S.D. Phillips PQI / IIGD&T 8/11/2011

27
 Error, Expanded Uncertainty,
Uncertainty Interval, Measured Value,
& True Value
Uncertainty Interval
Probability
Distribution of the
true value of the U U
measurand

y-U y+U
Error
True y = Measured Value
Value

S.D. Phillips PQI / IIGD&T 8/11/2011

28
US Documentary Uncertainty
Standards and Reports
The ASME B89.7 Series…
„ Addresses the issue of measurement uncertainty
in dimensional measurements
„ Particularly industrial measurements
„ Considers the “lifecycle” of uncertainty
„ B89.7.2 Dimensional Measurement Planning
„ B89.7.3.1 Decision rules for accept / reject decisions
„ B89.7.3.2 Simplified GUM evaluation
„ B89.7.3.3 Resolving differing uncertainty evaluations
„ B89.7.4.1 Risk analysis
„ B89.7.5 Measurement traceability

„ www.asme.org
S.D. Phillips PQI / IIGD&T 8/11/2011
29
Dimensional Measurement Planning
1999 and 2012(?) [under revision]

„ Provides the dimensional

measurement planner with
overview and check list of
requirements
„ Discusses rational of
measurements
„ Reviews other Standards in
B89.7 series

S.D. Phillips PQI / IIGD&T 8/11/2011

30
 Decision Rules used for inspection…
B89.7.3.1 (2001) & ISO 14253-1 (1998)

Measurement Uncertainty

Probability Distribution
of a measurement result

Tolerance zone

Given: Non Conforming Conforming Non Conforming

„ The product specification T L
Length x
xm TU

„ The measurement result

Note: A workpiece is Conforming
„ The measurement uncertainty
to specifications if the true value
A decision rule determines acceptance/rejection of the measurand is within the
specification zone
S.D. Phillips PQI / IIGD&T 8/11/2011
31
 Decision Rules
Decision Rule:
A documented rule, meeting the requirements of section 3 of
B89.7.3.1, that describes how measurement uncertainty will be
allocated with regard to accepting or rejecting a product
according to its specification and the result of a measurement.
B89.7.3.1
Section 3, B89.7.3.1, Requirements for Decision Rules
„ Zone Identification (all measurement outcomes must be identified)
„ Decision Outcome (all measurement results must yield a decision)
„ Repeated Measurements Policy
„ Data Rejection Policy

S.D. Phillips PQI / IIGD&T 8/11/2011

32
 Simple 4:1 Acceptance Decision Rule

Lower Upper
Specification Zone =
Specification Specification
Limit Simple Acceptance Zone Limit
Rejection U U Rejection
Zone Zone

Measurement Result
• The measurement uncertainty interval is of width 2 U
• The uncertainty interval is ≤ ¼ the product’s specification zone
• Hence the “measurement capability index” Cm = 4
• Acceptance if result in specification zone; rejection otherwise
• The measurement value shown results in product acceptance
S.D. Phillips PQI / IIGD&T 8/11/2011
33
 Stringent Acceptance
& Relaxed Rejection
Lower Upper
Specification Specification
Limit Limit
Specification Zone
gIn gIn

Relaxed Stringent Acceptance Relaxed

Rejection Rejection
Zone Zone Zone
• “g” is known as the guard band, typically expressed as a percentage of U
• The default rule of ISO 14253-1 is stringent acceptance with a 100 % U guard band.
• B89.7.3.1 treats the guard band as a business decision.
S.D. Phillips PQI / IIGD&T 8/11/2011
34
 ASME B89.7.3.2
Simplified Uncertainty Evaluation
„ Dimensional Measurement Uncertainty Evaluation
for Industrial Practitioners
„ Avoids:
„ partial derivatives
„ All input quantities in units of length

„ degrees of freedom
„ Suggestions to minimize this effect

„ correlation
„ Suggestions to avoid correlated quantities

„ Provides: Worked Examples

S.D. Phillips PQI / IIGD&T 8/11/2011

35
 ASME B89.7.3.3:
Assessing the Reliability of
Uncertainty Statements
„ Comparison of Uncertainty Budgets
„ Accounting for Uncertainty Sources
„ Magnitudes of Uncertainty Components
„ Effects of Uncertainty Sources
„ Third Party Review and Accreditation

„ Direct Measurement of the Measurand

„ Historical Measurements
„ Round Robins
„ Reproducibility Measurements (e.g. GR&R)
„ Measurement of Calibrated Artifacts
„ Third Party Measurements 36
S.D. Phillips PQI / IIGD&T 8/11/2011
 ASME B89.7.4:
Measurement Uncertainty and
Conformance Testing: Risk Analysis
„ ASME B89.7.4 Determines the guardband needed
based on business decisions

S.D. Phillips PQI / IIGD&T 8/11/2011

37
 ASME B89.7.5:
Dimensional Measurement
Traceability

„ Provides guidance on dimensional

measurement traceability especially for
industrial measurements
„ Provides one specific interpretation of
traceability that providers and users can agree
upon
„ Provides worked examples of traceability
requirements

S.D. Phillips PQI / IIGD&T 8/11/2011

38
 Overview of Uncertainty Evaluation

Coverage
Uncertainty Interval Factor
Probability
Distribution of the
true value of the U U U = 2 uc
measurand uc = combined
standard uncertainty
uc

y-U y+U
Error
True y = Measured Value
Value

S.D. Phillips PQI / IIGD&T 8/11/2011

39
 Overview of Uncertainty Evaluation

Key Point:
Combine Probability Distributions

= + + +

combined standard Influence quantity probability

uncertainty distributions….

S.D. Phillips PQI / IIGD&T 8/11/2011

40
 Overview of Uncertainty Evaluation
There are several methods to combine distributions…
„ Mathematical Analysis :
2
⎛ ∂f ⎞ 2 N −1 N
∂f ∂f
r ( xi , x j ) u ( xi ) u ( x j )
N
u = ∑⎜
2
⎟ u ( x ) + 2 ∑ ∑
i =1 ⎝ ∂xi ⎠ i =1 j =i +1 ∂xi ∂x j
c i

„ Experimental Observations:
“Super GR&R”
„ Monte Carlo Calculations
Computer Simulation … Up Next!
S.D. Phillips PQI / IIGD&T 8/11/2011
41
 Review of Basic Terminology
“A direct uncertainty of –5 μm resulted from a
type B error associated with the systematic
uncertainty of the resolution of the true value to
yield a 95% confidence interval.”

“An expanded (k =2) uncertainty of 5 μm included

an input quantity from a type B uncertainty
evaluation associated with the systematic effect of
the resolution of the measuring instrument to
yielding an uncertainty interval having a 95%
level of confidence.”

S.D. Phillips NACMA 10/4/2010

42