Evaluating the Measurement Process (EMP) Overview

How do you evaluate your measurement processes? In the days of

manual calculations, the average/range method was the preferred
way of evaluating the measurement process. This process was
championed by the AIAG (Automotive Industry Action Group). With
computers, the ANOVA method has become the preferred way of
analyzing a measurement process. It is time for ANOVA to move on
also. It is time to replace both techniques with something called
Evaluating the Measurement Process (EMP).

EMP is a collection of techniques that allows you to discover much, much more about your
measurement process than if you just use those traditional techniques. EMP has been developed over
the years by Dr. Donald J. Wheeler.

The experimental process is essentially the same. You have one or more operators. Each operator
measures the same part(s) multiple times. The difference is that EMP immediately places the results on
a control chart to assess how consistent, reproducible , and repeatable the measurement system is
before moving forward with the characterization (calculations) of the various statistics, like the
measurement error.

This month’s publication provides an overview of the techniques in EMP. In this issue:

• Introduction to EMP
• Relationship of Variances
• Consistency Study
• Short EMP Study
• Basic EMP Study
• Summary
• Quick Links

Introduction to EMP

The EMP techniques are designed to help you answer questions you should have about your
measurement system. Some of the questions from Dr. Wheeler’s book, EMP III: Evaluating the
Measurement Process and Using Imperfect Data, are:

1. Is the measurement system consistent over time?

2. What is the precision of the measurement system?
3. How many decimal places should be recorded?
4. What is the resolution of the measurement system?
5. Does the measurement system contain bias?
6. How useful is the measurement system in characterizing the product?
7. How much of the variation in the product measurements are due to the measurement system?
8. Can the measurement system detect changes in the process?

1 © 2019 BPI Consulting, LLC

www.spcforexcel.com
It should be noted that the other measurement systems evaluation methods rely on question number 7
to determine if a measurement system is “good.” The value for the % of the total variance due to the
measurement system is taken and compared to an arbitrarily set standard. So much information is
missed with this approach.

The EMP techniques involve the use of control charts to ensure that your data are “valid” before any
̅-
other calculations are done. The individuals control chart is used with the Consistency Study while the X
R control charts are used with the Short EMP Study and the Basic EMP Study. Each of these three
studies is described below. Each has a previous SPC Knowledge Base article written that explains the
technique in more detail.

The table below summarizes what you can learn from each of the three techniques.
EMP Technique Operators Parts Purpose
Consistency Study 1 1 • Determine if the measurement system is
consistent by using an individuals (X-mR) control
chart
• Estimate the measurement system error
• Determine product and total variances (assuming
you have a historical value for the process sigma)
o Classify the measurement system as a First
Class, Second Class, Third Class or Fourth
Calls monitor
• Determine if bias is present
• Determine if the data are chunky
• Determine if there is enough data
• Determine the effective resolution of the
measurement system
• Determine if the measurement increment is
adequate
• Determine internal manufacturing specifications
based on the probable error
• Determine the precision to tolerance ratio
Short EMP Study 1 Multiple • Analyze the results by using an X ̅ -R control
chart
• Determine if the range chart is consistent
• Estimate the measurement system error
• Determine product and total variances (using
parts in the study or historical process sigma)
• Classify the measurement system as a First Class,
Second Class, Third Class or Fourth Calls monitor
• Determine if there is enough data
• Determine the effective resolution of the
measurement system
• Determine if the measurement increment is
adequate
• Determine internal manufacturing specifications
based on the probable error
Determine the precision to tolerance ratio

2 © 2019 BPI Consulting, LLC

www.spcforexcel.com
 EMP Technique Operators Parts Purpose
Basic EMP Study Multiple Multiple • Analyze the results by using an X ̅ -R control
chart
• Determine if the range chart is consistent
• Determine if there are differences in the
operator averages using the Analysis of Main
Effects
• Determine if there are differences in the average
moving range for each operator by using the
Analysis of Mean Ranges
• Estimate the measurement system error
• Determine product and total variances (using
parts in the study or historical process sigma)
• Classify the measurement system as a First Class,
Second Class, Third Class or Fourth Calls monitor
• Determine if there is enough data
• Determine the effective resolution of the
measurement system
• Determine if the measurement increment is
adequate
• Determine internal manufacturing specifications
based on the probable error
• Determine the precision to tolerance ratio

An overview of the three are given below. We will start, however, with a reminder of the relationship
between the total variance, product variance and measurement error variance.

Relationship of Variances

You take a sample from your process. You test that sample using your measurement system. You get a
result (X1). You take another sample and test that sample. You get another result (X2). Usually X1 does
not equal X2. What are in these results? Two major components are present in each result: the
variation in the product itself and the variation in the measurement system.

The basic equation describing the relationship between these variances is given below.

σx2= σp2 + σe2

where σx2 = total variance of the product measurements, σp2 = the variance of the product, and σe2 = the
variance of the measurement system.

The Consistency Study

A Consistency Study is run by having one operator measure the same part multiple times. The results
are then placed on an individuals (X-mR) control chart. The control chart must be in statistical control
before proceeding. If it is not, the measurement system cannot really measure anything until the
reasons for the out of control points are found and corrected. Once the measurement system is
consistent, you can move forward with the characterization of the measurement system (e.g.,
calculating the measurement error and bias).
3 © 2019 BPI Consulting, LLC
www.spcforexcel.com
An example of a Consistency Study is given in our SPC Knowledge Base article, “Evaluating the
Measurement Process - Part 1.” In this example, a viscosity standard is measured repeatably by one
operator. The resulting control charts are shown below. The data are in the article.

Both charts are in statistical control. This means that the measurement system is consistent. Because
the measurement system is consistent, you can begin to answer some important questions about your
measurement system.

Figure 1: X Chart for Repeated Viscosity Measurements for the Standard

Do I Have Enough Data?

It is recommended that you have at least 10 degrees of freedom for a consistency study. This
corresponds to 17 sample results. This example has 25 samples and 15.0 degrees of freedom. So, you
have enough data. If not, you need to collect more data. Dr. Wheeler’s book contains tables to
determine the degrees of freedom.

Is there Bias Present?

You can check for bias if a known standard was used. You do this by constructing a confidence limit
around the average using the t-distribution. Two intervals are created: one for 90% and one for 99%.
If the value of the known standard falls within the 90% confidence limit, there is no bias present. If it
falls outside the 99% confidence interval, then there is detectable bias present. If it falls between the
90% and 99% confidence interval, there is potential bias present.

4 © 2019 BPI Consulting, LLC

www.spcforexcel.com
 Figure 2: Moving Range Chart for Repeated Viscosity Measurements for the Standard

Is the Data Chunky?

This tells you if the measurement results on the same part/standard are rounded excessively. If there is
excessive rounding, the data are considered chunky. Our publication, Chunky Data and Control Charts,
explains this in more detail. The moving range chart is key here. The following is the test:

For X-mR charts: the data is chunky if the range chart has three or fewer possible values below the upper
control limit.

As can be seen in Figure 2, there are more than three possible values, so the data are not chunky.

What is the Repeatability of the Measurement System?

This is usually given as a standard deviation using the average moving range:

𝑚𝑅̅ 2.06
𝜎𝑒 = = = 1.83
1.128 1.128
where 1.128 is a constant based on the moving range of two.

Is the Measurement Increment Adequate?

The raw data used for the control charts above were measured to the nearest 0.1. This is the
measurement increment. There is a statistic called the probable error (PE). It is given by 0.675𝜎𝑒 .

PE = .675(𝜎𝑒 ) = .675(1.83) = 1.235

Half the repeated measurements will fall between the average and ± PE. PE can also be used to
determine if the measurement increment is adequate. The resolution (measurement increment) should

5 © 2019 BPI Consulting, LLC

www.spcforexcel.com
be between 0.2(PE) and 2(PE). That range for this example is 0.2471 to 2.471. The measurement
increment is 0.1, which is outside this range. The data should be rounded to the nearest 0.2 and the
analysis repeated. We will skip that part here. Please see our article, Probable Error and Your
Measurement System, for more information.

How Much of the Total Variance is due to the Measurement System?

This is the characteristic that most people pay attention to. Remember the equation above: σx2= σp2 +
σe2. The estimate of σe2 is known already. An estimate of the total variance (σx2) is needed. The best
place to get this is from a range chart kept on the product viscosity. Suppose we have done that and the
process sigma from the range chart is 2.76. We can then determine the % of the total variance that is
due to the measurement system:

𝜎𝑒2 1.832
𝑇𝑜𝑡𝑎𝑙 𝑉𝑎𝑟𝑖𝑎𝑛𝑐𝑒 𝑑𝑢𝑒 𝑡𝑜 𝑀𝑒𝑎𝑠𝑢𝑟𝑒𝑚𝑒𝑛𝑡 𝑆𝑦𝑠𝑡𝑒𝑚 = 2 = = .44 = 44%
𝜎𝑥 2.762

How “Good” is My Measurement System?

This single value of the total variance due to the measurement system is what most people use to
determine if the measurement system is good enough. For example, if the value is between 1% and 9%,
AIAG says the measurement system may be acceptable for some situations. If it is greater than 9% (like
this example), it is unacceptable. Dr. Wheeler has fundamentally changed how to look at how “good” a
measurement system is. Dr. Wheeler based part of his rating on the value of the Intraclass Correlation
Coefficient (𝜌). This fancy name is simply the total variance due to the product itself (σp2).

𝜎𝑒2 1.832
𝜌 = 𝜎𝑝2 = 1 − = 1 − = .56 = 56%
𝜎𝑥2 2.762
The table below shows Dr. Wheeler’s system. It classifies a measurement system as a First Class, Second
Class, Third Class or Fourth Class based on the value of 𝜌.

Table 1: The Four Classes of Process Monitors

Ability to Track
Reduction of Process Chance of Detecting ± 3
 Type of Monitor Process
Signal Std. Error Shift
Improvements
More than 99% with
0.8 to 1.0 First Class Less than 10% Up to Cp80
Rule 1
More than 88% with
0.5 to 0.8 Second Class From 10% to 30% Up to Cp50
Rule 1
More than 91% with
0.2 to 0.5 Third Class From 30% to 55% Up to Cp20
Rules 1, 2, 3 and 4
0.0 to 0.2 Fourth Class More than 55% Rapidly Vanishing Unable to Track

The table above answers three additional questions:

6 © 2019 BPI Consulting, LLC
www.spcforexcel.com
 • How much the measurement system reduces the strength of a signal (out of control point) on a
control chart.
• The chance of the measurement system detecting a large shift.
• The ability of the measurement system to track process improvements.

So, in this example, we have a Second Class monitor that will produce a reduction of signal from 10 to
30%, will be able to detect a large shift 88% of the time and track improvement up to Cp50. That is quite
a bit for a measurement system that AIAG’s rating system would say is unacceptable. For more
information on this classification system, please see our SPC Knowledge Base articles, Evaluating the
Measurement Process - Part 1 and Acceptance Criteria for MSA. These articles provide a more detailed
explanation of the classification system.

How Does the Measurement Error Impact Specifications?

Suppose you measure a part, and it is right below the upper specification limit. Is it within specification?
Maybe or maybe not. The probable error defined above can help determine the probability that it is
within specifications. You can use the probable error to determine manufacturing specifications. For
example, with 96% manufacturing specifications, there is a 96% probability that an item with a
measured value that falls between the manufacturing LSL and manufacturing USL conforms to
specifications. The 96% manufacturing specifications are created by moving the customer specifications
in by 2 probable errors. You can do this calculation for different probabilities. For more information on
setting manufacturing specifications, please see our article, Specifications and Measurement Error.

Short EMP Study

A Short EMP Study is run by having one operator measure multiple parts multiple times. Using different
parts allows you to begin to assess if the measurement system can tell the difference between parts.
̅-R control chart. Each subgroup is composed of the measurements on
The results are first placed on a X
one part. The range is then an estimate of the measurement error (like in the Consistency Study). The
range chart must be in statistical control to move forward with the characterization of the measurement
system.

The range chart below in Figure 3 is an example from our article, The Short EMP Study. In this article,
an operator measured 10 different parts 3 times each. The range chart is consistent.
̅ chart. Since the average range
The average range is used in the calculation of the control limits for the X
̅
represents just measurement error, you want the part averages on the X chart to be out of control as
shown in Figure 4. This means that the measurement system can tell the difference between parts in
the process. If the parts are representative of the process, you can use that variation to determine the
total variance and part variance. If the parts are not representative, you can always use a historical
value for the process sigma as shown above in the Consistency Study.

Since the ̅
X control limits are based on the measurement error, the width of the control limits
represents the amount of part variation that is obscured by the measurement error. This is shown in
Figure 5. You cannot see the variation in parts within these control limits because the variation is
masked by the measurement error.

7 © 2019 BPI Consulting, LLC

www.spcforexcel.com
 Figure 3: Range Chart for Short EMP Study

1.4

UCL=1.26
1.2

1
Subgroup Range

0.8

0.6

Avg=0.49
0.4

0.2

0
1 2 3 4 5 6 7 8 9 10
Subgroup Number

̅ Chart for Short EMP Study

Figure 4: X
13

12.5

12

UCL=11.7
11.5
Subgroup Average

Avg=11.2
11

LCL=10.7
10.5

10

9.5

9
1 2 3 4 5 6 7 8 9 10
Subgroup Number

8 © 2019 BPI Consulting, LLC

www.spcforexcel.com
 Figure 5: Measurement Error Impact on Part Average Variation

Figure 5 also shows the width that is represented by the variation in the average of the parts. You want
this width to be greater than the width obscured by the measurement error. This is true in Figure 5. You
have indications from this chart that the measurement system can tell the difference between parts
most of the time.

If the range chart is consistent, you can begin to do all the characterizations shown in the Consistency
Study above. Some of the formulas will be different.

Basic EMP Study

The Basic EMP Study is performed with multiple operators running the same parts multiple times. The
results are assessed first by placing them on ̅
X-R control chart using the results for one operator and one
part to form the subgroups. The R control chart must be in control before you can characterize the
measurement system. Again, the range chart represents measurement error.

Our article, Evaluating the Measurement Process – Part 2, has an example of the Basic EMP Study. In
this example, 3 operators each measure the same 5 parts two times. The X ̅-R control chart (Figures 6
and 7) is shown below.

The ̅
X chart for the Basic EMP Study is like the ̅
X chart for the Short EMP Study – just one for each
operator.

Since there are multiple operators, the analysis also includes an ANOM chart for the main effects
(operator averages) to see if there are differences between the operators. Figure 8 below shows that
Operator A gets a significantly different average than Operators B and C. You will have to find out why
that is the case.

9 © 2019 BPI Consulting, LLC

www.spcforexcel.com
 ̅ Chart for the Operator-Part Subgroups
Figure 6: X

Figure 7: Range Chart for Operator-Part Subgroups

10 © 2019 BPI Consulting, LLC

www.spcforexcel.com
 Figure 8: Main Effects Chart

Figure 9 is the mean range chart which compares the average range for each operator. All three
operators in this example have the same moving range.

Figure 9: Mean Range Chart

If the range chart and mean range chart are consistent, you can begin to do all the characterizations
given in the Consistency Study. Some of the calculations will use different formulas.

11 © 2019 BPI Consulting, LLC

www.spcforexcel.com
Summary

This article has provided an overview to the three major techniques of EMP: the Consistency Study, the
Short EMP Study and the Basic EMP Study. The first technique, Consistency Study, involves one
operator and one part. The objective is to determine if the measurement system is consistent. If it is
not, there is nothing you can really do with it – the measurements are meaningless.

The second technique, the Short EMP Study, involves one operator still but multiple parts. This begins
to allow you to determine if the measurement system can tell the difference between parts.

The third technique, the Basic EMP Study, brings additional operators into the study. This allows you to
begin to see if differences exist between operators. Regardless of the technique, if the measurement
system is consistent, you can begin to characterize the measurement system in many ways as shown in
the Consistency Study above.

Quick Links

Visit our home page

SPC for Excel Software
SPC Training
SPC Consulting
SPC Knowledge Base
Ordering Information
Thanks so much for reading our publication. We hope you find it informative and useful. Happy charting
and may the data always support your position.

Sincerely,

Dr. Bill McNeese

BPI Consulting, LLC

12 © 2019 BPI Consulting, LLC

www.spcforexcel.com