TÜV Rheinland România

Measurement System Analysis

Revizia 2 din 07.02.2015 1 Academia TÜV Rheinland România

 About us

Ion Teohari
 Engineer in Electronics and Tc
 8 years – System engineer for minicomputers
 8 years – Sales Manager - IT
 8 years – Country Manager – META Group – IT&C consulting
 8 years – International Certified Lean Six Sigma Black Belt

Aurelian Iuscu
 Engineer TCM and Economist
 3 years – Retail banking
 7 years – Projects and organization
 3 years – Business processes improvement
 3 years – International Certified Lean Six Sigma Black Belt

Revizia 2 din 07.02.2015 2 Academia TÜV Rheinland România

 Objectives

At the end of this presenattion, you should be able to:

Understand Precision & Accuracy

Understand Bias, Linearity and Stability

Understand Repeatability & Reproducibility

Understand the impact of poor gage capability on product quality

Identify the various components of Variation

Perform the step by step methodology in Variable and Attribute MSA’s

Revizia 2 din 07.02.2015 3 Academia TÜV Rheinland România

Introduction to MSA

– How do you know that the data you have used is accurate and precise?
– How do know if a measurement is a repeatable and reproducible?

How good are these?

Measurement System Analysis

or
MSA
Revizia 2 din 07.02.2015 4 Academia TÜV Rheinland România
Introduction: Basic requirements by QS-9000 & TS16949

Base on QS9000 & TS16949 requirements, on all measurement system

which were mentioned in Quality Plan should be conducted Measurement
System Analysis.
MSA
Requirement

Revizia 2 din 07.02.2015 5 Academia TÜV Rheinland România

MSA Uses

MSA can be used to:

Compare internal inspection standards with the standards of your customer.

Highlight areas where calibration training is required.

Provide a method to evaluate inspector training effectiveness as well as

serves as an excellent training tool.

Provide a great way to:

Compare existing measurement equipment.
Qualify new inspection equipment.

Revizia 2 din 07.02.2015 6 Academia TÜV Rheinland România

Why MSA?

Measurement System Analysis is important to:

 Study the % of variation in our process that is caused by our
measurement system.
 Compare measurements between operators.
 Compare measurements between two (or more) measurement
devices.
 Provide criteria to accept new measurement systems (consider
new equipment).
 Evaluate a suspect gage.
 Evaluate a gage before and after repair.
 Determine true process variation.
 Evaluate effectiveness of training program.
Revizia 2 din 07.02.2015 7 Academia TÜV Rheinland România
Appropriate Measures

Appropriate Measures are:

 Sufficient – available to be measured regularly

 Relevant –help to understand/isolate the problems

 Representative - of the process across shifts and people

 Contextual – collected with other relevant information that

might explain process variability.

Wadda ya
wanna
measure!?!
Revizia 2 din 07.02.2015 8 Academia TÜV Rheinland România
Poor Measures

Poor Measures can result from:

 Poor or non-existent operational definitions

 Difficult measures

 Poor sampling

 Lack of understanding of the definitions

 Inaccurate, insufficient or non-calibrated

measurement devices
Measurement Error compromises decisions that affect:
 Customers

 Producers

 Suppliers

Revizia 2 din 07.02.2015 9 Academia TÜV Rheinland România

Examples of What to Measure

Examples of what and when to measure:

 Primary and secondary metrics
 Decision points in Process Maps
 Any and all gauges, measurement devices, instruments, etc
 “X’s” in the process
 Prior to Hypothesis Testing
 Prior to modeling
 Prior to planning designed experiments
 Before and after process changes
 To qualify operators

MSA is a Show Stopper!!!

Revizia 2 din 07.02.2015 10 Academia TÜV Rheinland România
Measurement System Analysis

MSA is a mathematical procedure to quantify variation introduced to a

process or product by the act of measuring.

Item to be Reference
Measured Measurement
Operator Measurement Equipment
Process

Procedure
Environment
The item to be measured can be a physical part, document or a scenario for customer service.
Operator can refer to a person or can be different instruments measuring the same products.
Reference is a standard that is used to calibrate the equipment.
Procedure is the method used to perform the test.
Equipment is the device used to measure the product.
Environment is the surroundings where the measures are performed.

Revizia 2 din 07.02.2015 11 Academia TÜV Rheinland România

Introduction: What are the variations of measurement process

Revizia 2 din 07.02.2015 12 Academia TÜV Rheinland România

Components of Variation

Whenever you measure anything, the variation that you observe can be
segmented into the following components…
Observed Variation

Unit-to-unit (true) Variation Measurement System Error

Precision Accuracy

Repeatability Reproducibility Stability Bias Linearity

All measurement systems have error. If you don’t know how much of the variation you
observe is contributed by your measurement system, you cannot make confident
decisions.

If you were one speeding ticket away from losing your license,
how fast would you be willing to drive in a school zone?

Revizia 2 din 07.02.2015 13 Academia TÜV Rheinland România

Precision

A precise metric is one that returns the same value of a

given attribute every time an estimate is made.

Precise data are independent of who estimates them or

when the estimate is made.

Precision can be partitioned into two components:

 Repeatability
 Reproducibility

Repeatability and Reproducibility = Gage R+R

Revizia 2 din 07.02.2015 14 Academia TÜV Rheinland România

Repeatability

Repeatability is the variation in measurements obtained with one

measurement instrument used several times by one appraiser while
measuring the identical characteristic on the same part.

Repeatability

For example:
 Manufacturing: One person measures the purity of multiple
samples of the same vial and gets different purity measures.
 Transactional: One person evaluates a contract multiple times
(over a period of time) and makes different determinations of errors.

Revizia 2 din 07.02.2015 15 Academia TÜV Rheinland România

Reproducibility

Reproducibility is the variation in the average of the measurements made

by different appraisers using the same measuring instrument when
measuring the identical characteristic on the same part.
Reproducibility

Y Operator A
Operator B

For example:
 Manufacturing: Different people perform purity test on samples
from the same vial and get different results.
 Transactional: Different people evaluate the same contract and
make different determinations.

Revizia 2 din 07.02.2015 16 Academia TÜV Rheinland România

Understanding Accuracy and Precision

 If the pictures to the right

represent weapons training by
two recruits, which one is
better?
 Green?
 Blue? (Green)

 Which one is more accurate

(better average)?
 Which one is more precise
(more consistency)?
(Blue)

Revizia 2 din 07.02.2015 17 Academia TÜV Rheinland România

Weapons Training Example

 On average, the green target is

centered on the bulls-eye,
therefore more accurate
 Accuracy is a measure of
“average distance from the
target”
(Green)
 However, the blue target is
more consistent, therefore
more precise
 Precision is a measure of
“average distance from
each other”
(Blue)

Revizia 2 din 07.02.2015 18 Academia TÜV Rheinland România

Weapons Training Example

 How could the recruit using the

green target improve
performance?
 How could the recruit using the
blue target improve
performance? (Green)

 Which recruit do you think has a

better chance of becoming an
expert shooter?

Typically, it is easier to shift the mean than

to reduce variation (Blue)

Revizia 2 din 07.02.2015 19 Academia TÜV Rheinland România

Goal: Shift the Mean / Reduce Variation

Too Much Spread Off Center

Centered (Blue)
(Green) Reduce On-Target Center
Spread Process

Result: Improved Customer Satisfaction and Reduced Costs

Revizia 2 din 07.02.2015 20 Academia TÜV Rheinland România

Time Estimate Exercise

Exercise objective: Demonstrate how well you can

estimate a 10 second time interval.

1. Pair up with an associate.

2. One person will say start and stop to indicate how
long they think the 10 seconds last. Do this 6 times.
3. The other person will have a watch with a second
hand to actually measure the duration of the estimate.
Record the value where your partner can’t see it.
4. Switch tasks with partner and do it 6 times also.
5. Record all estimates, what do you notice?

Revizia 2 din 07.02.2015 21 Academia TÜV Rheinland România

Accuracy

An accurate measurement is the difference between the observed average

of the measurement and a reference value.
 When a metric or measurement system consistently over or under estimates
the value of an attribute, it is said to be “inaccurate”
Accuracy can be assessed in several ways:
 Measurement of a known standard
 Comparison with another known measurement method
 Prediction of a theoretical value
True
What happens if we don’t have standards, Average
comparisons or theories?
Accuracy

Warning, do not assume your

metrology reference is gospel.

Measurement
Revizia 2 din 07.02.2015 22 Academia TÜV Rheinland România
Bias

Bias is defined as the deviation of the measured value from the actual
value.

Calibration procedures can minimize and control bias within acceptable

limits. Ideally, Bias can never be eliminated due to material wear and
tear!
Bias Bias

Revizia 2 din 07.02.2015 24 Academia TÜV Rheinland România

Stability

Stability of a gage is defined as error (measured in terms of Standard

Deviation) as a function of time. Environmental conditions such as cleanliness,
noise, vibration, lighting, chemical, wear and tear or other factors usually
influence gage instability. Ideally, gages can be maintained to give a high
degree of Stability but can never be eliminated unlike Reproducibility. Gage
Stability studies would be the first exercise after calibration procedures.
Control Charts are commonly used to track the Stability of a measurement
system over time.
Drift

Stability is Bias
characterized as a
function of time!

Revizia 2 din 07.02.2015 25 Academia TÜV Rheinland România

Linearity

Linearity is defined as the difference in Bias values throughout the

measurement range in which the gauge is intended to be used. This tells you
how accurate your measurements are through the expected range of the
measurements. It answers the question, "Does my gage have the same
accuracy for all sizes of objects being measured?"
Low Nominal High
Linearity = |Slope| * Process Variation
+e

B i a s (y)
% Linearity = |Slope| * 100
0.00
*
-e
*
*
Reference Value (x)
y = a + b.x
y: Bias, x: Ref. Value
a: Slope, b: Intercept

Revizia 2 din 07.02.2015 26 Academia TÜV Rheinland România

 The category of Measurement System

Most industrial measurement system can be divided two categories, one is variable
measurement system, another is attribute measurement system. An attribute gage
cannot indicate how good or how bad a part is , but only indicates that the part is
accepted or rejected. The most common of these is a Go/No-go gage.

Attribute Gage (Go/No-go Gage)

Variable Gage

Revizia 2 din 07.02.2015 27 Academia TÜV Rheinland România

Types of MSA’s

MSA’ s fall into two categories:

Attribute Variable
– Pass/Fail – Continuous scale
– Go/No Go – Discrete scale
– Document Preparation – Critical dimensions
– Surface imperfections – Pull strength
– Customer Service – Warp
Response

Transactional projects typically have Attribute based measurement

systems.

Manufacturing projects generally use Variable studies more often, but

do use Attribute studies to a lesser degree.

Revizia 2 din 07.02.2015 28 Academia TÜV Rheinland România

 Analysis Techniques

Variable MSA .

Revizia 2 din 07.02.2015 29 Academia TÜV Rheinland România

Purpose

The purpose of MSA is to assess the error due to measurement

systems.
The error can be partitioned into specific sources:
 Precision
 Repeatability - within an operator or piece of equipment
 Reproducibility - operator to operator or attribute gage to
attribute gage
 Accuracy
 Stability - accuracy over time
 Linearity- accuracy throughout the measurement range
 Resolution
 Bias – Off-set from true value
 Constant Bias
 Variable Bias – typically seen with electronic equipment,
amount of Bias changes with setting levels

Revizia 2 din 07.02.2015 30 Academia TÜV Rheinland România

Analysis Techniques: Preparation before MSA

1. The approach to be used should be planned.

2. The number of appraisers, number of sample parts, and number of
repeat readings should be determined in advance.
3. The appraisers should be selected form those who normally operate
the instrument.
4. The sample parts must be selected from the process and represent
its entire operating range.
5. The instrument must have a discrimination that allows at least
one-tenth of the expected process variation of the characteristic to be
read directly.
6. The measurement procedure should be defined in advance to
ensure the consistent measuring method.

Revizia 2 din 07.02.2015 31 Academia TÜV Rheinland România

Components of Variation

Whenever you measure anything, the variation that you observe can be
segmented into the following components…
Observed Variation

Unit-to-unit (true) Variation Measurement System Error

Precision Accuracy

Repeatability Reproducibility Stability Bias Linearity

All measurement systems have error. If you don’t know how much of the variation you
observe is contributed by your measurement system, you cannot make confident
decisions.

If you were one speeding ticket away from losing your license,
how fast would you be willing to drive in a school zone?

Revizia 2 din 07.02.2015 32 Academia TÜV Rheinland România

 Measurement Systems Capability

LSL Measurement USL

Error Distribution

EE  Rpt
22
Rpt
 
 
2 2
Rpd
Rpd

6 E

(USL - LSL)
6E
%R &R  100
USL  LSL
Revizia 2 din 07.02.2015 33 Academia TÜV Rheinland România
Gage R & R Study

Gage R&R Study

Is a set of trials conducted to assess the Repeatability and Reproducibility of the
measurement system.
Multiple people measure the same characteristic of the same set of multiple
units multiple times (a crossed study)

Example: 10 units are measured by 3 people. These units are then randomized
and a second measure on each unit is taken.

A Blind Study is extremely desirable.

Best scenario: operator does not know the measurement is a part of a test
At minimum: operators should not know which of
the test parts they are currently measuring.
NO, not that kind of R&R!

Revizia 2 din 07.02.2015 34 Academia TÜV Rheinland România

Design Types

Crossed Design

A Crossed Design is used only in non-destructive testing and assumes that all the
parts can be measured multiple times by either operators or multiple machines.
Gives the ability to separate part-to-part Variation from measurement system
Variation.
Assesses Repeatability and Reproducibility.
Assesses the interaction between the operator and the part.
Nested Design

A Nested Design is used for destructive testing and also situations where it is not
possible to have all operators or machines measure all the parts multiple times.
Destructive testing assumes that all the parts within a single batch are identical
enough to claim they are the same.
Nested designs are used to test measurement systems where it is not possible
(or desirable) to send operators with parts to different locations.
Do not include all possible combinations of factors.
Uses slightly different mathematical model than the Crossed Design.

Revizia 2 din 07.02.2015 35 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

The Average and Range Method

1. General Gage R&R Study:
The ANOVA Method

The common step for conducting Gage R&R study:

1. Verify calibration of measurement equipment to be studied.
2. Obtain a sample of parts that represent the actual or expected range of
process variation.
3. Add a concealed mark to each identifying the units as numbers 1 through 10.
It is critical that you can identify which unit is which. At the same time it is
detrimental if the participants in the study can tell one unit from the other
(may bias their measurement should they recall how it measured previously).
4. Request 3 appraisers. Refer to these appraisers as a A, B, and C appraisers.
If the measurement will be done repetitively such as in a production environment,
it is preferable to use the actual appraiser that will be performing the
measurement.

Revizia 2 din 07.02.2015 36 Academia TÜV Rheinland România

Gage R & R Study

Part Allocation From Any Population

10 x 3 x 2 Crossed Design is shown
A minimum of two measurements/part/operator is required
Three is better!
Trial 1
Operator 1
Trial 2
P
a
r Trial 1
1 2 3 4 5 6 7 8 9 10 Operator 2
t
s Trial 2

Trial 1
Operator 3
Trial 2

Revizia 2 din 07.02.2015 37 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

For extreme cases, a minimum of two appraisers can be used, but this is strongly
discouraged as a less accurate estimate of measurement variation will result.
5. Let appraiser A measure 10 parts in a random order while you record the data
noting the concealed marking. Let appraisers B and C measure the same 10 parts
Note: Do not allow the appraisers to witness each other performing the
measurement. The reason is the same as why the unit markings are concealed,
TO PREVENT BIAS.
6. Repeat the measurements for all three appraisers, but this time present the
samples to each in a random order different from the original measurements.
This is to again help reduce bias in the measurements.
10 Parts 3 Trials 3 Appraisers

……
Revizia 2 din 07.02.2015 38 Academia TÜV Rheinland România
 Analysis Techniques: Variable Gage Analysis

The Average and Range Method:

A range control chart is created to determine if the measurement process is stable
and consistent. For each appraiser calculate the range of the repeated
measurements for the same part.

Range of Repeated Measures

Range  R  X Max - X Min

40

30
0.01MM

20

10
0
1A 2A 3A 4A 5A 1B 2B 3B 4B 5B 1C 2C 3C 4C 5C

Revizia 2 din 07.02.2015 39 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

The average range for each operator is then computed.

R Operator 
R
No. of Parts
The average of the measurements taken by an operator is calculated.

X Operator 
X
Trials * Parts
A control chart of ranges is created. The centerline represents the average range
for all operators in the study, while the upper and lower control limit constants are
based on the number of times each operator measured each part (trials).

Revizia 2 din 07.02.2015 40 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

R
R
No.of Operators
UCLR  D4 R

LCLR  D3 R

The centerline and control limits are graphed onto a control chart and the
calculated ranges are then plotted on the control chart. The range control chart is
examined to determine measurement process stability. If any of the plotted
ranges fall outside the control limits the measurement process is not stable,
and further analysis should not take place. However, it is common to have the
particular operator re-measure the particular process output again and use that
data if it is in-control.

Revizia 2 din 07.02.2015 41 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

Repeatability - Equipment Variation (E.V.)

R
EV  5.615 * *
 RK 1
d 2
The constant d2* is based on the number of measurements used to compute the
individual ranges(n) or trials, the number of parts in the study, and the number of
different conditions under study. The constant K1 is based on the number of times
a part was repeatedly measured (trials).
The equipment variation is often compared to the process output tolerance or
process output variation to determine a percent equipment variation (%EV).

EV
%EV (TOL)  * 100
USL  LSL
EV
%EV (PROC)  * 100
5.615 m

Revizia 2 din 07.02.2015 42 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

Reproducibility - Appraiser Variation(A.V.)

AV  X Diff * K2 
2  EV 2
- 
 nt



 
Xdiff is the difference between the largest average reading by an operator and the
smallest average reading by an operator. The constant K2 is based on the number
of different conditions analyzed. The appraiser variation is often compared to the
process output tolerance or process output variation to determine a percent
appraiser variation (%AV).

AV
%AV (TOL)  * 100
USL - LSL
AV
%AV (PROC)  * 100
5.615 m

Revizia 2 din 07.02.2015 43 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

Repeatability and Reproducibility( Gage R&R)

R & R  EV 2  AV 2
The gage error (R&R) is compared to the process output tolerance to estimate the
precision to tolerance ratio (P/T ratio). This is important to determine if the
measurement system can discriminate between good and bad output.

R&R
P /T  * 100
USL - LSL
The basic interest of studying the measurement process is to determine if the
measurement system is capable of measuring a process output characteristic with
its own unique variability. This is know as the Percent R&R (P/P ratio, %R&R),
and calculated as follows:

Revizia 2 din 07.02.2015 44 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

R&R
%R & R  * 100
5.615 m
Process or Total Variation:
If the process output variation (m) is not known, the total variation can be
estimated using the data in the study. First the part variation is determined:

PV  R p K 3
Rp is the range of the part averages, while K3 is a constant based on the number
of parts in the study.
The total variation (TV) is just the square root of the sum of the squares of R&R
and the part variation

 m  TV  R & R 2  PV 2

Revizia 2 din 07.02.2015 45 Academia TÜV Rheinland România

Analysis Techniques: Variable Gage Analysis

1. The ANOVA Method:

A weakness with the Average-Range method of using the range to determine
gage R&R is that it does not consider the variation introduced into a measurement
through the interaction between different conditions (appraiser) and the gage.
Consequently, to account for this variation an analysis of variance
method (ANOVA) is utilized. In addition, when the sample size increases, use of
the range to estimate the variation in not very precise. Furthermore, with software
packages readily available, the ANOVA method is a viable choice.
The total variation in an individual measurement equals:

 t2   p2   o2   po
2
  r2
The part to part variation is estimated by p2; the operator variation is estimated by
o2; the interaction effect is estimated by op2; while repeatability is estimated by r2

Revizia 2 din 07.02.2015 46 Academia TÜV Rheinland România

Variable MSA’s with MINITAB™

MINITAB™ calculates a column of variance components (VarComp) which are used to calculate %
Gage R&R using the ANOVA Method.

Measured Value True Value

Estimates for a Gage R&R study are obtained by calculating the variance components for each term
and for error. Repeatability, Operator and Operator*Part components are summed to obtain a total
Variability due to the measuring system.
We use variance components to assess the Variation contributed by each source of measurement
error relative to the total Variation.

Revizia 2 din 07.02.2015 47 Academia TÜV Rheinland România

 Session Window Cheat Sheet

Contribution of Variation to the total

Variation of the study.

% Contribution, based on variance

components, is calculated by dividing each
value in VarComp by the Total Variation then
multiplying the result by 100.

Use % Study Var when you are interested in

comparing the measurement system Variation to
the total Variation.
% Study Var is calculated by dividing each value in
Study Var by Total Variation and Multiplying by 100.
Study Var is calculated as 5.15 times the Standard
Deviation for each source.
(5.15 is used because when data are normally
distributed, 99% of the data fall within 5.15
Standard Deviations.)

Revizia 2 din 07.02.2015 48 Academia TÜV Rheinland România

Number of Distinct Categories

When the process tolerance is entered in the

system, MINITABTM calculates % Tolerance
which compares measurements system
Variation to customer specification. This allows
us to determine the proportion of the process
tolerance that is used by the Variation in the
measurement system.

Distinct Categories = .00000529 * 1.41

.00000450
= 1.1755 * 1.41
= 1.0842 * 1.41
= 1.5287
=1

Always round down to the nearest whole number.

Revizia 2 din 07.02.2015 49 Academia TÜV Rheinland România
Number of Distinct Categories

NUMBER OF DISTINCT CATAGORIES

This number represents the ability of your measurement

device to segment the total range of values.

AIAG recommends a minimum of 5 ndc

If you require more ndc than the study determined

 Run the study with more parts that represent the
entire range
 Improve the measurement tool to deliver
more precision

Revizia 2 din 07.02.2015 50 Academia TÜV Rheinland România

Number of Distinct Categories

The number of distinct categories tells you how many separate

groups of parts the system is able to distinguish.

Unacceptable for estimating

process parameters and
indices
Only indicates whether the
1 Data Category process is producing
conforming or nonconforming
parts

Generally unacceptable for

estimating process parameters
and indices
2 - 4 Categories Only provides coarse
estimates

Recommended

5 or more Categories

Revizia 2 din 07.02.2015 51 Academia TÜV Rheinland România

NUMBER OF DISTINCT CATAGORIES

There is a mathematic relation between ndc and

% Total Variation

You will need less than approximately 27% Total

Variation to have a minimum of 5 ndc

Revizia 2 din 07.02.2015 52 Academia TÜV Rheinland România

AIAG Standards for Gage Acceptance

% Tolerance
or % Contribution System is…
% Study Variance

10% or less 1% or less Ideal

10% - 20% 1% - 4% Acceptable

20% - 30% 5% - 9% Marginal

30% or greater 10% or greater Poor

Revizia 2 din 07.02.2015 53 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

Acceptability Criteria:
For a Gage deemed to be INCAPABLE for it’s application. The team must review
the design of the gage to improve it’s intended application and it’s ability to
measure critical measurements correctly. Also, if a re-calibration is required,
please follow caliberation steps.
If repeatability is large compared to reproducibility, the reasons might be:
1) the instrument needs maintenance, the gage should be redesigned
2) the location for gaging needs to be improved
3) there is excessive within-part variation.
If reproducibility is large compared to repeatability, then the possible causes
could be:
1) inadequate training on the gage,
2) calibrations are not effective,
3) a fixture may be needed to help use the gage more consistently.

Revizia 2 din 07.02.2015 54 Academia TÜV Rheinland România

 Analyze And Interpret Results MSA Output

Session Window Graphs

Two-Way ANOVA Table With Interaction Gage name:
Date of study:
Source DF SS MS F P Gage R&R (ANOVA) for Response Reported by :
Tolerance:
Misc:
Part 9 2.05871 0.228745 39.7178 0.00000
Operator 2 0.04800 0.024000 4.1672 0.03256 Components of Variation By Part
200 1.1
%Contribution
1.0
Operator*Part 18 0.10367 0.005759 4.4588 0.00016 %Study Var
0.9

Percent
%Toleranc e 0.8
Repeatability 30 0.03875 0.001292 100 0.7
0.6
Total Gage 59 2.24912 0.5
0 0.4
R&R %Contribution Gage R&R Repeat Reprod Part-to-Part Part 1 2 3 4 5 6 7 8 9 10

Source VarComp (of VarComp) R Chart by Operator By Operator

0.15 1 2 3 1.1
1.0

Sample Range
UCL=0.1252
Total Gage R&R 0.004437 10.67 0.9
0.10
0.8
Repeatability 0.001292 3.10 0.7
0.05
R=0.03833 0.6
Reproducibility 0.003146 7.56 0.5
0.00 LCL=0 0.4
Operator 0.000912 2.19 Operator 1 2 3

Operator*Part 0.002234 5.37 Xbar Chart by Operator Operator*Part Interaction

1.1 1 2 3 1.1 Operator
1.0 1.0 1
Part-To-Part 0.037164 89.33

Sample Mean
0.9 UCL=0.8796 0.9 2

Average
0.8 Mean=0.8075 0.8 3
Total Variation 0.041602 100.00 0.7
LCL=0.7354
0.7
0.6
StDev Study Var %Study Var %Tolerance 0.5 0.6
0.4 0.5
Source (SD) (5.15*SD) (%SV) (SV/Toler) 0.3 0.4
Part 1 2 3 4 5 6 7 8 9 10

Total Gage R&R 0.066615 0.34306 32.66 68.61

Repeatability 0.035940 0.18509 17.62 37.02
Reproducibility 0.056088 0.28885 27.50 57.77
Operator 0.030200 0.15553 14.81 31.11
Operator*Part 0.047263 0.24340 23.17 48.68
Part-To-Part 0.192781 0.99282 94.52 198.56 What does all this mean?
Let’s go through a complete interpretation…
Total Variation 0.203965 1.05042 100.00 210.08
Number of Distinct Categories = 4

Revizia 2 din 07.02.2015 55 Academia TÜV Rheinland România

 Analyze And Interpret Results Graphical Output

Where we’ll find them on the graphical output:

The things we’ll Gage name:
Date of study:

look for… Gage R&R (ANOVA) for Response Reported by:

Tolerance:
Misc:

1 Stability Components of
1.1
By Part
200 Variation %Contribution
1.0

2 Discrimination %Study Var

%T olerance
0.9
Percent

0.8
100 0.7
0.6
3 Repeatability 0
0.5
0.4
Gage R&R Repeat Reprod Part-to-Part Part 1 2 3 4 5 6 7 8 9 10

4 Product vs. R Chart by Operator By Operator

0.15 1 2 3 1.1

Measurement 1 2 3 UCL=0.1252 1.0

Sample Range

0.10 0.9
0.8

variation 0.05
R=0.03833
0.7
0.6
5
5 Reproducibility 0.00 LCL=0
0.5
0.4
Operator 1 2 3

Xbar Chart by Operator Operator*Part

1.1 1.1 Operator
1.0
1 2 3
1.0
Interaction 1
2
Sample Mean

0.9 UCL=0.8796 0.9

Av erage
0.8 Mean=0.8075 0.8 3
0.7 LCL=0.7354
0.7
0.6
0.5
0.4
0.6
0.5 5
0.3 0.4
4 5 Part 1 2 3 4 5 6 7 8 9 10

Revizia 2 din 07.02.2015 56 Academia TÜV Rheinland România

 Analyze And Interpret Results Stability: Range Chart

• What it is R Chart by operator

1 2 3
0.03
- System is not predictable from
one measurement to the next

Sample
Range
0.02
(measure of predictability)
0.01
- The property of being in statistical
0.00
control on the Range Chart
- Unpredictable measurements
obtained when the same operator
measures the same part with the
same gage
• Where you look for it

Size
- The Range Chart Measurement
of part #4
• What you’d like to see shows 1 point
clearly different
- All points below the upper control
1 2 3 4 5
limit on the Range Chart Part

Revizia 2 din 07.02.2015 57 Academia TÜV Rheinland România

Analyze And Interpret Results Discrimination: Range Chart

What it is
R Chart by operator
The ability of the measurement 0.015
1 2 3
UCL=0.01476
systems units to adequately identify
variation in a 0.010

Sample
Range
measured parameter R=0.005733
0.005
Insufficient discrimination results from
inadequate measurement units being 0.000 LCL=0

used to differentiate one part from Apply the equation:

another D = 1+ (UCL on Rchart / Smallest
Where you look for it Measurement Unit Change for a Given
Operator)
- Range Chart and raw data Example: D = 1 + (.015 / 0.001) = 16
Smallest unit
What you’d like to see Oper
1
Part
1
Trial
1
Measure
1.616
1 1 2 1.623
of change
1 1 3 1.622 is 0.001
Sub-group Size Min # of Units, D 1 2 1 1.640
1 2 2 1.638
2 4 1 2 3 1.642
1 3 1 1.650
3 5 1 3 2 1.651
4 5 1 3 3 1.653
1 4 1 1.630
5 5 1 4 2 1.633
1 4 3 1.629
6 6 1 5 1 1.618
1 5 2 1.620
1 5 3 1.624

Revizia 2 din 07.02.2015 58 Academia TÜV Rheinland România

 Analyze And Interpret Results Repeatability: Range Chart

What it is
The variation between R Chart by operator
0.015 1 2 3
successive measurements
of the same part, same

S a mp le
Ra n g e
0.010
characteristic, by the same
person using the 0.005
same instrument
0.000
Also called Test-Retest Error
and Operator Uncertainty
Operator 1:
Operator 2:
Where you look for it Repeatability is poor,
Repeatability is good,
Ranges are high
Ranges are low
Range Chart
What you’d like to see
No significant average
differences in ranges
between operators

Revizia 2 din 07.02.2015 59 Academia TÜV Rheinland România

 Analyze And Interpret Results Product vs. Measurement
Variation: Xbar Chart

What it is
1 2 3 4

An indication of Measurement System

variation as compared to part
variation
Where you look for it
Xbar Chart Example 1: All points inside control limits –
More measurement variation than part variation
What you’d like to see
Measurement variation is less than 1 2 3 4

product variation if the majority of the

points are randomly distributed
outside the control limits
Goal is to have 50% or more of the
points outside the control limits Example 2: Significant number of points
outside control limits – More part variation than
measurement variation

Revizia 2 din 07.02.2015 60 Academia TÜV Rheinland România

Analyze And Interpret Results Reproducibility: Xbar Chart

What it is Xbar Chart by Operator

Julie Matt Mike
Reproducibility refers to the variability
0.70
among appraisers

Sam p le
M ean
The difference between the average 0.65
measures of the different operators
0.60
Consists of Operator Bias and the
Operator by Part Interaction (next page) No Operator Bias
All inspectors have very similar pattern and average.
Where you look for it
Xbar Chart Xbar Chart by Operator
Julie Matt Mike
“By Operator” Chart 0.75

“Operator by Part Interaction” Chart

Sam ple
0.70

Mean
What you’d like to see 0.65

The same patterns on the Xbar Chart

Straight line connecting the averages on Operator Bias Present
the “By Operator” Chart Inspector C has same pattern but has higher readings.
Notice how an operator bias issue might make the
Coincident lines on the “Operator by measurement variation seem artificially small by
making more points appear outside the control limits.
Part Interaction” Chart

Revizia 2 din 07.02.2015 61 Academia TÜV Rheinland România

Analyze And Interpret Results Reproducibility: Operator By Part
Interaction

An operator by part interaction is when operators measure different

parts “differently” but consistently for a given part
This contribution to Reproducibility is only available when the ANOVA
option is chosen
If MINITAB determines the contribution is statistically insignificant (p
> .25), it will not use it in the calculations

Xbar Chart by Operator

Juli Matt Mike
0.70 e

Note the very

Sa m ple M

0.6
ean

5 different patterns
0.6
0

Revizia 2 din 07.02.2015 62 Academia TÜV Rheinland România

 Analyze And Interpret Results Tabular Output Metrics

In the MINITAB Session Window…

10.7% of the total variation is from the
measurement system
3.1% of the 10.7% is from repeatability
problems and the remaining variation is
from reproducibility problems
89% of the variation is from the parts

% Study
** % Tolerance

** See effective resolution in later slides Number of Distinct Categories

Now let’s see what these mean.

Revizia 2 din 07.02.2015 63 Academia TÜV Rheinland România

Analyze And Interpret Results % Study Variation

% Study Variation 
R&R *100
 TOTAL
• Looks at standard deviations instead of variance Acceptance
Criteria
• Measurement System Standard Deviation (R&R) as
a percentage of Total Observed Process % Study
Standard Deviation Variation

• Includes both repeatability and reproducibility

30%
• This is the numerical equivalent to seeing points out
of control on the Xbar Chart 10%
• If this value is low, we can effectively see the
part differences

Revizia 2 din 07.02.2015 64 Academia TÜV Rheinland România

Analyze And Interpret Results % Tolerance

Precision to Tolerance P/T

6 * R&R
% Tolerance  * 100
Tolerance
Measurement error as a percent of tolerance Acceptance
Criteria
Includes both repeatability and reproducibility
% Tolerance
6 * R&R = 99.73% of the measurement
system variation
30%
Less than 10% is ideal
10-30% depends on the situation – If you’re using the 10%
measurement system in an incapable process and
relying on it to “screen” parts, you’ll make a fair
number of “errors”
- Errors are passing bad parts, scrapping, or
reworking good parts

Revizia 2 din 07.02.2015 65 Academia TÜV Rheinland România

 Analyze And Interpret Results Distinct Categories


2

 2* 
Process Output
Number of Distinct Categories

2
R&R
Acceptance
The number of distinct categories within the process Criteria
data that the Measurement System can discern # of Distinct
How well a measurement process can detect process Categories
output variation-process shifts and improvement 0-3
This number represents the number of non-overlapping
confidence intervals that will span the range of 4
product variation 5 or
more

For example, If the number of distinct categories

was three, my Measurement System could only
put things into small-medium-large buckets.

Revizia 2 din 07.02.2015 66 Academia TÜV Rheinland România

Signal Averaging

Signal Averaging can be used to reduce Repeatability error when a better

gage is not available.
– Uses average of repeat measurements.
– Uses Central Limit theorem to estimate how many repeat measures
are necessary.

Signal Averaging is a method to

reduce Repeatability error in a
poor gage when a better gage is
not available or when a better
gage is not possible.

Revizia 2 din 07.02.2015 67 Academia TÜV Rheinland România

Signal Averaging Example

Suppose SV/Tolerance is 35%.

SV/Tolerance must be 15% or less to use gage.

Suppose the Standard Deviation for one part measured by one person many
times is 9.5.

Determine what the new reduced Standard Deviation should be.

Revizia 2 din 07.02.2015 68 Academia TÜV Rheinland România

Signal Averaging Example

Determine sample size:

Using the average of 6
repeated measures will
reduce the Repeatability
component of
measurement error to the
desired 15% level.

This method should be considered temporary!

Revizia 2 din 07.02.2015 69 Academia TÜV Rheinland România
Practical Conclusions

The Variation due to the measurement system, as a percent of study Variation is causing 92.21% of
the Variation seen in the process.
By AIAG Standards this gage should not be used. By all standards, the
data being produced by this gage is not valid for analysis.
% Tolerance
or % Contribution System is…
% Study Variance

10% or less 1% or less Ideal

10% - 20% 1% - 4% Acceptable

20% - 30% 5% - 9% Marginal

30% or greater 10% or greater Poor

Revizia 2 din 07.02.2015 70 Academia TÜV Rheinland România

 An Exercise – Calculating EV, AV, R&R, and TV

Given: EV = 5.15(s0) , AV = 5.25(s1)

R&R =  (EV2 + AV2)
TV =  (EV2 + AV2 + PV2)
Where: s0 = gauge standard deviation = 0.05
s1 = true appraiser standard
deviation = 0.1
PV = part-to-part variation = 0.02

Revizia 2 din 07.02.2015 71 Academia TÜV Rheinland România

 An Exercise – Calculating EV, AV, R&R, and TV

Calculate R&R and TV

Is the calculated R&R acceptable?

Revizia 2 din 07.02.2015 72 Academia TÜV Rheinland România

Repeatability and Reproducibility Problems

Repeatability Problems:
Calibrate or replace gage.
If only occurring with one operator, re-train.
Reproducibility Problems:
Measurement machines
Similar machines
Ensure all have been calibrated and that the standard measurement
method is being utilized.
Dissimilar machines
One machine is superior.
Operators
Training and skill level of the operators must be assessed.
Operators should be observed to ensure that standard procedures are followed.
Operator/machine by part interactions
Understand why the operator/machine had problems measuring some parts and
not others.
Re-measure the problem parts
Problem could be a result of gage linearity
Problem could be fixture problem
Problem could be poor gage design

Revizia 2 din 07.02.2015 73 Academia TÜV Rheinland România

Components of Variation

Whenever you measure anything, the variation that you observe can be
segmented into the following components…
Observed Variation

Unit-to-unit (true) Variation Measurement System Error

Precision Accuracy

Repeatability Reproducibility Stability Bias Linearity

All measurement systems have error. If you don’t know how much of the variation you
observe is contributed by your measurement system, you cannot make confident
decisions.

If you were one speeding ticket away from losing your license,
how fast would you be willing to drive in a school zone?

Revizia 2 din 07.02.2015 74 Academia TÜV Rheinland România

Discrimination (Resolution)

Smallest scale unit of measure for an instrument

10 to 1 rule +/- .005 inch?

.005/10 = .0005 inch
.005/4 = .00125 inch
Requirement varies based on application
 Cost of gage must be considered and weighed against ROI
(Micrometer-$200, or optical comparator-$15,000)

Revizia 2 din 07.02.2015 75 Academia TÜV Rheinland România

Discrimination found on SPC Range Chart

Not recommended If:

3 or less values are displayed on the
chart or
More that ¼ of the values are 0

Revizia 2 din 07.02.2015 76 Academia TÜV Rheinland România

 Improve Measurement System Discrimination Problems

What caused it
 Measures are rounded off – They now appear the same

 Round off by operator

 Round off by gage (set to the wrong decimal)

 One operator uses a different operating procedure

 Insufficient gage

What do you do about it

 Round off

 Instruct operator on how to use the full gage and interpolate

 Use all of the gage resolution

 SOP

 Replace gage with one that can measure smaller units

Does a discrimination problem have to fixed?

Depending on what you’re trying to do with the gage, maybe not – Consider the following:
 Engineering tolerance

 Process capability
Cost and difficulty in replacing device

Revizia 2 din 07.02.2015 77 Academia TÜV Rheinland România

Stability (Drift)

The change in the difference between measurement value and

reference value over time
 Electronic instruments may change over time due to the drifting of
values
 Operators may deviate in methods over time
 Temperature may vary over time (or per day)

Revizia 2 din 07.02.2015 78 Academia TÜV Rheinland România

Determining Stability

Obtain a part to use as a master reference value

 Can be done by averaging repetitive measurements of a master
over time
 Measure this master 5 times per shift over four weeks until at
least 20 subgroups are obtained
 Plot the data on Xbar/R Chart
 Monitor using standard SPC analysis for special cause

Revizia 2 din 07.02.2015 79 Academia TÜV Rheinland România

Determining Stability

Revizia 2 din 07.02.2015 80 Academia TÜV Rheinland România

 Improve Measurement System Stability Problems

What caused it
 Incorrect data entry/transposing data
 Operator misreads gage

 Lack of an SOP
 Operator changed technique during MSA

 Round off (potentially discrimination issues)

 True instability
 Debris on the part

 Other?
What can you do about it
 Improve the SOP and constrain the technique
 Capture data correctly

 Ensure gage is functioning properly

 If appropriate, ensure we are clocking or locating on the same feature each time
 Remove debris
 Other?

Revizia 2 din 07.02.2015 81 Academia TÜV Rheinland România

 Bias (Accuracy)

The Measurement Bias:

Using a certified sample, and a control chart of repeated measurements, the
bias of a measurement process can be determined. Bias is the
difference between the known value and the average of repeated
measurement of the known sample. Bias is sometimes called accuracy.

BIAS  Known  X
Process Variation = 6 Sigma Range
BIAS
Percent Bias =
Process Variation

Revizia 2 din 07.02.2015 82 Academia TÜV Rheinland România

 Bias (Accuracy)

Possible causes
 Out of calibration
 Worn or damaged fixture, equipment, instrument
 Wrong gage
 Environmental conditions
 Operator skill level, performed wrong method

Revizia 2 din 07.02.2015 83 Academia TÜV Rheinland România

 Linearity

Collective variation over the range of measurement

 If gage gains .1 inch per foot and you measure 6
inches than variation in linearity may be .5 of your
range
 This “may” be acceptable if tolerance is +/- 1 one
inch and total length measured is 6 inches.
 This would not be acceptable if total length is over 10
feet .1 x 10 = 1”

Revizia 2 din 07.02.2015 84 Academia TÜV Rheinland România

 Linearity Study

1. Choose 5 parts representing the full range of values

2. Determine each parts reference value
3. Have best operator randomly measure each part 12
times
4. Determine bias of each part
5. Enter data in software to create linearity plot

Revizia 2 din 07.02.2015 85 Academia TÜV Rheinland România

 Linearity Plot

Revizia 2 din 07.02.2015 86 Academia TÜV Rheinland România

 Linearity Results

 Result is a linearity problem

 R-Sq is only 71.4% (0 to 100% - Larger better)
 Bias line intercepted by regression line (Unacceptable if
significantly different than “0”)
 Distribution is bimodal (7 data points at value 4 & 6)

Revizia 2 din 07.02.2015 87 Academia TÜV Rheinland România

 Analysis Techniques: Variable Gage Analysis

8) Determine linearity and percent linearity:

Linearity = Slope x Process variation(m)
%Linearity = 100[linearity/Process Variation]
The acceptability criteria of Bias, Linearity depend on Quality Control
Plan, characteristic being measured and gage speciality, suggested
criteria of ESG is as following:
Under 5% - acceptable
5% to 15% - may be acceptable based upon importance of application,
cost of measurement device, cost of repairs, etc.,
Over 15% - Considered not acceptable - every effort should be made to
improve the system
The stability is determined through the use of a control chart. It is important
to note that, when using control charts, one must not only watch for
points that fall outside of the control limits, but also care other special
cause signals such as trends and centerline hugging.Guideline for the
detection of such signals can be found in many publications on SPC.

Revizia 2 din 07.02.2015 88 Academia TÜV Rheinland România

 Analysis Techniques

Attribute MSA

Revizia 2 din 07.02.2015 89 Academia TÜV Rheinland România

Attribute MSA

A methodology used to assess Attribute Measurement Systems.

Attribute Gage Error

Repeatability Reproducibility Calibration

– They are used in situations where a continuous measure cannot be

obtained.
– It requires a minimum of 5x as many samples as a continuous study.
– Disagreements should be used to clarify operational definitions for the
categories.
 Attribute data are usually the result of human judgment (which category does
this item belong in).
 When categorizing items (good/bad; type of call; reason for leaving) you need
a high degree of agreement on which way an item should be categorized.

Revizia 2 din 07.02.2015 90 Academia TÜV Rheinland România

 Attribute MSA Purpose

Accuracy checks
Assess standards against customers’ requirements
Identify how well Measurement System conforms to a “known master”
Precision checks
To determine if inspectors (appraisers) across all shifts, machines, lines, etc…
use the same criteria to evaluate items – Reproducibility
To quantify the ability of inspectors (appraisers) or gages to accurately repeat
their inspection decisions – Repeatability
To identify how well inspectors/gages measure a known master (possibly defined by
the customer) to ensure no misclassification occurs
How often operators decide to ship truly defective product
How often operators do not ship truly acceptable product
To determine areas where
Training is needed
Procedures or control plans are lacking
Standards are not clearly defined
Gage adjustment or correlation is necessary

Revizia 2 din 07.02.2015 91 Academia TÜV Rheinland România

Attribute MSA Purpose

An Attribute MSA is similar in many ways to the

continuous MSA, including the purposes. Do you
have any visual inspections in your processes? In
your experience how effective have they been?

Revizia 2 din 07.02.2015 92 Academia TÜV Rheinland România

Visual Inspection Test

Take 60 seconds to count the number of times “F” appears in this paragraph?

The Necessity of Training Farm Hands for First Class

Farms in the Fatherly Handling of Farm Live Stock is
Foremost in the Eyes of Farm Owners. Since the
Forefathers of the Farm Owners Trained the Farm Hands
for First Class Farms in the Fatherly Handling of Farm
Live Stock, the Farm Owners Feel they should carry on
with the Family Tradition of Training Farm Hands of First
Class Farmers in the Fatherly Handling of Farm Live
Stock Because they Believe it is the Basis of Good
Fundamental Farm Management.

Revizia 2 din 07.02.2015 93 Academia TÜV Rheinland România

Visual Inspection Test

36
The Necessity of Training Farm Hands for First Class
Farms in the Fatherly Handling of Farm Live Stock is
Foremost in the Eyes of Farm Owners. Since the
Forefathers of the Farm Owners Trained the Farm Hands
for First Class Farms in the Fatherly Handling of Farm
Live Stock, the Farm Owners Feel they should carry on
with the Family Tradition of Training Farm Hands of First
Class Farmers in the Fatherly Handling of Farm Live
Stock Because they Believe it is the Basis of Good
Fundamental Farm Management.

Revizia 2 din 07.02.2015 94 Academia TÜV Rheinland România

How can we Improve Visual Inspection?

Visual Inspection can be improved by:

 Operator Training & Certification
 Develop Visual Aids/Boundary Samples
 Establish Standards
 Establish Set-Up Procedures
 Establish Evaluation Procedures
 Evaluation of the same location on each part.
 Each evaluation performed under the same lighting.
 Ensure all evaluations are made with the same standard.

Look closely now!

Revizia 2 din 07.02.2015 95 Academia TÜV Rheinland România

 Attribute Data With Unordered Categories – Kappa

Suppose that invoice quality is a key to the process throughput. In other words, if an
invoice is incomplete, the rework required impacts the quantity that can be
processed in a day. Two appraisers are asked to independently evaluate ten
invoices randomly selected from different days. The results of the study are shown
below:
Invoice # Appraiser Appraiser 2 Agreement?
1 1 Bad Bad Y
2 Good Bad N
3 Good Good Y
4 Bad Bad Y
5 Good Good Y
6 Good Good Y
7 Good Bad N
8 Good Bad N
9 Good Good Y
10 Good Good Y
We could simply look at the percent of the time they agree as metric for between
Appraiser agreement: % agree % disagree
But what would that not take into account?

Revizia 2 din 07.02.2015 96 Academia TÜV Rheinland România

Definition Of Kappa

Kappa compares the proportion of agreement between appraisers after removing

agreement by chance.

Pobserved Pchance
Kappa 
1  Pchance
The proportion that the judges are in agreement is Pobserved.
The proportion expected to occur by chance is:

Pchance = (P Insp1 Good) (P Insp2 Good) + (P Insp1 Bad)(PInsp2 Bad)

1. What would the Kappa be if the judges agreed on every part?

2. How would you interpret a Kappa of zero?
3. What would be the Kappa if the judges never agreed?

Let’s walk through the calculation by hand.

Revizia 2 din 07.02.2015 97 Academia TÜV Rheinland România

How To Calculate Kappa – By Hand

appraiser 2 Build contingency table

Good Bad
Sum columns and rows
Add
Calculate Pobs by adding diagonal
Good

5/10 = .5 3/10 = .3 .8
appraiser 1

Calculate Pchance
Add

Add

Add Calculate Kappa

Bad

0/10 = 0 2/10 = .2 .2

.5 .5 Pobs = .5 + .2 = .7

Pobs - Pchance
Pchance = (PR1bad)(PR2bad)+(PR1good)(PR2good) Kappa = =(.7-.5)/(1-.5)
= (.2)(.5)+(.8)(.5) 1 - Pchance
= .5 How is this interpreted? = 0.4

Revizia 2 din 07.02.2015 98 Academia TÜV Rheinland România

How To Interpret Kappa Results

• The general rule for interpreting Kappa results are as follows:

<0 – Non-random disagreement
< 0.7 – Measurement System needs attention
0.7 – 0.9 – Generally acceptable, improvement may be needed
depending on application and risk
> 0.9 – Excellent Measurement System

Remember, consider your Measurement System

and how well the above criteria might apply.

Revizia 2 din 07.02.2015 99 Academia TÜV Rheinland România

 MINITAB Output – Session Window

Attribute Gage R&R Study

Attribute Gage R&R Study for J1T1, J1T2, J2T1, J2T2
Within Appraiser
Assessment Agreement
Appraiser # Inspected # Matched Percent (%) 95.0% CI
1 20 19 95.0 ( 75.1, 99.9) Since Kappa > 0.70 for both
2 20 18 90.0 ( 68.3, 98.8) Appraisers, both agree well
# Matched: Appraiser agrees with him/herself across with themselves
trials.
Kappa € Adequate repeatability
Appraiser Response
Statistics Kappa SE Kappa Z P(vs. > 0)
1 b 0.8997 0.2236 4.0238 0.000
g 0.8997 0.2236 4.0238 0.000 A list of the approximate standard
2 b 0.8000 0.2236 3.5777 0.000 errors for the estimated Kappa
g 0.8000 0.2236 3.5777 0.000 statistic. A standard error for an
Between Appraisers
Assessment Agreement estimated Kappa statistic measures
# Inspected # Matched Percent 95.0% CI the precision of the estimate. The
20 (%) 18 90.0 ( 68.3, 98.8) smaller the standard error, the more
# Matched: All assessments agree with each precise the estimate.
appraisers' Kappa r.
othe
Respons Statistics
Kappa SE Z P(vs. > 0)
e b 0.8833 Kappa 9.6756 0.000
g 0.8833 0.0913 9.6756 0.000 Since Kappa > 0.70 for the
0.0913
Between Appraisers,
€ Adequate reproducibility

Revizia 2 din 07.02.2015 100 Academia TÜV Rheinland România

Guidelines For Kappa Studies (Cont’d)

Analysis- Review the repeatability portion first (Within Appraiser Kappa), if an

appraiser cannot agree with themselves, ignore comparisons to other
appraisers and go understand why (see Improvement below)
- For appraisers that have acceptable repeatability, review the reproducibility
portion (Between Appraiser)
- If a “gold standard” is available (ratings of the samples known by some other
means as being “correct”), compare each appraiser to them for “calibration”
• Use the field in MINITAB, “Known Standard Attribute”

Improvement - If the Within Appraiser Kappa scores are low, that appraiser may
need training. Do they understand characteristic they are looking for? Are
the instructions clear to them?
- If the Between Appraiser Kappa scores are low, each appraiser may have a
differing definition of the categories – A standardized definition can improve
this situation
- If improvements are made, the study should be repeated to confirm
improvements have worked

Revizia 2 din 07.02.2015 101 Academia TÜV Rheinland România

 Attribute MSA With Ordered Categories – Kendall’s
Coefficient Of Concordance

Recall what ordinal data is – Categorical variables that have three or more
possible levels with a natural ordering, such as strongly disagree, disagree,
neutral, agree, and strongly agree; or use a numeric scale such as 1-5
When the attribute data can be represented by three or more categories that can be
arranged in a rank order, Kendall’s Coefficient of Concordance (KCC) can be used
to evaluate the measurement system
KCC ranges from 0 to 1
0 1
No Strong
association association
Unlike Kappa, KCC does not treat misclassifications equally – e.g., the difference
between a “mild” and “medium” is less than the difference between “mild” and “very
hot”
KAPPA: Pass/Fail
KCC: Mild/Medium/Hot/Very Hot (Hot sauce)
Let’s walk through an example to
see how these are calculated.

Revizia 2 din 07.02.2015 102 Academia TÜV Rheinland România

 Kendall’s Coefficient Of Concordance Example

Three judges score the quality of a proposal. The scale they use is 1-5, 1
being “poor”, 5 being “excellent.” The results from their scoring are provided
in the following table:

Proposal Judge 1 Judge 2 Judge 3

1 4 3 3
2 4 2 4
3 5 4 5
4 3 4 4
5 3 2 1
6 2 3 2

Would you like to see more proposal-to-proposal variation or

Judge-to-Judge?
Let’s use KCC to assess the degree of rater association.

Revizia 2 din 07.02.2015 103 Academia TÜV Rheinland România

Kendall’s Analysis In MINITAB

1. Put data into MINITAB, each judges trial in a separate column

2. To analyze, go to Stat € Quality Tools € Attribute Agreement Analysis

Revizia 2 din 07.02.2015 104 Academia TÜV Rheinland România

 Kendall’s Analysis In MINITAB (Cont’d)

3. Select columns that contain

the values
4. Enter Judges (appraisers)
and quantity
5. Check the box to show that
the categories are ordered 3 6
(this will trigger the
Kendall’s calculation) 4
6. Select “Results” button
7. Click last option to display
the results
5 8

Hit OK twice to execute

7

Revizia 2 din 07.02.2015 105 Academia TÜV Rheinland România

MINITAB Output – Session Window

You still get the Kappa Statistics, but

you won’t need to use them here.

First we need to look at the p-value.

Remember, KCC > 0, indicates association, the
higher the p-value, the stronger that
association is. The p-value tells you the
probability that some association was found
purely by chance. This is saying that there is a
7.4% chance a non-zero KCC (some level of
association) was found by chance alone. You
decide based on the amount of risk you’re
willing to take. A value like 0.05 is typical. The
KCC is fairly low here, see criteria on next
slide…

Revizia 2 din 07.02.2015 106 Academia TÜV Rheinland România

 How To Interpret Kendall’s Coefficient Of Concordance

• In general, KCC can vary from 0.0 to 1.0

• The higher the value, the higher degree association among the
assessments made by the appraisers
• How close to 1.0 is needed?
- Although very situational dependent, the general guidelines are
as follows:
0-0.7 – Low degree of association, measurement system
needs attention
0.7-1.0 – Generally acceptable

As stated on the prior page, the p-value for KCC should also be low,
generally less than 0.05 – This reduces your risk of getting an
acceptable KCC just by random chance.

Revizia 2 din 07.02.2015 107 Academia TÜV Rheinland România

Guidelines For Kendall’s Studies

Planning
Sample size
More is better – As your sample size increases, your confidence
intervals around your KCC decrease
Collect as many samples as practically possible, 20 minimum is a
guideline, ≥ 30 is best
Perform at least two trials per appraiser
Sample part selection
Parts in the study should represent the full range of variation and
thus utilize the full range of the rating scale
Execution
Parts should be rated in random order independently (no comparisons)
Study should be blind
Rating time should be similar to that “normally” used

Revizia 2 din 07.02.2015 108 Academia TÜV Rheinland România

Guidelines For Kendall’s Studies (Cont’d)

Analysis
Prior to reviewing the KCC value, check to see that the p-value is low (generally < 0.05)
– If it is not, add more samples to the study or add another trial
Review the repeatability portion first (Within Appraiser), if an appraiser’s KCC is very low,
he/she may need improvement (see Improvement below)
For appraisers that have acceptable repeatability, review the reproducibility portion
(Between Appraiser)
If a “gold standard” is available (ratings of the samples known by some other means as being
“correct”), compare each appraiser to them for “calibration”
Use the field in MINITAB, “Known Standard Attribute”
Improvement
If the Within appraiser KCC scores are low, that appraiser may need training. Do they
understand the rating scale? Are the instructions clear to them?
If the Between Appraiser KCC scores are low, each appraiser may have a differing
definition of the rating scale – A standardized definition can improve this situation
If improvements are made, the study should be repeated to confirm improvements have
worked

Revizia 2 din 07.02.2015 109 Academia TÜV Rheinland România

Summary

At this point, you should be able to:

 Understand Precision & Accuracy

 Understand Bias, Linearity and Stability

 Understand Repeatability & Reproducibility

 Understand the impact of poor gage capability on product quality

 Identify the various components of Variation

 Perform the step by step methodology in Variable and Attribute MSA’s

Revizia 2 din 07.02.2015 110 Academia TÜV Rheinland România

 Questions

Revizia 2 din 07.02.2015 111 Academia TÜV Rheinland România

 TÜV Rheinland România

Vă mulţumim pentru atenţie!

Succes la examen!

www.tuv.ro
[email protected]

Revizia 2 din 07.02.2015 112 Academia TÜV Rheinland România