Number

1.8
ASSOCIATION CONNECTING
ELECTRONICS INDUSTRIES ® Subject
Measurement Precision Estimation for Binary Data
2215 Sanders Road
Northbrook, IL 60062-6135
Date Revision
01/03 A

IPC-TM-650 Originating Task Group

TEST METHODS MANUAL Measurement Precision Task Group (7-11a)

1 Scope Tests performed on presumably identical samples tested. If it is known that precision is worse at one end of the
under seemingly identical conditions do not always yield iden- range, evaluation could be limited to that end of the range. In
tical results. This is due to errors inherent in every measure- general, evaluations are generally advisable for all combina-
ment or evaluation. During the development of a new test tions of materials, levels, set-ups, and conditions. If resources
procedure or use of an existing test procedure, this variability are limited, begin the study with those combinations deemed
must be understood and precautions taken to ensure that it is to be the most critical, or where measurement error is likely to
controlled to within necessary limits. Performance of this test be greatest.
method will help to estimate measurement error and trouble- The number of samples will also depend on the difficulty
shoot causes of measurement variability. Use of this test involved in obtaining, processing, and distributing the test
method will provide some evidence that a new test procedure specimens, the difficulty, length of time required for, and
is suitable for use when submitted for review, or an existing expense of performing the test, and other prior known infor-
test procedure is capable of measuring the applicable param- mation.
eter.
This test method will assume that evaluations can be repeated
This method provides a standard procedure for determining on the same samples. For situations where this is not possible
the precision of a test method involving binary data or tests or the sample is consumed during the test, other methods
that result in two outcomes. These include evaluations where may be better suited (see 6.1).
the results are recorded as pass/fail or go/no-go. Examples
include solderability tests and visual inspections. This method 4 Apparatus The apparatus used will be as specified by
helps to estimate how often the disposition is performed cor- the test procedure under investigation.
rectly.
5 Procedure
This method is not useful for measurements which result in
variables data, or where more than three repeated measure-
5.1 Planning Evaluation Keep the evaluation as simple as
ments or more than ten testers are used. These situations are
possible to obtain data that is free of unintended secondary
covered under other methods (see 6.1).
effects.
1.1 Definitions Prepare a procedure that is complete and describes the test
Accuracy – The difference between an observed measure- parameters as well as recommended techniques for assess-
ment and the true (but perhaps unknown) value being mea- ing the outcome. Include known best practices and draw
sured. extensively on the experience of test users.

Precision – The closeness to each other of repeated mea- The method used in this procedure allows for up to 10 test
surements of the same quantity. conditions. Solicit participants from among the community of
facilities with the proper equipment, competent operators and
Binary Data – Inspections or tests in which parts are placed familiarity with the test. In order to obtain representative pre-
in one of two classes. This includes pass/fail, go/no-go tests cision estimates, do not select only from a small group of
and inspections. users who are considered exceptionally qualified. Be sure to
specify any special calibration procedures or material prepara-
2 Applicable Documents The test procedure under evalu- tion requirements.
ation.
The analysis method used in this procedure allows for up to
10 repeated evaluations per sample. Carefully evaluate the
3 Test Specimens The test specimens used will be as
materials to determine the appropriate classification or dispo-
specified in the test procedure under investigation.
sition before the study. Choose material representing a likely
The number and types of test materials to be used will range of conditions normally encountered during routine tests
depend on the range of levels in the class of materials to be or inspections. Randomize the samples prior to dividing into

Material in this Test Methods Manual was voluntarily established by Technical Committees of IPC. This material is advisory only
and its use or adaptation is entirely voluntary. IPC disclaims all liability of any kind as to the use, application, or adaptation of this Page 1 of 6
material. Users are also wholly responsible for protecting themselves against all claims or liabilities for patent infringement.
Equipment referenced is for the convenience of the user and does not imply endorsement by IPC.
 IPC-TM-650
Number Subject Date
1.8 Measurement Precision Estimation for Binary Data 01/03

Revision
A

test groups. Prepare more than the material required to Table 1 Recommended evaluation criteria
ensure an adequate amount is available for the study in case Metric Acceptable Marginal Inadequate
of lost or damaged specimens, errors, test set-up, etc. E >0.9 0.8 to 0.9 <0.8
Carefully package and label the material. Assign serial num- P(FR) <0.05 0.05 to 0.10 >0.10
bers, if possible. Identify the version of the test procedure. P(FA) <0.02 0.02 to 0.05 >0.05
Specify care and handling procedures. Provide a data sheet,
and describe any documentation required. Require a test log, If the test effectiveness is inadequate, then steps should be
and insist that observations of any unusual events be taken to diagnose and improve the causes of the deficiency.
recorded. The probabilities of false acceptance and false rejection
should help in this diagnosis. Marginal tests should also be
5.2 Conducting the Evaluation Ensure the samples are improved.
inspected on receipt. Send replacement material if damaged
An acceptable test effectiveness rating (E) indicates that the
or tests are performed improperly.
test method dispositions the products with reasonable cor-
Inspect the data sheets when returned. Review the test logs rectness.
for unusual events. Review the results. Question unusual dis- The results of this evaluation should be compared to the test
positions or comments. Incorrect dispositions and typos must efficiency goals for this inspection. The rules of thumb noted
be fixed prior to analysis. above have been found to be useful. These goals could be
amended, depending on the criticality of the inspection, and
5.3 Analyzing the Data Analysis may be performed on the the impact of incorrect disposition.
data sheet or on the Excel spreadsheet (see 6.2).
6 Notes
The basic techniques involve beginning with a set of parts or
materials for which the classification has been previously
6.1 Methods for Analyzing Repeatability and Reproduc-
determined. Several inspectors or testers then examine and
ibility This test method covers situations where the mea-
classify the parts and the results are compared with the
surements result in binary data, such as go and no-go, or
known standard classification.
pass and fail tests. The precision of the test is determined by
The effectiveness of the test is the number of correct determi- calculating the consistency and correctness of the sample
nations divided by the total number of classification opportu- dispositions.
nities (number of parts times the number of inspectors). Measurements that result in variables data can be analyzed
using IPC Test Method IPC-TM-1.9.
Number of correct dispositions
E = (1)
Number of parts x Number of testers In some cases, the measurement cannot be repeated more
than once on the same sample. This is common where the
The probability of a false reject and the probability of a false
sample is consumed during the test, such as chemical analy-
accept can be defined as follows:
sis, or changed during testing, such as solderability evalua-
Number of dispositions where good parts were rejected tions. In these cases, the analysis using a modified average
P(FR) = (2)
Number of good parts x Number of testers and range method is possible. This method is under develop-
ment.
Number of dispositions where bad parts were accepted
P(FA) = (3)
Number of bad parts x Number of testers
6.2 References
5.4 Preparing Analysis Conclusions Goals for measure- a. ISO 5725-1 Accuracy (trueness and precision) of measure-
ment precision should be established before the study begins. ment methods and results (parts 1 to 6), 1998(E), Interna-
The goals should be established using knowledge of the tional Organization for Standardization, Geneva, Switzer-
anticipated levels of product variability (or process capability), land (www.iso.org).
specifications, customer needs and the possible impact of b. Measurement Systems Analysis, 2nd edition, June 1998,
dispositioning test samples improperly. As a rule of thumb, the Automotive Industry Action Group (AIAG), 26200 Lahser
guidelines shown in Table 1 have been extensively applied. Road, Southfield, MI 48034 (www.aiag.org).

Page 2 of 6
 IPC-TM-650
Number Subject Date
1.8 Measurement Precision Estimation for Binary Data 01/03

Revision
A

c. Standard Practice for Conducting an Interlaboratory Study can be used for this purpose. Reference (a) is an Excel
to Determine the Precision of a Test Method, E691-99, spreadsheet written to perform the calculations in this proce-
ASTM, Philadelphia, PA (www.astm.org). dure.
d. Concepts for R&R Studies, Larry B. Barrentine, (ISBN a. Measurement Precision Calculator For Binary Data, Excel
0-87389-108-2), ASQC Press, Milwaukee, WI ((www.quali- spreadsheet, available at http://www.ipc.org/html/
typress.asq.org). testmethods.htm, free of charge.
e. Basic Statistics, 4th Edition, Mark J. Kiemele, Stephen R. b. Statgraphics Plus, Manugistics Corp, 2115 East Jefferson
Schmidt, Ronald Berdine, Air Academy Press, 1997, ISBN Street, Rockville, MD, 20852-4999 (www.statgraphic-
1-880156-06-7, pages 9-71 to 9-77 s.com).
f. ‘‘Is 100% Test 100% Effective,’’ W. Russell, 1998 IPC c. SPC XL, Air Academy Press, 1155 Kelly Johnson Blvd,
EXPO, San Jose, CA (gives methods for calculating the Colorado Springs, CO 80920 (www.airacad.com).
likely outcomes on product test for differing levels of mea- d. Minitab, Minitab. Inc., 3081 Enterprise Dr, State College,
surement precision.) PA 16801 (www.minitab.com).
6.3 Software Measurement precision studies are greatly e. Interlaboratory Data Analysis Software for E691, ASTM,
facilitated by use of software to perform the calculations. 100 Barr Harbor Dr, West Conshohocken, PA 19428
Below are just a few of the many software packages which (www.astm.org).

Page 3 of 6
 IPC-TM-650
Number Subject Date
1.8 Measurement Precision Estimation for Binary Data 01/03

Revision
A

Measurement Precision Study – Binary Data

Table 1: Data Entry Form

Enter test results into the table below.
Samples
Tester 1 2 3 4 5 6 7 8 9 10
True Standard
1
2
3
4
5
6
7
8
9
10

Calculations

Table 2: Samples Dispositioned Correctly

Score a ‘‘1’’ where disposition in Table 1 above matched the true standard.
Score a ‘‘0’’ where disposition did not match the true standard.
Note these scores for each of the testers in the table below.
Samples
Tester 1 2 3 4 5 6 7 8 9 10 Total
True Standard
1
2
3
4
5
6
7
8
9
10

Page 4 of 6
 IPC-TM-650
Number Subject Date
1.8 Measurement Precision Estimation for Binary Data 01/03

Revision
A

Table 3: Good Parts That Were Rejected

Score a ‘‘1’’ where good parts were rejected in Table 1 above.
Score a ‘‘0’’ everywhere else.
Note the scores for each tester in the table below.
Samples
Tester 1 2 3 4 5 6 7 8 9 10 Total
True Standard
1
2
3
4
5
6
7
8
9
10

Table 4: Bad Parts That Were Accepted

Score a ‘‘1’’ where bad parts were accepted in Table 1 above.
Score a ‘‘0’’ everywhere else.
Samples
Tester 1 2 3 4 5 6 7 8 9 10 Total
True Standard
1
2
3
4
5
6
7
8
9
10

Page 5 of 6
 IPC-TM-650
Number Subject Date
1.8 Measurement Precision Estimation for Binary Data 01/03

Revision
A

Table 5: Measurement System Scorecard

Tester
Results 1 2 3 4 5 6 7 8 9 10 Total
Disposed
correctly
Good and
rejected
Bad and
accepted

Total tests Number of testers times the number of parts.

Acceptable parts Count acceptable parts in the True Standard line of the data input table.
Rejectable parts Count rejectable parts in the True Standard line of the data input table.
# of testers Count the number of participants.

Table 6: Measurement System Effectiveness

Needs
Metric Calculation Result Acceptable Improvement
Total parts dispositioned correctly
Test effectiveness (%) x 100 >90 <80
Total parts tested
Total good and rejected parts
Probability of false rejects (%) x 100 <5 >10
(No. of testers) x (Good parts)
Total bad and accepted parts
Probability of false acceptance (%) x 100 <2 >5
(No. of testers} x (Bad parts)

Page 6 of 6