Designation: E 1601 – 98 (Reapproved 2003)e1

Standard Practice for

Conducting an Interlaboratory Study to Evaluate the
Performance of an Analytical Method1
This standard is issued under the fixed designation E 1601; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (e) indicates an editorial change since the last revision or reapproval.

e1 NOTE—Caution notes were moved into the text editorially in November 2003.

1. Scope 3.2.1 interlaboratory study (ILS)—study undertaken to as-

1.1 This practice covers procedures and statistics for an certain if a test method is suitable for its intended use. The ILS
interlaboratory study (ILS) of the performance of an analytical includes preparation, testing, and evaluation phases.
method. The study provides statistical values which are useful 3.2.2 interlaboratory test—measures the variability of re-
in determining if a method is satisfactory for the purposes for sults when a test method is applied many times in a number of
which it was developed. These statistical values may be laboratories.
incorporated in the method’s precision and bias section. This 3.2.3 replicate results—results obtained by applying a test
practice discusses the meaning of the statistics and what users method a specified number of times to a material.
of analytical methods may learn from them. 3.2.4 result—the numerical value obtained by applying a
1.2 This standard does not purport to address all of the test method once to a material.
safety concerns, if any, associated with its use. It is the 3.2.5 test method—gives directions for producing a single
responsibility of the user of this standard to establish appro- result.
priate safety and health practices and determine the applica- 3.2.6 test protocol—gives instructions to each participating
bility of regulatory limitations prior to use. laboratory, detailing the way it is to conduct its part of the
interlaboratory test program.
2. Referenced Documents 3.3 Unless the test method destroys the test portion each
2.1 ASTM Standards: 2 time it is applied, the protocol for a Plan A test specifies, if
E 135 Terminology Relating to Analytical Chemistry for possible, replicate results on a single test portion (which may
Metals, Ores, and Related Materials be in solution). The protocol for a Plan B test specifies the
E 691 Practice for Conducting an Interlaboratory Study to number of test portions of a material and requires duplicate
Determine the Precision of a Test Method results (2 only) on each portion (which may be in solution).
E 1169 Guide for Conducting Ruggedness Tests
4. Summary of Practice
E 1763 Guide for Interpretation and Use of Results from
Interlaboratory Testing of Chemical Analysis Methods 4.1 Instructions are provided for planning and conducting a
cooperative evaluation of a proposed analytical method.
3. Terminology 4.2 The following list describes the organization of this
3.1 Definitions—For definitions of terms used in this prac- practice:
tice, refer to Terminology E 135. 4.2.1 Sections 1-5 define the scope, significance and use,
3.2 Definitions of Terms Specific to This Standard: referenced documents, and terms used in this practice.
4.2.2 Section 6 helps users of analytical methods understand
and use the statistics found in the Precision and Bias section of
methods.
1
This practice is under the jurisdiction of ASTM Committee E01 on Analytical 4.2.3 Sections 7 and 8 instruct the ILS coordinator and
Chemistry for Metals, Ores, and Related Materials and is the direct responsibility of
Subcommittee E01.22 on Statistics and Quality Control. members of the task group on how to plan and conduct the
Current edition approved Oct. 1, 2003. Published November 2003. Originally experimental phase of the study.
approved in 1994. Last previous edition approved in 1998 as E 1601 – 98.
2
4.2.4 Section 9 discusses the procedures for collecting,
For referenced ASTM standards, visit the ASTM website, www.astm.org, or
contact ASTM Customer Service at [email protected]. For Annual Book of ASTM
evaluating, and disseminating the data from the interlaboratory
Standards volume information, refer to the standard’s Document Summary page on test.
the ASTM website. 4.2.5 Section 10 presents the statistical calculations.

Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, United States.

1
 E 1601 – 98 (2003)e1
4.2.6 Sections 11 and 12 discuss the use of statistics to analytical instruments and equipment must be in good condi-
evaluate a test method and the means of incorporating the ILS tion; and (4) the method must be performed exactly as written.
statistics into Precision and Bias statements. 6.2.1 Reproducibility Index, R—This statistic estimates the
4.2.7 The Annex A1 gives the rationale for the calculations expected range of differences in results reported from two
in Section 10. laboratories, a range that is not exceeded in more than 5 % of
such comparisons. Use R to predict how well your results
5. Significance and Use should agree with those from another laboratory: First, obtain
5.1 Ideally, interlaboratory testing of a method is carried out a result under the conditions stated in 6.2, then add R to, and
by a randomly chosen group of laboratories that typifies the subtract R from, this result to form a concentration confidence
kind of laboratory that is likely to use the method. In actuality, interval. Such an interval has a 95 % probability of including a
this ideal is only approximated by the laboratories that are result obtainable by the method should another laboratory
available and willing to undertake the test work. The coordi- analyze the same sample. For example, a result of 46.57 % was
nator of the program must ensure that every participating obtained. If R for the method at about 45 % is 0.543, the 95 %
laboratory has appropriate facilities and personnel and per- confidence interval for the result (that is, one expected to
forms the method exactly as written. If this goal is achieved, include the result obtained in another laboratory 19 times out of
the statistics developed during the ILS will be adequate for 20) extends from 46.03 to 47.11 %.
determining if the method is capable of producing satisfactory NOTE 1—For those not conversant with statistical concepts, it is
precision in actual use. If the program includes certified important to realize that in most such comparisons, the differences will be
reference materials, the test data also provide information much smaller than the confidence interval implies. The 50 % confidence
concerning the accuracy of the method. The statistics provide interval is only about one third (34.6 %) as wide. Thus, the “average”
interval for the above result (one expected to include the result obtained
a general guide to the expected performance of the method in by another laboratory half the time) extends from 46.4 to 46.8 %. The
the laboratories of those who will use it. obvious implication is that, although half the differences will be more than
0.2 %, half will be less than 0.2 %.
6. Statistical Guide for the Users of Analytical Methods 6.2.2 Repeatability Index, r—This statistic is given in the
Evaluated in Accordance With This Practice method only if the interlaboratory test was designed to measure
6.1 Standard Deviations: sr. It estimates the expected range of results reported in the
6.1.1 Minimum Standard Deviation of Method, sM—This same laboratory on different days, a range that is not exceeded
statistic measures the precision of test results under conditions in more than 5 % of such comparisons.
of minimum variability. Because it is improbable that a method
in ordinary use will exhibit precision this good, no predictive 7. Interlaboratory Test Planning
index is calculated for sM. Users adept in statistics may wish to 7.1 Analytical test methods start from a perceived need to
compare sM and the short-term standard deviation of the service one or more material specifications.
method measured in their laboratory. For most methods, 7.1.1 Develop a performance requirement for a method
short-term variability refers to results obtained within several from the material specification(s). Include the following fac-
minutes. (Caution—The standard deviation of results obtained tors: expected ranges of chemical compositions of the materials
on different occasions, even in the same laboratory, probably to be covered (method’s general scope); specified elements and
will exceed sM.) their concentrations (determination concentration ranges); and
6.1.2 Between-Laboratory Standard Deviation, sR—This the precision required.
statistic is a measure of the precision expected for results 7.1.2 Prepare a table of the elements and concentration
obtained in different laboratories. It reflects all sources of ranges to cover the critical values in the material specifications.
variability that operate during the interlaboratory test (except Use this information together with knowledge of the charac-
test material inhomogeneity in tests designed to eliminate that teristics of the candidate analytical method to select test
effect). It is used to calculate the reproducibility index, R. Use materials for the interlaboratory program.
sR for evaluating the precision of methods. It represents the 7.2 Draft Method—The process of developing methods and
expected variability of results when a method is used in testing them in a preliminary way is beyond the scope of this
different laboratories. practice. All analytical skill and experience available to the
6.1.3 Within-Laboratory Standard Deviation, sr—This sta- task group must be exerted to ensure that the method will meet
tistic cannot be calculated in a normal interlaboratory test. It is the project requirements in 7.1 and that it is free of technical
determined only in tests designed to measure variability within faults. A preliminary, informal test of a method must be carried
laboratories. When this statistic is given in a method, it reflects out in several laboratories before the final draft is prepared.
all variability that may occur from day-to-day within a labo- Individuals responsible for selecting the method may find
ratory (for example, from calibration, standardization, or envi- helpful information in Practice E 691 and Guide E 1169. The
ronmental changes). It is used to calculate the repeatability formal interlaboratory test must not start until the task group
index, r. The user is cautioned that additional sources of reaches consensus on a clearly written, explicitly stated, and
variation may affect results obtained in other laboratories. unambiguously worded draft of the method in ASTM format,
6.2 Predictive Indexes—For the following indexes to apply, which has completed editorial review.
these conditions must be met: (1) the test materials must be 7.3 Test Materials—Appropriate test materials are essential
homogeneous; (2) analysts must be competent and diligent; (3) for a successful ILS. The larger the number of test materials

2
 E 1601 – 98 (2003)e1
included in the test program, the better the statistical informa- NOTE 2—If all reasonable effort fails to recruit at least six cooperating
tion generated. On the other hand, the burden of running a very laboratories, up to two of the recruited laboratories may each volunteer to
large number of materials may reduce the number of labora- submit two independent sets of test data as an expedient to provide a total
of at least six sets of data. Minimum requirements for independence are
tories willing to participate. A method must cover a concen- that two typical analysts, who do not consult with each other about the
tration range extending both above and below the specified method, perform the test protocol on different days. They should use
value(s). If possible, provide test materials near each limit. separate equipment if possible and must not share calibration solutions or
Concentration ranges covering several orders of magnitude calibration curves.
should be tested with three or more materials.
7.3.1 Material composition and form must be within the 8. Conducting the Interlaboratory Test
general scope of the method. If possible, include all material 8.1 Program Coordinator—One individual (presumably the
types the scope is expected to cover. Often, only limited task group chairman) will coordinate the entire ILS. A pro-
numbers of certified reference materials are available. Use spective ILS program coordinator will find helpful information
those that best meet the criteria for the test. If they do not cover on conducting the program in Practice E 691. One way to
all concentration levels, find or prepare other materials to fill in organize the work to provide close control while moving the
missing values. program steadily to its conclusion is as follows:
7.3.2 The quantity of the material must be sufficient to 8.1.1 Prepare a draft of the method to be tested.
distribute to all laboratories participating in the test with about 8.1.2 Recruit a task group of participating laboratories.
50 % held in reserve to cover unforeseen eventualities. 8.1.3 Select a set of test materials and assemble them into
7.3.3 Materials should be homogeneous on the scale of the kits, one for each laboratory.
test portion consumed in each determination as well as among 8.1.4 Write the test protocol to instruct each laboratory how
the portions sent to different laboratories. Usually certified to run the test.
reference materials have been tested for homogeneity, but test 8.1.5 Prepare a report form.
materials from other sources may have had only a minimal 8.1.6 Establish a realistic time schedule for each part of the
examination. The use of laboratory-scale melting and casting test program.
to produce test materials can sometimes lead to segregation of 8.1.7 Assemble and deliver to each participating laboratory
one or more components in an alloy. Unless specially gathered everything needed to run the test: the draft method; the test
or prepared materials have been subjected to a thorough materials and a document which describes them; the test
homogeneity test, they require the use of Test Plan B. It protocol; the report forms; and a cover letter which includes the
statistically removes the effect of moderate test material deadline for return of results; and the name, address, telephone,
inhomogeneity from the estimates of the ILS statistics. and FAX numbers of the person who will handle problems and
7.3.4 Test material sent to each laboratory must be perma- receive the completed report forms.
nently marked with its identity in such a manner that the 8.1.8 Expedite the laboratory testing. Follow up to be sure
identification is not likely to be lost or obliterated. that the laboratories receive the test materials and understand
7.3.5 If the test program is to evaluate the accuracy of the what is expected of them. Encourage laboratories to complete
method, at least one test material must be certified for the the work.
concentration of each element. More certified materials pro- 8.1.9 Inspect results on each report form as it is received.
vide more complete information on accuracy. Resolve omissions and apparent clerical errors at once. Obtain
7.3.6 Prepare a list of the test materials, their identifying missing values. If obviously erroneous data is submitted, find
numbers, a brief description of material type (for example, out the cause, if possible, and help the laboratory eliminate the
low-carbon steel), and approximate concentration of the ele- problem. Encourage the laboratory to submit a replacement set
ments to be determined. This table becomes part of the of data, if circumstances permit. (The final decision about
documentation sent to participating laboratories and provides replacing data will be made by the task group later, after the
information needed for the research report and the precision testing is complete.)
and bias statement. 8.1.10 Perform a preliminary statistical analysis. Summa-
7.4 Number of Cooperating Laboratories—Conventional rize the comments from laboratories to explain questionable
wisdom holds that the more laboratories participating in an results. Present this information to the task group.
ILS, the better. Further, the laboratory types included in the 8.1.11 As approved by the task group, prepare the final
study task group should consist of typical users’ laboratories. statistical evaluation and the research report. Obtain the task
There is wide agreement that estimates of precision based upon group’s approval for the completed study.
fewer than six laboratories become increasingly unreliable as 8.1.12 Modify the scope of the method, if necessary, and
the number decreases. A test program involving fewer than six prepare the precision and bias statement. Submit the completed
laboratories does not comply with the requirements of this method to the technical subcommittee chairman for editorial
practice (Note 2). An effort should be made to enlist at least review, followed by subcommittee ballot.
seven qualified laboratories before beginning a test program, to 8.2 Task Group—The task group usually consists of one
allow for attrition. To be qualified to participate, a laboratory representative from each participating laboratory. The labora-
must have proper equipment and personnel with sufficient tory representative’s name, address, and telephone and FAX
training and experience to enable them to perform the method numbers should be given to the task group chairman when a
exactly as it is written. laboratory agrees to participate.

3
 E 1601 – 98 (2003)e1
8.2.1 The laboratory representative shall be fully cognizant specified number of (not necessarily sequential) days. Several
of the laboratory’s capabilities and be in a position to ensure conditions must be explicitly spelled out in the protocol, as
the following: follows:
8.2.1.1 The laboratory is capable of performing the method 8.3.2.1 For methods in which samples are dissolved, pre-
properly, pare a single test solution each day. For solid specimens,
8.2.1.2 Appropriate personnel are assigned to perform the prepare them each day in the manner specified by the method.
work and the method is followed exactly as written, 8.3.2.2 Each day the method must be performed in its
8.2.1.3 Test materials are handled properly, entirety, including instrument setup, preparation of the calibra-
8.2.1.4 The test protocol is complied with in all details, tion solutions and calibration, and other steps necessary for
8.2.1.5 The results are recorded accurately on the report each day’s work in accordance with the method. If the method
form, and includes standardization, it must be performed before each
8.2.1.6 The laboratory adheres to the program time sched- day’s work whether or not need for it is indicated.
ule.
8.2.2 As a member of the task group, the laboratory repre- 8.3.2.3 Determine the duplicate results on a single test
sentative must be familiar enough with the analytical tech- solution. For solid samples, determine the duplicate results
niques used in the method to be able to understand the with as little disturbance of the specimen as the method
significance of the test statistics and render considered judg- permits.
ment on how well the method’s performance meets the original 8.3.3 The test protocol includes other details of the way the
analytical requirements. laboratory is expected to conduct its part of the ILS (see Note
8.3 Test Protocol—Preparation of the test protocol is the 5).
responsibility of the coordinator. The protocol gives instruc-
NOTE 5—The following is an illustrative rather than exhaustive ex-
tions to the participating laboratories such as the following: ample: (1) Specify the number of significant digits with which results are
8.3.1 Test Pattern—Practice E 691 requires estimates of the to be recorded for each concentration level. (2) Show how to fill out the
performance of a method under two extreme conditions of report forms. (3) Emphasize the importance of keeping written observa-
variability, minimum variability, and variability among differ- tions that might reveal the cause of unexpected results. (4) Emphasize the
ent laboratories. Minimum variability requires that replicate necessity for immediate communication with the coordinator when a
results be obtained with as little elapsed time as possible. For problem is encountered. (5) Ask for information that might prove useful in
the task group’s evaluation of the test data, such as a description of test
a material of proven homogeneity, specify Test Plan A: three or
equipment.
more sequential replicate results on one portion of the material
(Note 3). Direct each laboratory to analyze test materials in 8.4 Report Forms—Provide official report forms to each
random order, but to complete readings for the replicate results laboratory. Data forms should be convenient to fill out and
(number specified in the protocol) on one test material before simple to use when transcribing the data for statistical analysis.
proceeding to another. For a test material of unknown homo- Provide spaces for the laboratory to identify itself and the date
geneity, specify Test Plan B (Note 4): sequential duplicate the test was performed.
results on at least three portions of the material. Direct each
laboratory to obtain the readings for duplicate results on one 9. Evaluating Data
test portion, followed by the specified number of other portions
9.1 The task group must ensure that data are handled
of the same material before proceeding to another material.
properly both in the laboratory and during statistical analysis.
Give explicit instructions to the analyst for each test material,
Laboratory representatives should be cautioned against submit-
especially if the study uses Test Plan A for some materials and
ting “selected” data. For example, a laboratory might be
Test Plan B for others.
tempted to take extra readings and submit only those that agree
NOTE 3—In some methods, the test portion is completely consumed in well with each other. Such practices or other deviations from
obtaining one result. In these cases, select the sequential test portions to the test protocol must not be tolerated because they destroy the
minimize variation in composition, if possible. Any variation that does integrity of the test design and make correct interpretation of
occur will increase the method’s minimum standard deviation.
the test results impossible. No result may be rejected just
NOTE 4—Test Plan B is effective only when duplicate results can be
taken on a relatively homogeneous test portion. Ideal methods for this
because it does not look good or exceeds a statistical rejection
approach are those in which replicate test portions can be put into solution limit, but only for assignable cause. Assignable cause is
and duplicate results obtained on each solution. If determinations are evidence that the method was not performed as written or that
made directly on solid specimens, Test Plan B should be attempted only standard laboratory practice was not followed. This may
if each laboratory can be provided with at least three portions of the test involve human error or equipment malfunction, or both. In this
material and there is reason to expect that duplicate results on each portion event, the laboratory should correct the problem and, if
will show less variability than results obtained from different portions.
possible, rerun the test or the portion of the test affected by it.
8.3.2 A third test pattern may be used if the task group However, laboratory personnel must not make changes in the
wishes to measure the within-laboratory standard deviation, sr, method. Problems that are perceived as stemming from the
and calculate the repeatability index, r. Obtain sequential method must be discussed with the coordinator. Any unautho-
duplicate results on a test material of proven homogeneity on rized deviation from the written method, no matter how trivial
each of at least three days. Direct each laboratory to obtain it may seem to the analyst, may render the laboratory’s results
duplicate results on one test portion of a material on the unusable.

4
 E 1601 – 98 (2003)e1
9.2 When test data are received from a laboratory, the ment, or failure to follow generally accepted procedures or
coordinator immediately reviews it for consistency and adher- specific instructions of the method. The task group must use
ence to the test protocol. principles of chemistry and physics as well as its analytical
9.2.1 The coordinator discusses questionable values with experience to show that flagged data are inconsistent with
the laboratory representative and clarifies the reasons for rerun reasonable interpretation and execution of the instructions
data (if any). He transfers the original data to test material provided in the method and test protocol. Failing that, the task
tables, marking any values that were questioned or warranted a group must retain the data.
rerun and recording substitute values (if any) as footnotes. The 10.2 The equations are arranged for manual calculation of
reasons for proposed deletions or substitutions are docu- the statistics, but the coordinator is encouraged to use a
mented, observations on the method reported by the laborato- computer version to save time and avoid errors. A separate
ries are summarized, and a preliminary statistical evaluation to statistical analysis is performed for each test material.
flag inconsistent data by the h and k statistics is performed. The 10.3 The data for an ILS run according to Test Plan A are
coordinator questions laboratories that submitted flagged data shown in Table 1. Each column represents a test material with
to see if assignable causes can be found. each laboratory’s replicate results in rows.
9.3 When all data have been received and the tables and 10.4 Test Plan A Calculations—The results of the statistical
comments have been assembled, the coordinator presents this calculations on the data in Table 1 are displayed in Table 2. (In
information to the task group at a meeting. The task group must these equations, x represents the replicate results reported by a
decide whether or not the evidence supplied by the contributing laboratory, n equals the number of replicate results per labo-
laboratory supports rejecting questionable data. When rerun ratory, and p equals the number of laboratories which provided
data are presented, it should also consider whether or not the the data used for this material.)
integrity of the test is jeopardized by substitution of the rerun 10.4.1 For each laboratory, calculate the mean, standard
data for the rejected data. If a misunderstanding of the method deviation, and the square of the standard deviation:
contributed to a problem, the task group may wish to edit the x̄ 5 ~ (~x!/n
language of the method (Note 6) to ensure that it will not s= =(~x 2 x̄!2/~n 2 1!; and s2
continue to trouble future users.
NOTE 6—An editorial change to a method, proposed after testing is 10.4.2 Calculate the overall mean result for the material:
completed, must be examined carefully to ensure that it does not make or 5
x 5 ~ (x̄!/p
imply a change in the technical substance of the method nor that such a
change can be inferred from the edited wording.
9.4 The coordinator performs a final statistical analysis TABLE 1 Nickel ILS Data (% Nickel)
using the data authorized by the task group in the previous step
Laboratory Test Materials
and prepares the research report and the precision and bias
Number A B C D E
section of the method. If the method meets the original project
requirements, the task group authorizes its chairman to submit 1 0.0053 0.053 0.122 0.217 1.08
0.0053 0.052 0.120 0.215 1.07
the method to the technical subcommittee chairman for final 0.0054 0.053 0.120 0.215 1.07
editorial review and subcommittee ballot. If the task group 2 0.0057 0.052 0.124 0.207 1.07
decides that the method does not meet the requirements, it 0.0077 0.054 0.124 0.204 1.06
0.0059 0.053 0.119 0.195 1.05
should examine the test data (with the help of a person who is 3 0.0060 0.053 0.120 0.221 1.08
both adept at using statistics and experienced in analytical 0.0057 0.055 0.113 0.213 1.05
chemistry) in order to change the method to improve its 0.0060 0.053 0.119 0.220 1.07
4 0.0058 0.057 0.121 0.219 1.06
performance. Proposed changes to the method should be tested 0.0053 0.056 0.123 0.225 1.08
by a small group of laboratories before attempting a full-scale 0.0065 0.058 0.130 0.230 1.14
retest. Because such changes affect the technical substance of 5 0.0058 0.054 0.125 0.220 1.06
0.0050 0.054 0.123 0.220 1.06
the method, the revised method must undergo another ILS. 0.0057 0.053 0.126 0.219 1.08
6 0.0060 0.054 0.120 0.215 1.05
0.0059 0.054 0.115 0.215 1.05
10. Calculation 0.0060 0.054 0.120 0.210 1.05
10.1 The ILS test program measures the variability of the 7 0.0055 0.056 0.120 0.221 1.05
0.0060 0.057 0.125 0.221 1.07
test method in typical laboratories. The between-laboratory 0.0050 0.057 0.125 0.215 1.05
standard deviation, sR, and reproducibility index, R, are calcu- 8 0.0069 0.058 0.118 0.218 1.07
lated for this purpose. If the calculated values of these statistics 0.0069 0.058 0.121 0.216 1.06
0.0063 0.057 0.118 0.217 1.08
are to reflect the expected future performance of the method, 9 0.0066 0.056 0.117 0.213 1.10
the test data should not contain extraneous results. The h and k 0.0060 0.057 0.130 0.220 1.05
statistics are provided to aid the task group in its search for 0.0062 0.054 0.123 0.225 1.05
10 0.0058 0.055 0.122 0.221 1.08
extraneous data, but the task group is cautioned that statistics 0.0056 0.053 0.124 0.223 1.06
alone cannot provide sufficient cause for excluding data. For 0.0055 0.055 0.120 0.220 1.08
the relatively small data set produced in a typical ILS using this 11 0.0049 0.055 0.127 0.220 1.03
0.0043 0.057 0.132 0.216 1.06
practice, a result is truly extraneous only if it is caused by 0.0053 0.054 0.125 0.214 1.05
errors in chemical manipulations, improper operation of equip-

5
 E 1601 – 98 (2003)e1
TABLE 2 Statistical Calculations for Nickel Material E (NBS 82a, 1.07 % Nickel)
Laboratory Test Results, x
x̄ s d s2 d2 h k
Number 1 2 3
1 1.08 1.07 1.07 1.0733 0.0058 0.0076 0.00003329 0.00005746 0.59 0.32
2 1.07 1.06 1.05 1.0600 0.0100 −0.0058 0.00010000 0.00003318 −0.45 0.55
3 1.08 1.05 1.07 1.0667 0.0153 0.0009 0.00023348 0.00000083 0.07 0.84
4 1.06 1.08 1.14 1.0933 0.0416 −0.0276 0.00173306 0.00076066 2.16 2.28
5 1.06 1.06 1.08 1.0667 0.0116 0.0009 0.00013340 0.00000083 0.07 0.63
6 1.05 1.05 1.05 1.0500 0.0000 −0.0158 0.00000000 0.00024838 −1.24 0.00
7 1.05 1.07 1.05 1.0567 0.0116 −0.0091 0.00013340 0.00008263 −0.71 0.63
8 1.07 1.06 1.08 1.0700 0.0100 0.0042 0.00010000 0.00001798 0.33 0.55
9 1.10 1.05 1.05 1.0667 0.0289 0.0009 0.00083348 0.00000083 0.07 1.58
10 1.08 1.06 1.08 1.0733 0.0116 0.0076 0.00013340 0.00005625 0.59 0.63
11 1.03 1.06 1.05 1.0467 0.0153 −0.0191 0.00023348 0.00036443 −1.50 0.84
x = 1.0658 ((s2) = 0.00366699

n = 3, p = 11 ((d 2) = 0.00162346
sx̄ = =0.00162346/10 = 0.01274;sM = =0.00366699/11 = 0.01826;
st = =0.000162346 1 ~0.000333363!~2/3! = 0.01961; sR = 0.01961;
R = (2.8)(0.01961) = 0.0594; Rrel = (100)(0.0594)/1.0658 = 5.15 %.
ILS Statistics Summary:
Material Mean Concentration: x = 1.066
Minimum Standard Deviation of the Method: sM = 0.0183
Reproducibility Standard Deviation: sR = 0.0196
Reproducibility Index: R = 0.0549; Rrel = 5.15 %

TABLE 3 Iron Material 1A Data, ppm Iron

10.4.3 For each laboratory, calculate its laboratory differ-
Test Results
ence and the square of the difference: Laboratory
Replicate
Replicate
DA D2
Number x1 x2 Mean, X
d 5 x̄ 2 5
x; and d2 1 1 348 345 346.5 3 9
10.4.4 Calculate the standard deviation of laboratory differ- 2 343 339 341.0 4 16
3 332 327 329.5 5 25
ences: 2 1 347 356 351.5 9 81
2 333 340 336.5 7 49
sx̄ 5 =(~d !/~p 2 1!
2
3 363 357 360.0 6 36
3 1 325 317 321.0 8 64
2 313 310 311.5 3 9
10.4.5 Calculate the method’s minimum standard deviation: 3 330 320 325.0 10 100
4 1 326 322 324.0 4 16
sM 5 =(~s2!/p 2 322 329 325.5 7 49
3 325 337 331.0 12 144
5 1 338 336 337.0 2 4
10.4.6 Calculate a trial value for the reproducibility stan- 2 335 331 333.0 4 16
dard deviation: 3 325 343 334.0 18 324
6 1 339 335 337.0 4 16
st 5 =~sx̄! 2 1 @~sM! 2 ~n 2 1!/n# 2 333 335 334.0 2 4
3 338 340 339.0 2 4
7 1 356 346 351.0 10 100
2 336 331 333.5 5 25
10.4.7 Select the final value for the reproducibility standard 3 343 346 344.5 3 9
deviation: n = 3, p = 7 ((D2) = 1100
sM = =1100/~2!~3!~7! = 5.118
sR 5 the larger of st or sM
10.4.8 Calculate the reproducibility index and percent rela- A
A The difference between duplicate test results is D.
tive reproducibility index:
R 5 2.8~sR!; and Rrel 5 100R/5
x It is arranged like Table 1, except that space is provided for
10.4.9 For each laboratory, calculate its between-laboratory duplicate results on each replicate portion analyzed by a
consistency statistic: laboratory. Other test materials in the iron method test are not
shown. The results of the statistical calculations start in the last
h 5 d/sx̄ two columns of Table 3 and continue in Table 4. For a test
including data for day-to-day within-laboratory variability
10.4.10 For each laboratory, calculate its within-laboratory (replicates analyzed in duplicate on different days in the same
consistency statistic: laboratory), proceed in accordance with 10.6. For a test
k 5 s/sM including data for material variability (replicates are separate
portions analyzed on the one day), proceed in accordance with
10.5 Test Plan B Calculations—Data for a single material 10.7.
obtained in accordance with Test Plan B is shown in Table 3. NOTE 7—In the following equations, x1 and x2 represent the duplicate

6
 E 1601 – 98 (2003)e1
TABLE 4 Statistical Calculations for Iron Material 1A
Laboratory Replicate Means, X Laboratory
s d s2 d2 h k
Number 1 2 3 mean, x̄

1 364.5 341.0 329.5 339.00 8.675 3.476 75.255625 12.082576 0.35 1.20
2 351.5 336.5 360.0 349.33 11.899 13.810 141.586201 190.716100 1.38 1.64
3 321.0 311.5 325.0 319.17 6.934 −16.357 48.080356 267.551449 −1.63 0.96
4 324.0 325.5 331.0 326.83 3.686 −8.690 13.586596 75.516100 −0.87 0.51
5 337.0 333.0 334.0 334.67 2.082 −0.857 4.334724 0.734449 −0.09 0.29
6 337.0 334.0 339.0 336.67 2.517 1.143 6.335289 1.306449 0.11 0.35
7 351.0 335.5 344.5 343.00 8.846 7.476 78.251716 55.890576 0.75 1.22
n = 3, p = 7 x̄|Av = 335.5238 ((s2) = 367.430507 ((d2) = 603.797699

sM2 = 26.190476 (from Table 3); sX2 = 367.430507/7 = 52.490072; sx̄2 = 603.797699/6 = 100.632950;
sM = 5.118; Proceed to either (1) or (2) (but not both), depending on the provisions of the test protocol:
(1) Statistics for Day-to-Day ILS:

1
sr2 5 sX2 1 2 sM2 5 52.490072 1 26.190476/2 5 65.58531
sr = 8.098
n21 1
sR 2 5 sx 2 1 n sX 2 1 2 sM 2

2 1
5 100.632950 1 52.490072 1 26.190476
3 2
= 148.721569
SR = 12.195
r = 2.8 3 8.098 = 22.67; R = 2.8 3 12.195 = 34.15
Rrel = 100 3 34.15/335.52 = 10.18 %
(2) Statistics for ILS to Eliminate Material Variability Effect:

1 1
sH2 5 sX2 2 2 sM2 5 52.490071 2 2 26.190476 5 39.394834

1 1
st2 5 sx̄2 2 n sX2 1 2 sM2

1 1
5 100.632950 2 52.490072 1 26.190476 5 96.231497
3 2
sR 5 =st 5 9.810; R 5 2.8 3 9.810 5 27.47
2

Rrel = 100 3 27.47/335.52 = 8.19 %

sM 2 1 2sH 2
FH 5 5 ~26.190476 1 2 3 39.394834!/26.190476
sM 2
= 4.01, with
f1 = 2 3 7 = 14 and f2 = 3 3 7 = 21 degrees of freedom

results from one replicate in one laboratory, X represents their mean, n 5

x 5 (x̄/p
equals the number of replicates per laboratory, and p equals the number of
laboratories providing data used in the calculations for one material.
10.6 Test Plan B—Day-to-Day Variability (see Note 7)— 10.6.5 For each laboratory, calculate its laboratory differ-
The replicates are portions of the test material that are analyzed ence and the square of the difference:
in duplicate on each of several days in each laboratory (see d 5 x̄ 2 5
x; and d2
8.3.2).
10.6.1 For each test portion, calculate the mean of the
duplicate results, their difference, and the square of the 10.6.6 Calculate the pooled standard deviation of the repli-
difference: cate means and its square:
X 5 ~x1 1 x2!/2 sx 5 =(s 2/p; and sx 2
D 5 x1 2 x2; and D2
10.6.2 Calculate the method’s minimum standard deviation: 10.6.7 Calculate the standard deviation of the laboratory
means and its square:
sM 5 =(D 2/2pn
sx̄ 5 =(d 2/~p 2 1!; and sx̄ 2

10.6.3 For each laboratory, calculate the laboratory mean,

the standard deviation of the replicate means, and the square of 10.6.8 Calculate the repeatability standard deviation:
the standard deviation:
x̄ 5 (X/n st1 5 Œ 1
sX 2 1 2 sM 2
s= =(~X 2 x̄!2/~n 2 1!; and s2

sr 5 the larger of st1 or sM

10.6.4 Calculate the overall mean result for the material:

7
 E 1601 – 98 (2003)e1
10.6.9 Calculate the reproducibility standard deviation:
S 1
D
If sX2 2 2 sM2 is negative or zero,
st2 5 Œ Ssx̄ 2
1
n21
n D sx 2 1
1 2 sM 2 set sH2 5 0; otherwise,
1
sH2 5 sX2 2 2 sM2
sR 5 the larger of st2 or sr
10.7.9 Calculate the reproducibility standard deviation:

Œ
10.6.10 Calculate the repeatability index, the reproducibility 1
index and percent relative reproducibility index: st3 5 sx̄ 2 2 n sX 2 1 sM 2

r 5 2.8~sr!; R 5 2.8~sR!; and Rrel 5 100R/5

x
sR 5 the larger of st3 or sM

10.6.11 For each laboratory, calculate its between-

laboratory consistency statistic: 10.7.10 Calculate the reproducibility index and percent
relative reproducibility index:
h 5 d/sx̄
R 5 2.8~sR!; Rrel 5 100R/5
x 5

10.6.12 For each laboratory, calculate its within-laboratory

consistency statistic: 10.7.11 For each laboratory, calculate its between-
laboratory consistency statistic:
k 5 s/sX
h 5 d/sx̄

10.7 Test Plan B—Material Variability (see Note 7)—

10.7.12 For each laboratory, calculate its within-laboratory
Separate replicate portions of a test material are analyzed in
consistency statistic:
duplicate on one day in each laboratory (see 8.3.1)
k 5 s/sX
10.7.1 For each replicate, calculate the mean of the dupli-
cate results, their difference, and the square of the difference:
10.7.13 Optional (see Note 8)—Calculate the material ho-
X 5 ~x1 1 x2!/2
mogeneity F-statistic and its numerator and denominator
D 5 x1 2 x2; and D2 degrees of freedom:
10.7.2 Calculate the method’s minimum standard deviation: FH 5 ~sM2 1 2sH2!/sM2

sM 5 =(D 2/2np f1 5 p~n 2 1!

f2 5 pn
10.7.3 For each laboratory, calculate the laboratory mean, NOTE 8—Those adept at statistics may wish to calculate the homoge-
the standard deviation of the replicate means, and the square of neity F-statistic to test the hypothesis that the test material is homoge-
the standard deviation: neous.
x̄ 5 ~ (X/n 11. Using Statistics in Task Group Decisions
s= =(~X 2 x̄!2/~n 2 1!; and s2
11.1 Preliminary Screening of Test Data for Consistency—
Most outright mistakes (of the types where equipment fails
10.7.4 Calculate the overall mean result for the material: during the test, a wrong reagent is used, or a test solution is
5 spilled) are caught immediately in the laboratory and are
x 5 (x̄/p
corrected before the test data are submitted. In the same
category are misunderstandings about the calculations, tran-
10.7.5 For each laboratory, calculate its laboratory differ- scription errors, and so forth, which often produce such gross
ence and the square of the difference: distortion of the data that the coordinator can see them at a
glance and ask for immediate clarification from the laboratory.
d 5 x̄ 2 5
x; and d2
Other errors may produce more subtle changes. The pattern of
10.7.6 Calculate the pooled standard deviation of the repli- the affected results may not be obvious within the random
cate means and its square: variation of the rest of the test data. The h and k statistics help
the task group locate such data in its search for assignable
sx 5 =(s 2/p; and sx 2
causes.
11.2 h and k Tables—Place the h and k statistics in tables,
10.7.7 Calculate the standard deviation of the laboratory arranged by test material (columns) and laboratory (rows) as in
means and its square: Tables 5 and 6. Some trends are more easily recognized if the
sx̄=(d 2/~p 2 1!; and sx̄ 2
materials are arranged by increasing concentration from the
first to last column. Consult Table 7 to find the critical value
10.7.8 Calculate the variance of the material homogeneity (CV) for each statistic: CV depends upon the number of
effect: laboratories actually contributing data to the statistics in the

8
 E 1601 – 98 (2003)e1
TABLE 5 Nickel—h StatisticA TABLE 7 Critical Values of h and k at the
0.5 % Significance Level
NOTE—Between-laboratory consistency statistic.
Critical Critical Values of k
Laboratory Test Material Number of Replicates, n
Value pA
Number A B C D E of h 2 3 4 5 6 7 8 9 10
1 −0.90 −1.31 −0.47 −0.22 0.59 1.15 3 1.72 1.67 1.61 1.56 1.52 1.49 1.47 1.44 1.42
2 1.17 −1.11 0.06 x−2.58xA,B −0.45 1.49 4 1.95 1.82 1.73 1.66 1.60 1.56 1.53 1.50 1.47
3 0.17 −0.72 −1.53 0.18 0.07 1.74 5 2.11 1.92 1.79 1.71 1.65 1.60 1.56 1.53 1.50
4 0.10 1.25 0.80 1.33 2.16A 1.92 6 2.22 1.98 1.84 1.75 1.68 1.63 1.59 1.55 1.52
5 −0.59 −0.72 0.80 0.47 0.07 2.05 7 2.30 2.03 1.87 1.77 1.70 1.65 1.60 1.57 1.54
6 0.29 −0.52 −1.21 −0.63 −1.24 2.15 8 2.36 2.06 1.90 1.79 1.72 1.66 1.62 1.58 1.55
7 −0.59 1.05 0.37 0.35 −0.71 2.23 9 2.41 2.09 1.92 1.81 1.73 1.67 1.62 1.59 1.56
8 1.67 1.64 −1.00 0.01 0.33 2.29 10 2.45 2.11 1.93 1.82 1.74 1.68 1.63 1.59 1.56
9 0.85 0.46 0.37 0.41 0.07 2.34 11 2.49 2.13 1.94 1.83 1.75 1.69 1.64 1.60 1.57
10 −0.34 −0.32 −0.05 0.75 0.59 2.38 12 2.51 2.14 1.96 1.84 1.76 1.69 1.64 1.60 1.57
11 −1.84 0.27 1.85 −0.05 −1.50 2.41 13 2.54 2.15 1.96 1.84 1.76 1.70 1.65 1.61 1.58
CV 62.34 62.34 62.34 62.34 62.34 2.44 14 2.56 2.16 1.97 1.85 1.77 1.70 1.65 1.61 1.58
A
Underlined values exceed approximately 87 % of CV. 2.47 15 2.57 2.17 1.98 1.86 1.77 1.71 1.66 1.62 1.58
B
Values flagged with x___x exceed CV. 2.49 16 2.59 2.18 1.98 1.86 1.77 1.71 1.66 1.62 1.58
2.51 17 2.60 2.19 1.99 1.86 1.78 1.71 1.66 1.62 1.59
2.53 18 2.61 2.20 1.99 1.87 1.78 1.72 1.66 1.62 1.59
TABLE 6 Nickel—k StatisticA 2.54 19 2.62 2.20 2.00 1.87 1.78 1.72 1.67 1.62 1.59
2.56 20 2.63 2.21 2.00 1.87 1.79 1.72 1.67 1.63 1.59
NOTE—Within-laboratory consistency statistic. 2.57 21 2.64 2.21 2.00 1.88 1.79 1.72 1.67 1.63 1.59
Test Material 2.58 22 2.65 2.21 2.01 1.88 1.79 1.72 1.67 1.63 1.59
Laboratory 2.59 23 2.66 2.22 2.01 1.88 1.79 1.72 1.67 1.63 1.59
Number A B C D E 2.60 24 2.66 2.22 2.01 1.88 1.79 1.73 1.67 1.63 1.60
1 0.12 0.59 0.34 0.30 0.32 2.61 25 2.67 2.23 2.01 1.88 1.79 1.73 1.67 1.63 1.60
2 x2.29xAB 1.02 0.85 1.64 0.55 2.62 26 2.67 2.23 2.02 1.89 1.80 1.73 1.68 1.63 1.60
3 0.36 1.17 1.11 1.15 0.84 2.62 27 2.68 2.23 2.02 1.89 1.80 1.73 1.68 1.63 1.60
4 1.25 1.02 1.39 1.45 x2.28xA,B 2.63 28 2.68 2.23 2.02 1.89 1.80 1.73 1.68 1.63 1.60
5 0.91 0.59 0.45 0.15 0.63 2.64 29 2.69 2.24 2.02 1.89 1.80 1.73 1.68 1.64 1.60
6 0.12 0 0.85 0.76 0 2.64 30 2.69 2.24 2.02 1.89 1.80 1.73 1.68 1.64 1.60
7 1.04 0.59 0.85 0.91 0.63 A
p = number of laboratories.
8 0.72 0.59 0.51 0.26 0.55
9 0.64 1.55 1.91 1.58 1.58
10 0.32 1.17 0.59 0.40 0.63
11 1.05 1.55 1.06 0.80 0.84 11.2.1.3 A laboratory’s h-values exhibit a preponderance of
CV 2.13 2.13 2.13 2.13 2.13 one sign at low concentrations but the opposite at high
A
Underlined values exceed approximately 87 % of CV. concentrations, or a consistent trend to larger or smaller values
B
Values flagged with x___x exceed CV. as the concentration increases. That laboratory may have a
problem with the slope of its calibration curve.
11.2.2 k Statistic—The k-statistic is a measure of the vari-
column. CV for k also depends upon the number of replicates ability of a laboratory’s replicate results compared with the
reported by each laboratory. Label the line following the last common variability of all other laboratories for a given
laboratory, “CV,” and enter the appropriate value at the bottom material. If all laboratories have similar variability, the k-values
of each column. In Table 5, eleven laboratories provided data are randomly small and large, but none should exceed CV. The
for each material. CV for h, found in Table 7 on the line for p task group should investigate if any of these conditions exist:
= 11, is 2.34. In Table 6, the eleven laboratories each reported 11.2.2.1 An individual k-value is flagged as larger than CV.
three results. CV for k, found in Table 7 on the line for p = 11 One or more of the replicate results reported by that laboratory
and in the column for n = 3, is 2.13. Mark for subsequent on that material may have been incorrectly transcribed or
investigation each column entry that equals or exceeds the CV perhaps were influenced by a condition in the laboratory
of that column. environment that did not affect the other results.
11.2.1 h Statistic—The h statistic is a measure of how close 11.2.2.2 A laboratory has several k-values flagged, espe-
the laboratory’s mean is to the grand mean of all laboratories cially if others approach CV. Some condition of that laborato-
for a given material. If the laboratory’s mean is higher, h is ry’s environment (which includes instruments and personnel)
positive; if it is lower, h is negative. Each laboratory should may not have been as well controlled as in other laboratories.
have approximately equal numbers of positive and negative 11.2.2.3 A laboratory exhibits only unusually small
values, and none should be larger in absolute value than the k-values, especially if many are zero. The laboratory may have
CV. The task group should investigate if any of the these an instrument that is insensitive in its response, an insensitive
conditions exist: range of readings may have been used, or the analyst may have
11.2.1.1 An individual h-value is flagged as larger than CV. rounded readings to produce results with artificially small
Something may have happened to affect the mean result for variability.
that material in that laboratory. 11.3 Interpretation of Statistical Values—When the consis-
11.2.1.2 A laboratory’s h-values are the same sign for most tency statistics exceed their critical values, it merely suggests
materials. That laboratory may have a problem that caused a that a problem might exist. The task group, with the help of the
bias. It is of more concern if one or more materials exceed CV. appropriate laboratory personnel, has the responsibility of

9
 E 1601 – 98 (2003)e1
determining if a specific problem was likely to have occurred 11.3.4 Tables 8 and 9 display h and k statistics for the
and, if it did, whether to replace the defective data (if substitute revised data. The task group accepted the revised data after a
values can be obtained), discard it, or retain it. Tables 5 and 6 discussion on how to obtain more typical results in future ILS
display the h and k statistics for the nickel data shown in Table programs (Note 9). Although the h and k statistics still suggest
1. These data were collected long ago and it is now impossible some laboratory bias or calibration slope effects, the task group
to follow up on the questionable results. For purposes of this could find no reason to believe that the laboratories had failed
discussion, we are assuming a scenario to illustrate how a task to use accepted laboratory practices or had failed to carry out
group might handle them. the method as written (with the exceptions already addressed).
11.3.1 The coordinator noted the following items in Tables The revised test data were used to calculate the test statistics
5 and 6: summarized in Table 10. The task group considered the
Item 1—Material D, Laboratory 2: h = −2.58 exceeds reproducibility standard deviation and index at each concen-
CV. tration level. While these statistics did not quite achieve the
Item 2—Material E, Laboratory 4: h = 2.16 nearly precision hoped for at the inception of the program, the task
exceeds CV and k = 2.28 exceeds CV. group felt that the test method would meet the practical needs
Item 3—Material A, Laboratory 2: k = 2.29 exceeds CV. of the industry and approved the test method as ready for
Item 4—Material C, Laboratory 9: k = 1.91 nearly subcommittee ballot.
exceeds CV. NOTE 9—Prospective coordinators will recognize that a discussion
Item 5—Laboratories 1, 5, 6, and 8 all had a preponder- designed to improve an ILS will be most effective if it precedes the
ance of small k-values. laboratory testing phase. For an ILS, the reported results must include
11.3.2 The coordinator contacted representatives of Labora- what the analyst might consider “extra” digits. These are the part of the
result that provides the variability information and must be reported.
tories 2, 4, and 9 to determine if causes could be found for each While participants should be urged to take care to obtain reliable values,
suspected problem. The information was evaluated and pre- they must be discouraged from deliberately gathering data in a more
sented as a report to the task group: precise or accurate way during an interlaboratory test than they would use
11.3.2.1 Laboratory 2 found that the second reading on Test in normal activities. The prudent ILS coordinator, in his protocol and in
Material A was actually 0.0057 rather than the 0.0077 reported his pretest discussions with the task group, will stress these points.
(miscopied from the notebook). The analyst performing the test 11.4 Plan B Test—For an ILS conducted in accordance with
had noticed that the Test Material D solution had “bumped a one of the Test Plan B protocols, the data and statistical
bit” on the hot plate, but, because he believed the coverglass calculations follow the patterns and equations of the example
had retained the sample, the results were reported without shown in Tables 3 and 4. The task group makes a decision
comment. before the laboratory phase of the ILS begins to select Test
11.3.2.2 Laboratories 4 and 9 could find no reason to Plan B to test either day-to-day repeatability or, if the test
question the data they submitted. When asked about the method is amenable to this option (Note 4), to test reproduc-
apparently high value of 1.14 reported on Test Material E, the ibility free from the effects of suspected test material inhomo-
representative from Laboratory 4 said that it was not unusual to geneity. This practice will not allow a task group to estimate
find one such disagreement among so many replicates. The both kinds of statistics in a single ILS. Although the same
analyst from Laboratory 9 noticed no problems during the test, pattern of results is obtained from both test protocols, a
believing that 0.117, 0.130, and 0.123 represented reasonable repeatability-oriented ILS gives meaningless statistics if ana-
agreement for Test Material C. lyzed in accordance with the equations for eliminating hetero-
11.3.3 The following actions were recommended to the task geneity effects, while data obtained for the purpose of elimi-
group: nating the adverse effects of inhomogeneity will not correctly
11.3.3.1 Eliminate the data for Test Material D from Labo- estimate repeatability. (The data set in Table 3 is analyzed both
ratory 2. The analyst had not followed good analytical practice
by losing the sample. Laboratory 2 was unable to provide a TABLE 8 Nickel—h Statistic
replacement data set.
11.3.3.2 Retain the data for Test Material E from Laboratory NOTE—Between-laboratory consistency statistic.
4 because no cause could be found for the high result of 1.14. Laboratory Test Material
The coordinator agreed with the laboratory representative that Number A B C D E
the result could have been caused by random variation. 1 −0.85 −1.31 −0.47 −0.89 0.59
11.3.3.3 Substitute the correct value 0.0057 for the errone- 2 0.03A −1.11 0.06 ...A −0.45
3 0.30 −0.72 −1.53 −0.15 0.07
ous value 0.0077 for Test Material A from Laboratory 2. 4 0.23 1.25 0.80 1.97 2.16B
11.3.3.4 Retain the data for Test Material C from Laboratory 5 −0.51 −0.72 0.80 0.38 0.07
9 because the agreement did appear to be reasonable in the 6 0.44 −0.52 −1.21 −1.63 −1.24
7 −0.51 1.05 0.37 0.17 −0.71
absence of an observed problem. 8 1.93 1.64 −1.00 −0.47 0.33
11.3.3.5 The data reported by some laboratories seemed to 9 1.05 0.46 0.37 0.28 0.07
10 −0.24 −0.32 −0.05 0.91 0.59
be unusually precise. The task group had the option of 11 −1.87 0.27 1.85 −0.57 −1.50
rerunning the entire test, but the coordinator recommended CV 62.34 62.34 62.34 62.29 62.34
accepting the results because the reproducibility was not likely A
Underlined values exceed approximately 87 % of CV.
to be affected. B
Data revised or deleted.

10
 E 1601 – 98 (2003)e1
TABLE 9 Nickel—k Statistic 12. Preparation of Research Report, Precision and Bias
NOTE—Within-laboratory consistency statistic. Statement, and Adjustment of the Method’s Scope
Test Material Limits
Laboratory
Number A B C D E 12.1 Research Report—The research report provides a per-
1 0.17 0.59 0.34 0.33 0.32 manent record of the data of the task group that is kept on file
2 0.33A 1.02 0.85 ...A 0.55 at ASTM Headquarters for future reference. The following
3 0.50 1.17 1.11 1.26 0.84
4 1.72 1.02 1.39 1.59 x2.28xB,C should be considered the minimum contents of a research
5 1.25 0.59 0.45 0.17 0.63 report:
6 0.17 0 0.85 0.83 0
7 1.43 0.59 0.85 1.00 0.63 12.1.1 The full title of the method;
8 0.99 0.59 0.51 0.29 0.55
9 0.87 1.55 1.91B 1.74 1.58
12.1.2 The names and affiliations of the ILS coordinator and
10 0.44 1.17 0.59 0.44 0.63 the representatives of the participating laboratories;
11 1.44 1.55 1.06 0.88 0.84
CV 2.13 2.13 2.13 3.11 2.13
12.1.3 The test materials, their identification code and
A
Data revised or deleted.
material type, source from which obtained, and the critical
B
Underlined values exceed approximately 87 % of CV. concentration values (if an accepted reference material).
C
Values flagged with x___ x exceed CV.
12.1.4 The test pattern (from the test protocol), that is, how
laboratories handled each portion of the test materials to obtain
TABLE 10 Nickel—Statistical Summary
the results reported in the data tables.
Number
Test
of Rrel, 12.1.5 The table of test data as reported by the participating
Mate- Mean, x sM sR R
rial
Labo- % laboratories. Include in the body of the table any substituted or
ratories
corrected values. Use ellipses for rejected data. Footnote each
A 11 0.00575 0.000349 0.000567 0.0016 27.6 such entry with a brief description of the action taken and the
B 11 0.0549 0.000985 0.00188 0.0053 9.6
C 11 0.122 0.00341 0.00421 0.0118 9.6 reason for the action.
D 10 0.219 0.00347 0.00423 0.0118 5.4 12.1.6 Include a table of the ILS test statistics to be used in
E 11 1.066 0.0183 0.0196 0.0549 5.2
the test method’s Precision and Bias section. For example, the
statistics shown in Table 10 may suffice. Include other method
ways in order to emphasize the difference between the calcu- parameters, such as the upper and lower concentration limits
lations appropriate for each experimental design.) for the test method scope. If calculations are made in accor-
11.4.1 Interpretation of Day-to-Day Statistics—For this dance with a standard practice, it is only necessary to identify
type of ILS, the protocol specifies duplicate results from a test which practice was followed. If other statistical relationships
material on each of three or more days in each laboratory. If the are used, these should be explained in detail.
method specifies a calibration each time the method is used, a 12.1.7 Include the research report when the test method is
complete calibration shall be performed each day. If the submitted to the technical subcommittee chairman for editorial
method specifies standardization, it must be performed each review and subcommittee ballot.
day. If the method specifies standardization, it must be per-
12.2 Precision and Bias—For methods that use this prac-
formed each day without exception before the test results are
tice, the mandatory Precision and Bias section will contain the
obtained. Under these conditions, a Plan B Test protocol will
information shown in the example in Table 11. Other informa-
produce data that is likely to include the most important
sources of within-laboratory day-to-day variability. This test tion may be included as appropriate.
design estimates a repeatability standard deviation (day-to-day 12.2.1 Precision—Use the following format:
within-laboratory) as well as the minimum standard deviation
of the method and reproducibility standard deviation obtained
in the Test Plan A design. Follow the interpretative procedures TABLE 11 Statistical Information—Nickel
outlined in 11.1-11.3. Number of
Min Reproduc- Reproduc-
11.4.2 Interpretation of Statistics to Exclude Material Test Nickel SD ibility SD ibility
Labora- Rrel, %
Material Found,% (sM, (sR, Index (R,
Variability—For this type of ILS, the protocol specifies that tories
E 1601) E 1601) E 1601)
duplicate results be obtained from each of three or more A 11 0.0058 0.00035 0.00057 0.0016 27.6
replicate portions of a test material in an uninterrupted analyti- B 11 0.0549 0.00098 0.00188 0.0053 9.6
cal session in each laboratory. The task group should expect C 11 0.122 0.0034 0.0042 0.012 9.6
D 10 0.219 0.0035 0.0042 0.012 5.4
that variability between duplicates will be less than variation E 11 1.066 0.0183 0.0196 0.055 5.2
between replicate material portions; for example, if the dupli- Certified
cates are aliquots from a test sample solution, they will exhibit Number Source Description
Nickel, %
nearly perfect homogeneity in comparison with separate solu- A 0.005 SRM 10g NBS carbon steel
tions prepared from replicate sample portions. Follow the B 0.056 SRM 152a NBS carbon steel
interpretive procedures outlined in 11.1-11.3. The homogeneity C 0.120 SRM 7g NBS cast iron, high phosphorus
D 0.217 SRM 106b NBS Nitralloy G
effect statistics, sH and FH, relate only to the test material, not E 1.07 SRM 82a NBS cast iron
the method, and need not concern the task group.

11
 E 1601 – 98 (2003)e1
Experience has shown, for the 95 % confidence level at which
Eleven laboratories cooperated in testing this method and obtained the preci- R is calculated, that a value of 50 % for emax yields results
sion information summarized in Table 11. Supporting data have been filed at
ASTM Headquarters. Request RR:E01-XXXX [where XXXX is the Research
useful for determining residual levels of trace elements (Note
Report number assigned by ASTM for this set of data]. 11). For such methods, the calculation reduces to L = 2R. (For
the nickel example, L = 2(0.0016) = 0.003 %.)
12.2.2 Bias: NOTE 10—It is important that at least one of the test materials in the ILS
12.2.2.1 If certified reference materials have been tested, be near or below the lowest concentration level sought. At these low
use this format: concentrations there is no generally valid relationship for extrapolating
standard deviations to lower concentrations, so this practice takes the
The accuracy of this method has been deemed satisfactory based upon the conservative approach of calculating the lower scope limit from the
bias data in Table 11. Users are encouraged to use these or similar reference standard deviation of the lowest test material(s). If the concentration of the
materials to verity that the method, is performing accurately in their laboratories. lowest test material is considerably higher than the level of interest, the
calculated lower limit will probably be higher than it would be if estimated
12.2.2.2 If certified reference materials have not been tested, from a test material of optimum concentration. Although unfortunate, this
error is preferable to claiming an unsubstantiated extrapolated value.
use this format:
NOTE 11—Under no circumstance may emax be larger than 50 %. Use
No information on the accuracy of this method is known, because at the time it smaller values of emax for applications requiring greater precision.
was tested, no accepted reference materials were available. Users are encour-
aged to employ suitable reference materials, if available, to verify the accuracy 12.3.2 Upper Limit (U)—The upper limit is the concentra-
of the method in their laboratories.
tion in a material above which use of the method is not
12.3 Method’s Scope Limits: recommended. Set the upper limit to a value that the task group
12.3.1 Lower Limit (L)—The lower limit is the concentra- believes is warranted by the ILS test results. A reasonable
tion in a material below which a method may not be used to extrapolation above the highest test material concentration is
report quantitative values. If the method is to be used near the sometimes permissible, although the task group is cautioned
lower end of its effective concentration range, calculate L: not to extend method limits to concentrations with which no
L 5 100R/emax one has actual experience.
12.3.3 In the scope of the test method, set the lower end of
where: the method’s concentration range to any desired value equal to
R = reproducibility index of the lowest test material, or greater than L. Set the upper end of the method’s concen-
and tration range equal to or less than U.
emax = maximum acceptable percent relative error (Note
10). 13. Keywords
13.1 bias; interlaboratory test; precision; statistics

ANNEXES

(Mandatory Information)

A1. BACKGROUND INFORMATION

A1.1 The statistical basis for this practice can be found in they suspect are less homogeneous than the majority of other
Practice E 691. materials in the study. Both versions are represented in Table
A1.1.
A1.2 Test Plan A—This basic ILS design assumes (in
addition to the other assumptions common to all analysis of A1.3.1 Repeatability—In the chemical analysis laboratory,
variance) that the test material is homogeneous in composition the term “repeatability” has traditionally been associated with
or, if the composition does vary, that it is satisfactory to include very long-term variability within a laboratory. A good approxi-
that variability in the estimate of the error SD (method’s mation of this long term test can be obtained if the participating
minimum SD). Test Plan A follows the test protocol and laboratories perform duplicate determinations under conditions
statistical analysis recommended in Practice E 691. The user of of minimum variability on three or more days, repeating each
this practice should look there for the theoretical justification of day all aspects of the method most affecting the precision and
the basic aspects of this practice. accuracy of the results. Consequently, this type of ILS is quite
expensive, requiring nearly three times the effort in each
A1.3 Test Plan B—Task groups developing methods of laboratory as a Plan A experiment. The repeatability index, r,
chemical analysis have encountered two situations not covered predicts the range between two results obtained on the same
by Test Plan A. They may wish to estimate a standard deviation material on any two days in the same laboratory. The highest
relating to results obtained in the same laboratory on separate level represents the variability due to the p laboratories, the
occasions, or they may need to include in the ILS test materials next lower level the variability due to the n replicates (days)

12
 E 1601 – 98 (2003)e1
within each laboratory, and the lowest level is the variability of tions on the replicates under conditions of minimum variabil-
duplicate results (variance of the minimum standard deviation, ity. The test is performed only one day in each laboratory and
sM,) nested within laboratories and days. The repeatability the additional work of the extra determination per replicate is
standard deviation, sr, is the square root of the sum of the minimal. Task groups may find this alternative ILS test design
replicate and error variances, while the reproducibility standard useful and not costly. Each laboratory reports duplicate results
deviation, SR, is the square root of the sum of the variances of from each of at least three replicates under conditions of
all three sources. minimum variability (for example, from aliquot portions of
A1.3.2 Imperfectly Homogeneous Test Materials—If the dissolved replicate samples). The highest level represents the
task group conducting the ILS is not assured of the homoge- variability due to the p laboratories, the next lower level
neity of a test material and does not want to include that represents the variability due to the n replicates within labora-
material’s variability in the method’s statistics, the third tories, and the lowest level is the variability of duplicate results
variable of Test Plan B may be used for the material homoge- (variance of the minimum standard deviation, sM,) nested
neity effect and the statistical calculations modified to elimi- within labortories and replicates. The reproducibility standard
nate that source from their estimate of the method’s reproduc- deviation, sR, is calculated as the square root of the sum of the
ibility. This kind of experiment is possible only for methods in laboratory and error variances (omitting the contribution of the
which each laboratory can perform the duplicate determina- material’s inhomogeneity).

A2. STATISTICAL THEORY

A2.1 Model—As with Test Plan A, Test Plan B provides The corresponding pooled variances are as follows:
data that may be analyzed in accordance with a completely
sD2 5 (D2/2pn
randomized model. Level 1 corresponds to the effect of
laboratories, an effect that sums to zero over all laboratories sX2 5 (d12/p~n 2 1!
and exhibits a variability measured by slab. Level 2 corre- sx̄2 5 (d22/~p 2 1!
sponds to the effect of replication within each laboratory, an
effect that sums to zero over all test portions and exhibits a A2.3.1 The variances of the three effects from Table A1.1:
variability measured by srepl. Level 3 corresponds to the se2 5 (D2/2pn 5 sD2
residual error for results produced by the method. Error is
1 1
assumed to be randomly distributed over all laboratories and srepl2 5 2 ~2(d12/p~n 2 1! 2 se2! 5 sX2 2 2 sD2
test portions, sums to zero, and is measured by se. In this
practice, the minimum error of the method is estimated from 1 1
slab2 5 2n ~2n(d22/~p 2 1! 2 2srepl2 2 se2! 5 sx̄2 2 2 sX
duplicate results on each replicate.
A2.3.2 The minimum standard deviation of the method is
A2.2 ANOVA Table—Practice E 691 does not follow the
traditional calculation scheme. This practice follows the same sM 5 =sD2.
approach used in Practice E 691. Table A1.1 displays the A2.3.3 For ILS conducted in accordance with 8.3.2 to
analysis of variance relationships for the Plan B design. The measure the day-to-day within-laboratory variability (repeat-
derivation of the calculations used in Section 10 of the practice ability standard deviation,) the required standard deviations
is based upon Table A1.1. are:
A2.3 Derivations—The standard deviations are obtained by
setting the expected mean squares (EMS) equal to the corre-
sr 5 =srepl 2 1 se 2 5 Π1
sX 2 2 2 sD 2 1 sD 2 5 Π1
sX 2 1 2 SD 2

sponding experimental mean squares (MS). Type B experi-

ments generate three kinds of differences used to measure the
variability contribution of each of the three levels included in
sR 5 =slab 2 1 srepl 2 1 se 2 5 Π1 1
sx̄ 2 2 n sX 2 2 2 sD 2 1 sD 2

the experiment: differences between duplicate results on a test

portion: D = (x1 + x2); differences between a replicate average
and the laboratory’s average: d1 = (X − x̄); and differences 5 Œ sx̄ 2 1
n21 2 1 2
n sX 1 2 s D
between a laboratory average and the average of all laborato-
A2.3.4 For ILS conducted in accordance with 8.3.1 for
ries: d2 = ( x̄ − =x).
materials of unknown homogeneity to eliminate the effects of
material inhomogeneity, the homogeneity effect variance and
TABLE A1.1 ANOVA TableA
the standard deviation for the reproducibility are:
Source Definition MS EMS
1
Laboratories SS1 5 2n(~x̄ 2 x̄!2 SS1/(p − 1) se2 + 2srepl2 + 2nslab2 sH 2 5 srepl 2 5 sX 2 2 2 sD 2
Replicates SSr 5 2(~X 2 x̄!2 SSr/p(n − 1) se2 + 2srepl2
Error SSe= ((D)2/2 SSe/pn(2 − 1) se2
A
The variable symbols are defined in Section 10. SS = sum of squares,
MS = mean squares, and EMS = expected mean squares. sR 5 =slab 2 1 se 2 5 Π1
sx̄ 2 2 n sX 2 1 sD 2

13
 E 1601 – 98 (2003)e1
The homogeneity F-ratio is the ratio of the replicate EMS to which follows the F distribution with p(n − 1) and pn
the error EMS: degrees of freedom.
FH 5 ~sM 2 1 2sH 2!/sM 2,

ASTM International takes no position respecting the validity of any patent rights asserted in connection with any item mentioned
in this standard. Users of this standard are expressly advised that determination of the validity of any such patent rights, and the risk
of infringement of such rights, are entirely their own responsibility.

This standard is subject to revision at any time by the responsible technical committee and must be reviewed every five years and
if not revised, either reapproved or withdrawn. Your comments are invited either for revision of this standard or for additional standards
and should be addressed to ASTM International Headquarters. Your comments will receive careful consideration at a meeting of the
responsible technical committee, which you may attend. If you feel that your comments have not received a fair hearing you should
make your views known to the ASTM Committee on Standards, at the address shown below.

This standard is copyrighted by ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959,
United States. Individual reprints (single or multiple copies) of this standard may be obtained by contacting ASTM at the above
address or at 610-832-9585 (phone), 610-832-9555 (fax), or [email protected] (e-mail); or through the ASTM website
(www.astm.org).

14