Guidelines on Assessment and Reporting of Compliance with Specication

(based on measurements and tests in a laboratory)

ILAC-G8:1996

 Copyright ILAC 1996 ILAC publications may not be copied for sale by any individual or body other than ILAC member organisations

ILAC-G8:1996

Guidelines on Assessment and Reporting of Compliance with Specification

(based on measurements and tests in a laboratory)

ILAC-G8:1996

Guidelines on Assessment and Reporting of Compliance with Specification

PREAMBLE These guidelines have been prepared to assist laboratories worldwide on the method for stating test results and assessment of compliance with specification for testing laboratories. In order to satisfy the requirements of EN 45000 and ISO/IEC Guide 25, laboratories shall provide clients with statements of test results, their uncertainty, and the assessment of compliance with specification when requested to do so within the scope of these guidelines. For calculation of uncertainty, other internationally agreed documents are available4. The homogeneity of samples is assumed. The use of validated test methods and calibrated equipment is assumed. All calibrations should, whenever possible, be traceable to national or international standards.

PURPOSE The guidelines give rules for supplier and client concerning the assessment and reporting of compliance or non-compliance for a single unit of product using an agreed test method. Legal requirements supersede any agreement.

AUTHORSHIP These guidelines were prepared by a Working Group of ILAC Committee 3. The Convenor was Erik L Daae of the Netherlands. The guidelines were endorsed for publication by ILAC Resolution No. 17/96.

Page 4

T ABLEOFCONTENTS Guidelines on Assessment and Reporting of Compliance with Specification

ILAC-G8:1996 ILAC-G8:1996

PREAMBLE .............................................................................................. 4 PURPOSE................................................................................................... 4 AUTHORSHIP .......................................................................................... 4 1. METHOD OF STATING TEST RESULTS ............................ 6 1.1 1.2 1.3 2. General Approach ....................................................... 6 Possibilities of Obtaining a Test Result ...................... 6 Special Cases ............................................................... 7

ASSESSMENT OF COMPLIANCE WITH SPECIFICATION ...................................................................... 7 REFERENCES ........................................................................... 8 APPENDIX A ........................................................................... 10

3. 4.

Page 5

ILAC-G8:1996

Guidelines on Assessment and Reporting of Compliance with Specification

1. 1.1 1.1.1

METHOD OF STATING TEST RESULTS General Approach The extent of the information given when reporting the test result and its uncertainty should be related to the requirements of the client, the specification and the intended use of the result. The methods used to calculate the result and its uncertainty should be available either in the report or in the records of the test including: sufficient documentation of the steps and calculations in the data analysis to enable a repeat of the calculation if necessary; all corrections and constants used in the analysis, and their sources; sufficient documentation to show how the uncertainty is calculated.

1.1.2

When reporting the test result and its uncertainty, the use of excessive numbers of digits should be avoided. In most cases the uncertainty need be expressed to no more than two significant figures (although at least one more figure should be used during the stages of estimation and combination of component uncertainties in order to minimise rounding errors). Unless otherwise specified, the test result should be reported together with the expanded uncertainty appropriate to the approximately 95% level of confidence, in the following manner: Measured value Uncertainty of measurement 100.1 (units) 0.1 (units)

1.1.3

1.1.4

When specification describes an interval with an upper and lower limit, the ratio of the uncertainty of measurement to the specified interval shall be reasonably small (i.e. 1:3).

1.2 1.2.1

Possibilities of Obtaining a Test Result Testing to limit values When testing is done by comparing the test result with a limit value rather than measuring a specific value, the estimation of the uncertainty is also mandatory. Depending on the definition of the specification limits, the test result should be reported in the same way as >, > , < or < .

1.2.2

Testing results in specific values When testing is done with a specific value as a result there are two possibilities: only one test sample is available (single test sample) two or more test samples from the same unit of product are available (duplicate, triplicate etc. test samples).

The certainty of the average test result is dependent on the number of samples. When the number of samples increases, the average test result will better approximate the true value. Therefore the number of test samples should be noted in the report. The (average) test result and uncertainty of measurement should be reported as mentioned under section 1.1.3. The calculation methods of the average test result and uncertainty are outside the purpose of these guidelines.

Page 6

Guidelines on Assessment and Reporting of Compliance with Specification

ILAC-G8:1996

1.3 1.3.1

Special Cases In exceptional cases, where a particular factor or factors can influence the results, but where the magnitude can not be either measured or reasonably assessed, the statement will need to include reference to that fact. Any uncertainty that results from the test sample, not being fully representative of the single unit of product, should normally be identified separately in the evaluation of uncertainty. However, there may be insufficient information to enable this to be done, in which case this should be stated in the report. A possible remark could be: The test results in this report relate only to the test sample as analysed and not to the single unit of product from which the test sample was drawn.

1.3.2

2. 2.1

ASSESSMENT OF COMPLIANCE WITH SPECIFICATION These guidelines require that, when a test is carried out to a stated specification and the client or the specification requires a statement of compliance, the report must contain a statement indicating whether the test results show compliance with the specification. There are a number of possible cases where the uncertainty has a bearing on the compliance statement and these are examined below. The simplest case is where the specification clearly states that the test result, extended by the uncertainty at a given level of confidence, shall not fall outside or within a defined specification limit or limits. In these cases (Case 1, 5, 6 and 10 of Appendix A), assessment of (non)compliance would be straightforward. More often, the specification requires a compliance statement in the certificate or report but makes no reference to taking into account the effect of uncertainty on the assessment of compliance. In such cases it may be appropriate for the user to make a judgement of compliance, based on whether the test result is within the specified limits with no account taken of the uncertainty. This is often referred to as shared risk since the end-user takes some of the risk that the product may not meet the specification after being tested with an agreed measurement method. In this case there is an implicit assumption that the uncertainty of the agreed measurement method is acceptable and it is important that it can be evaluated when necessary. National regulations can overrule the shared risk principle and can put the uncertainty risk on one party. An agreement between the client and the laboratory or a code of practice or specification may state that uncertainty can be ignored when judging compliance. Similar considerations as for shared risk (above) apply in such circumstances. In the absence of any criteria, test specifications, clients requirements, or codes of practice, the following approach is recommended: (a) if the specification limits are not breached by the test result, extended by half of the expanded uncertainty interval at a level of confidence of 95%, then compliance with the specification can be stated (Case 1 and 6 of Appendix A); where an upper specification limit is exceeded by the test result even when it is extended downwards by half of the expanded uncertainty interval, then noncompliance with the specification can be stated (Case 5 of Appendix A);

2.2

2.3

2.4

2.5

(b)

Page 7

ILAC-G8:1996

Guidelines on Assessment and Reporting of Compliance with Specification

(c)

if a lower specification limit is breached even when the test result is extended upwards by half of the expanded uncertainty interval, then non-compliance with the specification can be stated (Case 10 of Appendix A); if the measured single value without the possibility of testing more samples from the same unit of product falls sufficiently close to a specification limit, such that half of the expanded uncertainty interval overlaps the limit, it is not possible to confirm compliance or non-compliance at the stated level of confidence. The test result and expanded uncertainty should be reported together with a statement indicating that neither compliance nor non-compliance was demonstrated. A suitable statement to cover these situations (Case 2, 4 , 7 and 9 of Appendix A) would be, for example: The test result is above (below) the specification limit by a margin less than the measurement uncertainty; it is therefore not possible to state compliance/non-compliance based on 95% level of confidence. However, where a confidence level of less than 95% is acceptable, a compliance/non-compliance statement may be possible. If the law requires a decision anyway concerning rejection or approval, case 2 and 7 can be stated as compliance with the specification limit (with a lower than 95% confidence level). In Case 4 and 9 of Appendix A, non-compliance with the specification limit can be stated (with a lower than 95% confidence level). If two or more samples of a single unit of product can be tested, replicate testing is advisable. After estimating the average value for all test results on the same samples and the new uncertainty for this average-value, the same judgement as above described should be made.

(d)

(e)

If the test result is exactly on the specification limit, it is not possible to state compliance or non-compliance at the stated level of confidence. The test result and expanded uncertainty should be reported together with a statement indicating that neither compliance nor non-compliance was demonstrated at the stated level of confidence. A suitable statement to cover these situations (Case 3 and 8 of Appendix A) would be for example: The test result is equal to the specification limit; it is therefore not possible to state either compliance or non-compliance at the stated level of confidence. If the law requires a statement concerning the assessment in the form of compliance or non-compliance regardless of the level of confidence taking into account the provisions of Section 2.3, the statement depends on the definition of the specifications: if the specification limit is defined as < or > and the test result is equal to the specification limit, then non-compliance can be stated. if the specification limit is defined as < or > and the test result is equal to the specification limit, then compliance can be stated.

Page 8

Guidelines on Assessment and Reporting of Compliance with Specification

ILAC-G8:1996

3. 1.

REFERENCES NIS 80 (1994), Guide to the Expression of Uncertainties in Testing, NAMAS, Teddington, UK NF E 02-204, Verification des tolerances des produits, Declaration de conformite, December 1993, AFNOR, Paris, France. ISO/DIS 14253, part 1, Decision rules for proving conformance or non-conformance with specification, 1995, International Organisation for Standardization, Geneva, Switzerland. BIPM, IEC, IFCC, ISO, IUPAC, IUPAP, OIML, Guide to the Expression of Uncertainty in Measurement. International Organisation for Standardization, Geneva, Switzerland, ISBN 92-67-10188-9, First Edition, 1993. ISO 3534 Part 1, Probability and General Statistical Terms - 1993, Statistics - Vocabulary and symbols, International Organisation for Standardization, Geneva, Switzerland. VIM, ISO (1993), International Vocabulary of Basic and General Terms in Metrology, International Organisation for Standardization, Geneva, Switzerland, ISBN 92-67-01075, Second Edition.

2.

3.

4.

5.

6.

Page 9

ILAC-G8:1996

Guidelines on Assessment and Reporting of Compliance with Specification

APPENDIX A Case 1
The measured result is under the upper limit, even when extended upwards by half of the uncertainty interval. The product therefore complies with the specification.

Case 2
The measured result is below the upper limit, but by a margin less than half of the uncertainty interval; it is therefore not possible to state compliance. However, where a confidence level of less than 95% is acceptable, a compliance statement may be possible.

Case 3
The measured result is on the limit itself; it is therefore not possible to state compliance nor non-compliance. However, where a confidence level of less than 95% is acceptable, and the specification limit is defined as <, a compliance statement may be possible. When the specification limit is defined as <, a noncompliance statement may be possible.

Case 4
The measured result is above the upper limit, buy by a margin less than half of the uncertainty interval; it is therefore not possible to state non-compliance. However, where a confidence level of less than 95% is acceptable, a noncompliance statement may be possible.

Case 5 The measured result is beyond the upper limit, even when extended downwards by half of the uncertainty interval. The product therefore does not comply with the specification.

Specified upper limit

t t

Specified lower limit

Case 6
The measured result is above the lower limit, even when extended downwards by the half of the uncertainty interval. The product therefore complies with the specification.

Case 7
The measured result is above the lower limit, but by a margin less than half of the uncertainty interval; it is therefore not possible to state compliance. However, where a confidence level of less than 95% is acceptable, a compliance statement may be possible.

Case 8
The measured result is on the limit itself; it is therefore not possible to sate compliance nor non-compliance. However, where a confidence level of less than 95% is acceptable, and the specification limit is defined as >, a compliance statement may be possible. When the specification limit is defined as >, a noncompliance statement may be possible.

Case 9
The measured result is below the lower limit, but by a margin less than half of the uncertainty interval; it is therefore not possible to state non-compliance. However, where a confidence level of less than 95% is acceptable, a noncompliance statement may be possible.

t
Case 10 The measured result is beyond the lower limit, even when extended upwards by half of the uncertainty interval. The product therefore does not comply with the specification.

Page 10

= measurement result with agreed method

I = uncertainty interval of
agreed method

The International Laboratory Accreditation Cooperation (ILAC) is the principal international forum for the exchange of ideas and information on laboratory accreditation. Established in the late 1970s, ILAC membership has grown rapidly and includes representatives from the worlds major laboratory accreditation systems in Europe, Asia, North America, Australia and the Pacic. Countries that are developing their own laboratory accreditation systems are also welcome to participate and contribute. ILAC operates a series of committees which investigate issues such as the harmonisation of international laboratory accreditation practices, the effectiveness of mutual recognition agreements in facilitating trade and the promotion of the aims and awareness of laboratory accreditation around the world. There are regular meetings of individual ILAC committees as well as a major plenary meeting of all ILAC members. The activities of ILAC affect a diverse range of areas including standardisation, accreditation, certication, testing, calibration, and regulation in both the public and private sectors.

ILAC Publications Currently Available

Information Documents (I Series)
ILAC-I1:1994 ILAC-I2:1994 ILAC-I3:1996 ILAC-I4:1996 Legal Liability in Testing Testing, Quality Assurance, Certication and Accreditation The Role of Testing and Laboratory Accreditation in International Trade Guidance Documents for the Preparation of Laboratory Quality Manuals

Guidance Documents (G Series)

ILAC-G2:1994 ILAC-G3:1994 ILAC-G4:1994 ILAC-G7:1996 ILAC-G8:1996 ILAC-G9:1996 ILAC-G10:1996 ILAC-G11:1998 ILAC-G12:2000 ILAC-G13:2000 ILAC-G14:2000 ILAC-G15:2001 Traceability of Measurement Guidelines for Training Courses for Assessors Guidelines on Scopes of Accreditation Accreditation Requirements and Operating Criteria for Horseracing Laboratories Guidelines on Assessment and Reporting of Compliance with Specication Guidelines for the Selection and Use of Certied Reference Materials Harmonised Procedures for Surveillance & Reassessment of Accredited Laboratories Guidelines on Assessor Qualication and Competence Guidelines for the Requirements for the Competence of Reference Material Producers Guidelines for the Requirements for the Competence of Providers of Prociency Testing Schemes Guidelines for the Use of Accreditation Body Logos and for Claims of Accreditation Status Guidance for Accreditation to ISO/IEC 17025

Secretariat Documents (S Series)

ILAC-S1:2000 ILAC-S2:1998 Guidelines for the Preparation, Layout and Numbering of ILAC Publications Rules

Procedural Documents (P Series)

ILAC-P1:2000 ILAC-P2: 2000 ILAC Mutual Recognition Arrangement (Arrangement): Requirements for Evaluation of Accreditation Bodies ILAC Mutual Recognition Arrangement (Arrangement): Procedures for Evaluation of Regional Cooperation Bodies for Purpose of Recognition