Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.

html

Conformance Testing and Certification Model for

W3C Specifications
W3C Working Draft 2 January 2002
This version:
http://www.w3.org/QA/2002/01/Note-qa-certif-20020102
Latest version:
http://www.w3.org/QA/qa-certif
Previous version:
This is the first draft from the W3C QA Activity.
It it based on a NIST White Paper entitled: Conformance Testing and Certification
Model for Software Specifications, by Lynne Rosenthal, Mark Skall, and Lisa
Carnahan
Editors:
Daniel Dardailler ([email protected])

Copyright �2001 W3C� (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and
software licensing rules apply.

Abstract
The use of conformity assessment as a means by which buyers and sellers can
communicate requirements will increase as information technology systems and
applications grow more complex. Models for conformance testing and certification
programs are necessary to understand principles and issues that are essential for
successful conformity assessment programs. This paper presents one such model by
identifying key roles, activities and products involved in any conformance testing and
certification program. This model has been successfully used by NIST in helping private-
sector organizations establish their certification programs.

Status of this document

This document is a Note, made available by the W3C Quality Assurance Activity (QAA)
for discussion on the QA email discussion list. It is submitted as to stimulate discussion
within the W3C Quality Assurance Interest Group regarding guidance to external
organizations that may wish to establish certification programs.

It may form the basis of a W3C Note and as such may be modified, replaced or obsoleted
by other documents. Publication of this document does not imply endorsement by the
W3C, its membership or its staff. It is inappropriate to use W3C Working Drafts as

1 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

reference material or to cite them as other than "work in progress". Please send
comments on the publicly archived list [email protected].

Table of Content
1. Introduction

2. Conformance Testing and Certification Model

3. Roles

4. Activities

5. Products

6. Examples

7. Conclusion

References

1. Introduction
As the pervasiveness of information technology increases, so does the importance of
ensuring the quality of products (i.e., software and systems). Conformance testing is
defined in ISO/IEC Guide 2, "as any activity concerned with determining directly or
indirectly that relevant requirements are fulfilled". The W3C QA activity's goal is to make
sure that all W3C specifications are covered by adequate tools supporting their
conformance testing, but it is not chartered, nor is the W3C itself, to run certification
program. The QA activity on the other hand is interested in understanding principles and
issues that are essential for the development of successful conformity assessment
programs run externally to W3C. The goal of this document is to describe a model by
identifying key roles, activities and products involved in any conformance testing and
certification program.

Communication between Buyers and Sellers

In the marketplace, conformance testing provides a vehicle for exchanging information

between buyer and seller. It increases a buyer's (and/or user's) confidence in a product
and its ability to meet their needs. It provides an independent, objective method for
evaluating products and not becoming locked-into a single vendor. For sellers (and
developers), conformance testing can help to substantiate claims that a product meets
the given specification.

Conformance testing is a means of measuring whether a product faithfully implements a

specification. The level and formality of the testing are determined by the market - the
requirements of the buyer directly or an organization acting on behalf of a community of
buyers, or by regulation (e.g., safety, health, national security concerns). For example,
some programs may require a very formal testing and certification approach consisting of
independent (i.e., third party), nationally accredited testing laboratories while others may

2 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

be more appropriate for self declaration and demonstration testing.

The sections below describe a generic model for establishing a conformance testing and
certification program. It describes the processes and procedures for establishing,
administering a testing program. While much has changed regarding conformity
assessment given the growth and changes in the software industry, the conformance and
certification model has not. Examples are used to describe how the model is applied to
support the changes in the software industr

2. Conformance Testing and Certification Model

It is well recognized that conformance testing and certification is a way to ensure that
"standard-based" products are implemented. The advantage afforded by testing and
certification are fairly obvious: quality products, competitive markets with more choices,
commodity pricing, and less opportunity to become "locked in" to a particular vendor.
Moreover, a testing and certification program based on well understood and sound
principles will be acceptable and credible to its community of users.

The conformance testing and certification model described herein contains the
fundamental roles, activities, and products that are necessary in administering and
operating a testing and certification program (see Table 1). By adjusting and modifying
the various activities, roles and products, the model can be applied and used in
establishing any testing and certification program. Figure 1 highlights the interactions
between the roles and activities. The model allows for roles, activities and/or products to
be consolidated or further partitioned.

Roles Activities Products

Buyer Require Certification Specification
Implementation Under Test
Seller Test Implementation
(IUT)
Test Laboratory
Recognize Test Method Test Method
(TL)
Certificate Issuer Recognize Tester Test Report
Control Board (CB) Validate Results Certification Program Policy
Anser Programmatic
Testing Laboratory Criteria
Queries
Answer Test Method
Certificate of Conformance
Queries
Resolve Test Method
Disputes
Validate Conformance
Issue Certificate
Table 1: Roles, Activities, and Products

3 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

Figure 1: Interaction Among Roles and Activities

While actual testing and certification can be carried out by various organizations, it is
essential that there be a centralized sponsor or owner of the testing and certification
program. The sponsor has a fundamental interest in ensuring the success of the
program. Typically, the sponsor establishes and maintains the conformance testing and
certification program. It assumes responsibility for insuring that the components of the
program are in place and becomes the centralized source for information about the
program. The sponsor may be composed of one or more organizations. Examples of
sponsors are consortia, trade associations, standards groups, or a government agency.
More often than not, the sponsor of the program is also the Certificate Issuer.

3. Roles
To execute the activities of the model, five roles are defined. In the realization of this
model, some roles may be combined and performed by a single organization or further
distributed among several organizations.

Buyer requires conformance to the Specification.

Seller builds the product with the intent of meeting the conformance requirement of
the purchaser. Products that undergo testing are called Implementation Under Test
(IUT).
Test Laboratory (TL) performs the operational testing of the IUT .
Certificate Issuer (CI), issues a Certificate of Conformance for IUTs that have
successfully completed the testing process.
Control Board (CB), resolves dispute and answers queries on behalf of the CI.

Buyer

The Buyer requires that a product be tested for conformance. The buyer uses the results
of the testing to verify that a seller provides a product that conforms to the specification

4 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

and meet procurement requirements. In general, the buyer is the impetus for sellers to
undergo conformance testing. Specifically, if buyers don't demand that a product be
tested and show evidence of that testing, it is most likely that sellers will not undertake
having their products tested.

Seller

The Seller or developer uses the conformance tests and undergoes testing to
demonstrate that the product adheres to the specification and thus, meets established
conformance requirements. Additionally, developers may use the tests to debug their
products prior to market

Test Laboratory

The Test Laboratory (TL) conducts the conformance testing using the prescribed test
method. The testing is performed on the seller/developer's product. A TL can be an
organization or individual. A TL can be accredited from a formal accreditation
organization such as NIST's National Voluntary Laboratory Accreditation Program
(NVLAP) or recognized by the buyer, seller, and certificate issuer, as qualified to perform
the testing.

Certificate Issuer

The Certificate Issuer (CI) is responsible for issuing certificates for conforming products.
The decision to issue a certificate is based on the testing results and established criteria
for issuing certificates

Control Board

The Control Board (CB) is an impartial body of experts who function on behalf of the CI.
The CB is responsible for resolving queries and disputes related to the testing process.

4. Activities
The activities comprising the model can be categorized into one of four areas:

Recognition of competent testing laboratories,

Testingwith an approved test method,
Testing process,
Resolution of queries and disputes.

Recognition of Competent Testing Laboratory

A Testing Laboratory (TL) is an entity that provides services to measure, examine, test, or
otherwise assess conformance of an implementation with its specification. Within the
buyer/seller model, a TL can be either a first-party, (the seller performs the testing),
second-party (the buyer performs the testing), or third-party (an independent organization
performs the testing) testing organization. All three types of testing are used in the
software industry. Often there will be multiple TLs for a conformance testing and
certification program

5 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

The Certificate Issuer (CI) as well as Sellers and other interested parties, must have
confidence in the competency of the TL. Competence is based on three concepts,

1. the ability to apply the test method correctly,

2. the ability to repeat a given test and generate the same results,
3. the ability to operate the TL in a manner that maintains objectivity and neutrality
(obviously, first and second party testing organizations are not neutral).

The CI defines competence through requirements and criteria. The CI can then apply the
criteria to a TL, determine its level of competency and, if appropriate, recognize the TL as
competent to perform testing. This practical approach to identifying and recognizing
qualified testing organizations is appropriate when costs, time and efforts do not warrant
seeking accreditation from a formal accreditation organization.

If a more formal and rigorous approach is appropriate, there exists many accreditation
bodies exist that are capable of performing this function. The National Voluntary
Laboratory Accreditation Program (NVLAP) is a NIST organization that accredits testing
organizations based on the requirements of ISO Guide 17025 and additional subject-
matter requirements.

The purpose of the recognition criteria or accreditation is to assure that TLs are capable
and competent to meet the needs of the testing and certification program. The basic
activities to make this determination include:

proficiency testing - demonstration of a TL's competency to successfully perform

the conformance testing using the test method,
on-site assessment - visit by a technical expert to determine compliance with the
recognition criteria and ensure the TL is a legally identifiable organization with staff
and resource to discharge their duties,
quality assurance - documentation and practices to ensure technical integrity of
testing and analyses and adherence to quality practices appropriate to the testing
and certification program.

Additional attributes required of a third-party TL include that it:

ensure that its personnel are free from any commercial, financial and other
pressures which might adversely affect the quality of their work,
ensure that the protection of sellers' confidential information and proprietary rights
are protected,
ensure that sellers are served with impartiality and integrity,
maintain a functional record keeping system for each seller testing process,
have the adequate facilities and equipment to fulfill the requirements of a TL.

Testing with an Approved Test Method

For a Certificate of Conformance to be meaningful, all implementations must be tested in

the same manner. Testing reflects the essence of technical requirements of specifications
and measures whether a product faithfully implements the specification. A test method is
a defined technical procedure for performing a test. A test is the technical operation that
consists of the determination of one or more characteristics of a given product, process
or service according to a specified procedure. A test suite is the collection of tests.
Critical to the success of any conformance testing and certification program is an

6 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

appropriate and adequate test method.

An adequate test method is one that provides test results that give enough information for
the CI to be satisfied that conformance can be measured. An adequate test method
meets the requirement of rigor. An appropriate test method is one that, while adequate,
does not place undue requirements on the IUT and is cost justifiable. If the test method is
too expensive to employ then it will not be used. The definition of adequate and
appropriate is left to the CI to determine.

Testing Process

The Testing Process is described in a conformance testing and certification policy and
procedures document. The document identifies the administrative as well as testing
processes.

The testing process initiates with a seller (or anyone desiring to be tested) contracting
with the TL to have an implementation tested for conformance. The seller and TL
negotiate the scope of testing, the cost of testing, and the timeliness of testing. For a
given seller, the TL must not be in a position to benefit nor suffer (beyond the testing
fees) from the resulting pass or failure of the implementation under test (IUT).

Using the approved Test Method, the TL tests the IUT for conformance and reports the
results in a Test Report. The TL forwards the Test Report and an indication of pass/fail to
the CI. If the IUT successfully completes all the tests and meets the criteria for issuing
certificates, the CI issues a Certificate of Conformanceto the seller. Typically, the CI
maintains and makes available to the public, a listing of products that have received
certificates of conformance.

Resolution of Queries and Disputes

Queries and disputes involving the test method, procedures, test results, and program
administration are directed to the Control Board (CB). The purpose of the CB is to resolve
these issues and communicate the decision to all parties involved. The CB acts on behalf
of the CI. A query or dispute can be initiated by a seller, TL or entity (e.g., developer) at
any point in the testing process. Queries and disputes should contain a statement of the
problem, rationale for dispute, and desired resolution. All matters to be resolved by the
CB should be determined by consensus or as determined by documented CB policy and
procedures.

Additional activities that may be under the auspices of the CB include:

maintain liaison with appropriate standards bodies and test laboratories,

participate in the assessment of TL's seeking recognition status,
recommend changes to new versions of the test method or test laboratory
recognition criteria,
serve as technical advisor to the CI and TLs,
maintain the test suite,
control changes to the conformance testing process.

5. Products

7 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

The following products are used in the model:

Certification Program Policy,

Testing Laboroatory Criteria,
Specification,
Implementation Under Test (IUT)
Test Method,
Test Report,
Certificate of Conformance

Certification Program Policy

The Certification Program Policy (CPP) defines the certification system. ISO/IEC Guide 2
defines a certification system as a system having its own rules of procedure and
management for carrying out conformity certifications. The CPP addresses the following:

responsibilities of the CI,

responsibilities of the TLs,
responsibilities of the seller (the IUT owner),
policy and procedures for test laboratory recognition,
policy and procedures for testing process,
policy and procedures for handling queies and disputes,
complete deinition of the certificate of conformance.

Test Laboratory Criteria

Testing Laboratory Criteria serves three purposes. The first purpose is to define the
competence and quality-related requirements that a testing laboratory must possess to
be designated as a recognized testing laboratory. The second purpose is to describe the
manner in which the laboratory will be assessed against the requirements. The third
purpose is to show those who want to use the testing laboratory (e.g., sellers), or those
who want to accept the conformance certificate as evidence of conformance (e.g.,
buyers) the rigor under which the testing laboratory operates

Specification

First and foremost to conformance testing and certification is the specification. This paper
delineates "standards-based" software specification from other types of specification.
This is because not all specifications can be objectively tested for conformance. We
recognize that not all "standards-based" specifications can be objectively tested.
However objective measurement (not necessarily conformance testing per se) is usually
a goal in these specification development efforts

If the specification can not be objectively tested, then a alternate approach to

conformance testing should be used to measure whether a produce faithfully implements
the specification. This is because an accepted test method cannot be developed, thus
repeatability and reproducibility cannot be ensured.

Implementation Under Test

8 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

The implementation under test (IUT) is the object that is being tested for conformance.
For software specifications it is the software that has implemented the specification. For
any certification program, the scope of the IUT must be defined and delineated from the
rest of the supporting software and hardware of the total system (referred to as the
system under test). In many current certification programs the hardware that is used by
the software must also be defined. The software and supporting hardware constitute the
IUT and are listed in both the test report and certificate of conformance.

Test Method

The test method must be adequate and appropriate within the conformance testing and
certification program in which it is used. Beyond these properties, test methods (and thus
the tests) should be objective, have adequate coverage, and correctly implement the
specification. In trying to meet these requirements, those using and applying the test
method should not make the common mistake of allowing the test method to become the
specification. This means that sellers (builders of IUTs) will build the IUT to pass the
conformance tests, rather than building to the specification.

An objective test method allows for test results to be reproducable by the same testing
laboratory and to be repeatable by a different laboratory. Initially some test methods do
not quite achieve a sufficient level of objectivity. However objectivity should be something
that is always strived for in the development and ongoing refinement of a test method.

Test Report

A test report contains the results of the testing effort, along with any additional
information required by the CI. The test report should provide enough information that, if
necessary, the testing effort could be duplicated. The testing report should contain:

a complete description of the IUT,

the name of the testing laboratory,
the signature of a testing laboratory official,
the date that the testing was completed,
the name and version number of the test method (and test suite),
the results of the test method,
an unambiguous statement indicateing pass or fail.

Certificate of Conformance

The certificate of conformance is typically a summation of the test report. Since it is often
used in the procurement process, it includes information most pertinent between the
busyer and the seller.

The certificate includes statements made by the CI. These statements articulate what the
CI is asserting as being conformant. Typically these statements indicate that "this IUT
was tested in this environment, on this day, using this test method: the test results
produced were consistent with expected test results". The certificate also includes the
signature of a CI official.

6. Examples

9 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

ATA Computer Graphics Metafile (CGM) Conformance Testing Program

The Air Transport Association (ATA) CGM Program was originally established and
operated by NIST to support the ATA 2100 Specification, Graphics Exchange (a.k.a. ATA
CGM profile). The testing program is a critical component of the ATA's program to
represent maintenance manuals in digital form and move to completely on-line
maintenance manuals. Testing is done to ensure that the fidelity and quality of the digital
information is sufficient to satisfy the airline companies' safety and quality concerns. The
program is a means whereby a seller of a CGM implementation can formally demonstrate
conformance to the ATA CGM profile.

NIST is currently working with the ATA in its assumption of the testing program. The ATA
CGM Conformance Testing Program will consist of recognized Testing Laboratories to
conduct the testing and a Control Board to handle disputes and serve as an advisor to
the ATA. The ATA will act as the sponsor and administrator of the program. The ATA or
an ATA designate will issue certificates of conformance. The roles, activities, and
products as described in the generic model apply here with little modification. The Control
Board takes on the additional activity of assessing the testing laboratories according to
pre-established criteria. Additionally, the ATA Technical Information Communication
Committee's Graphics Working Group serves as a technical advisor to both the ATA and
the control board.

The test method consists of a NIST developed test suite and test procedures. The test
method has been accepted and used by the community. It is publicly available along with
other program documents.

IEEE POSIX Validation Service

The IEEE established a validation service for the POSIX (Portable Operating System
Interface). The IEEE Validation Service uses accredited POSIX testing laboratories, issue
certification of validated test results, and maintains a register of accredited laboratories
and successfully tested products. The laboratories are accredited by the NVLAP under its
POSIX program

The requirement for testing is buyer driven. Initially, federal agencies in their requests for
procurement (RFP) of POSIX systems required certificates of validation prior to
purchase. However, the benefits of POSIX testing and its acceptance in the industry has
resulted in sellers requesting to be tested as a matter of course, rather than a
procurement requirement.

The test suite was produced in a joint effort between NIST and several computer
vendors. The original testing policy and procedures produced by NIST have been
adopted by the IEEE.

7. Conclusion
This model describing the conformance testing and certification process has been used
many times over in certification programs for standards-based software specifications.
The examples above illustrate just a few of these programs. It will continue to be used as
a communication mechanism between buyers and sellers.

10 de 11 11-07-2019 18:40
Conformance Testing and Certification Model for W3C Specifications https://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html

Test method developers must continue to develop test methods that have adequate
coverage with regard to the specification; are well defined in terms measurement (i.e.,
what does each test case prove); and be adequate and appropriate as defined by the
Certificate Issuer.

As the industry moves toward component based software, the challenge will be to
develop test methods and associated certification programs that can provide meaningful
measurement in this environment

References
Breitenberg, Maureen, The ABC's of the U.S. Conformity Assessment System,
NISTIR 6014, April 1997.
Breitenberg, Maureen, The U.S. Certification System from a Government
Perspective, NISTIR 6077, October, 1997.
Carnahan, Lisa, Developing Federal Standards and Accreditations for Data
Protection Products, Proceeding of SPIE Conference, October, 1995.
Dashiell, William H., L. Arnold Johnson and Lynne S. Rosenthal, Overview of Model
for United States Geological Survey Recognition of Spatial Data Transfer Standard
Certification System, NIST IR 6124, May 1998.
Horlick Jeffrey, and Lisa Carnahan, Cryptographic Module Testing, Handbook
150-17, April, 1995.
ISO/IEC Guide 2: 1996, Standardization and Related Activities: General
Vocabulary.
ISO/IEC Guide 17025: 1999, General Requirements for the Competence of
Calibration and Testing Laboratories.
NIST, Derived Test Requirements for FIPS 140-1, Security Requirements for
Cryptographic Modules, March, 1995.
NIST, Procedures and Requirements, NIST Handbook 150, March 1994.

11 de 11 11-07-2019 18:40