SIMPLIFIED SOFTWARE LIFE-CYCLE DIAGRAM

FEASIBILITY
STUDY

PROJECT TIME
I SOFTWARE P
FUNCTIONAL I
SPECIFICATION!

SOFTWARE
SYSTEM
DESIGN
DETAILED

C MODULES
ECIFICATION

MODULES
DESIGN

SOFTWARE
INTEGRATION
AND
TESTING

SYSTEM
TESTING

••COMMISSIONING I
AND
HANDOVER |

DECOMMISSION

DESIGN
DESIGN SPECIFICATION VERIFICATION OPERATION AND MAINTENANCE

SOFTWARE LIFE-CYCLE PHASES

TECHNICAL REPORTS SERIES No. 2 8 2

Manual on Quality Assurance

for Computer Software
Related to the Safety
of Nuclear Power Plants

f I N T E R N A T I O N A L ATOMIC ENERGY AGENCY, V I E N N A , 1988

MANUAL ON QUALITY ASSURANCE
FOR COMPUTER SOFTWARE
RELATED TO THE SAFETY
OF NUCLEAR POWER PLANTS
 T h e following States are M e m b e r s of the International A t o m i c Energy A g e n c y :

AFGHANISTAN GUATEMALA PARAGUAY

ALBANIA HAITI PERU

ALGERIA HOLY SEE PHILIPPINES

ARGENTINA HUNGARY POLAND

AUSTRALIA ICELAND PORTUGAL

AUSTRIA INDIA QATAR

BANGLADESH INDONESIA ROMANIA

BELGIUM IRAN, ISLAMIC REPUBLIC OF SAUDI ARABIA

BOLIVIA IRAQ SENEGAL

BRAZIL IRELAND SIERRA LEONE

BULGARIA ISRAEL SINGAPORE

BURMA ITALY SOUTH AFRICA

BYELORUSSIAN SOVIET JAMAICA SPAIN

SOCIALIST REPUBLIC JAPAN SRI LANKA
CAMEROON JORDAN SUDAN
CANADA KENYA SWEDEN
CHILE KOREA, REPUBLIC OF SWITZERLAND
CHINA KUWAIT SYRIAN ARAB REPUBLIC
COLOMBIA LEBANON THAILAND
COSTA RICA LIBERIA TUNISIA
COTE D'lVOIRE LIBYAN ARAB JAMAHIRIYA TURKEY
CUBA LIECHTENSTEIN UGANDA
CYPRUS LUXEMBOURG UKRAINIAN SOVIET SOCIALIST
CZECHOSLOVAKIA MADAGASCAR REPUBLIC
DEMOCRATIC KAMPUCHEA MALAYSIA UNION OF SOVIET SOCIALIST
DEMOCRATIC PEOPLE'S MALI REPUBLICS
REPUBLIC OF KOREA MAURITIUS UNITED ARAB EMIRATES
DENMARK MEXICO UNITED KINGDOM OF GREAT
DOMINICAN REPUBLIC MONACO BRITAIN A N D NORTHERN

ECUADOR MONGOLIA IRELAND

EGYPT MOROCCO UNITED REPUBLIC OF

EL SALVADOR NAMIBIA TANZANIA

ETHIOPIA NETHERLANDS UNITED STATES OF AMERICA
FINLAND NEW ZEALAND URUGUAY
FRANCE NICARAGUA VENEZUELA
GABON NIGER VIET NAM
GERMAN DEMOCRATIC REPUBLIC NIGERIA YUGOSLAVIA
GERMANY, FEDERAL REPUBLIC OF NORWAY ZAIRE
GHANA PAKISTAN ZAMBIA
GREECE PANAMA ZIMBABWE

T h e A g e n c y ' s Statute was a p p r o v e d on 23 October 1956 by the C o n f e r e n c e on the Statute of t h e

I A E A held at United Nations H e a d q u a r t e r s , N e w Y o r k ; it e n t e r e d into f o r c e o n 29 July 1957. T h e H e a d -
quarters of the Agency are situated in V i e n n a . Its principal objective is " t o accelerate and enlarge the
contribution of atomic energy to peace, health and prosperity throughout the w o r l d " .

© I A E A , 1988

Permission to reproduce or translate the information contained in this publication m a y be

obtained by writing to the International A t o m i c E n e r g y A g e n c y , W a g r a m e r s t r a s s e 5, P . O . Box 100,
A - 1 4 0 0 Vienna, Austria.

Printed by the I A E A in Austria

April 1988
 TECHNICAL REPORTS SERIES No. 282

MANUAL ON QUALITY ASSURANCE

FOR COMPUTER SOFTWARE
RELATED TO THE SAFETY
OF NUCLEAR POWER PLANTS

INTERNATIONAL ATOMIC ENERGY AGENCY

VIENNA, 1988
MANUAL ON QUALITY ASSURANCE FOR COMPUTER SOFTWARE
RELATED TO THE SAFETY OF NUCLEAR POWER PLANTS
IAEA, VIENNA, 1988
STI/DOC/10/282
ISBN 92-0-155288-2
 FOREWORD

The International Atomic Energy Agency's plans for establishing safety

standards for nuclear power plants, referred to as the NUSS Programme, include the
development of three types of documents:

(1) Codes of Practice for thermal neutron nuclear power plants that establish the
objectives and minimum requirements which must be fulfilled to provide
adequate safety for these plants
(2) Safety Guides that provide additional requirements and recommend procedures
that should be followed to implement the Codes of Practice
(3) User's Manuals, directed primarily to nuclear power plant operators, that
normally present possible methods and techniques for solving specific
problems.

Work on Codes and Guides was initiated in 1975 in five main fields: govern-
ment organization, siting, design, operation and quality assurance.
In the field of quality assurance the Code of Practice and ten Safety Guides had
been developed by 1986 and published in English, French, Spanish and Russian, as
well as some in Chinese. These documents are now used in a number of Member
States as quality assurance requirements for nuclear power plants. To facilitate the
use of these publications, the IAEA Technical Review Committee on Quality
Assurance has stressed on a number of occasions the need for and importance of
proceeding with the development of User's Manuals. These documents should
provide Member States implementing the Code and the Safety Guides with practical
examples of procedures, practices and documents illustrating the quality assurance
methods and techniques used in those organizations in Member States which have
broad experience in quality assurance. The same opinion was expressed in the
discussions during the International Symposium on Quality Assurance for Nuclear
Power Plants organized by the IAEA and held in Paris in May 1981. A number of
topics have been identified for which User's Manuals could provide additional
information and facilitate correct implementation of the Code and Guides in nuclear
power plant project activities.
To implement these recommendations, work has been initiated in the
Secretariat to develop those User's Manuals which are most needed in Member
States embarking on nuclear power programmes and starting quality assurance
activities. Keeping in mind the difference between User's Manuals and Codes and
Safety Guides, work on the development of these documents is undertaken outside
the NUSS Programme and the established procedures for development, review and
approval of the documents used in this Programme. For User's Manuals it was
decided to follow the standard practices used in the development of other Agency
publications such as Guidebooks and Technical Reports. This procedure will reduce
the time and cost of preparation of User's Manuals, which are the lower levels in
the hierarchy of NUSS Programme documents and do not contain requirements for
whose formulation a broad consensus of quality assurance experts would be needed.
The present Manual on Quality Assurance for Computer Software Related to
the Safety of Nuclear Power Plants provides guidance in the assurance of quality of
specification, design, implementation, maintenance and use of computer software
related to items and activities important to safety in nuclear power plants. This
guidance is consistent with, and supplements, the requirements and recommenda-
tions of Quality Assurance for Safety in Nuclear Power Plants: A Code of Practice,
50-C-QA, and related Safety Guides on quality assurance for nuclear power plants.
The Manual is intended to be of use to all those who, in any way, are involved
with software for safety related applications for nuclear power plants, including
auditors who may be called upon to audit management systems and product software.
 CONTENTS

1. INTRODUCTION 1

1.1. Objectives of the Manual 1

1.2. Need for software quality assurance 1
1.3. Applicability of the Manual 1
1.4. Software quality •...; 2
1.5. References 3

2. SOFTWARE LIFE-CYCLE 3

2.1. Requirements specification 7

2.2. Functional specification 7
2.3 Architectural and detailed designs 7
2.4. Coding and software implementation 8
2.5. Integration testing and commissioning 8
2.6. Operation, maintenance and enhancement 8

3. QUALITY ASSURANCE PROGRAMME ..i 9

3.1. General 9
3.2. Scope of the quality assurance programme 9
3.3. Responsibility 9
3.4. Quality assurance programme establishment 10
3.4.1. General 10
3.4.2. Quality assurance activities 10
3.5. Quality assurance programme documentation 10
3.5.1. Software quality plan 10
3.5.2. Work procedures and instructions 13

4. ORGANIZATION 13

4.1. Structure 13
4.2. Interfaces 14
4.2.1. Transfer of responsibility 14
4.3. Duties, authorities and responsibilities 14
4.4. Staffing and training 15
5. SOFTWARE DOCUMENT, CONFIGURATION, MEDIA AND
SERVICES CONTROL 16

5.1. Software control and computing facilities 16

5.1.1. Software document control 16
5.1.2. Computing facilities control 16
5.2. Configuration management 17
5.2.1. General 17
5.2.2. Software identification, labelling, cataloguing 17
5.2.3. Documentation 17
5.2.4. Configuration identification 19
5.2.5. Configuration control 19
5.2.6. Tools, techniques and methods 20
5.2.7. Software library 21
5.3. Media and services control 21
5.3.1. Physical media utilized 21
5.3.2. Backup copy location and maintenance 21
5.3.3. Security 22

6. DESIGN CONTROL 22

6.1. Design specifications 23

6.1.1. Requirements specification 23
6.1.2. Functional specification 23
6.1.3. Detailed design specifications 23
6.2. Design verification 26
6.2.1. Verification 26
6.2.2. Design reviews 26
6.3. Validation 27
6.4. Tools, techniques and methods 27
6.5. Design changes 28

7. PROCUREMENT CONTROL 28

8. TESTING 28

8.1. Test planning 29

8.2. Test performance 29
8.3. Test reviews 30
8.4. Acceptance testing and certification 30
9. NON-CONFORMANCE CONTROL 30

10. CORRECTIVE ACTION 31

11. RECORDS : : 31

12. AUDITING 31

ANNEXES

Annex A: IAEA documents referred to in the Manual 33

Annex B: Glossary 34
Annex C: Software quality attributes 42
Annex D: List of documents 45
Annex E-l: Contents of a typical procedure: Software archiving and
control procedure 46
Annex E-2: Typical procedure: Procedure for the use of computer
programs for safety related design purposes 47
Annex F: Typical responsibilities of the quality assurance function 50
Annex G: Typical responsibilities of project management 51
Annex H: Typical duties, authorities and responsibilities for some
key activities in a project 52
Annex J: Staff training record form 53
Annex K-l: Typical records and verification responsibility table 54
Annex K-2: Typical form for test record log 57
Annex L-l: Form for media control: Register of media 58
Annex L-2: Form for media control: Program location record 59
Annex L-3: Form for media control: Archives record 60
Annex L-4: Form for media control: Record of faulty media 61
Annex L-5: Form for media control: Program modification record 62
Annex M-l: Form for configuration control: Module release 63
Annex M-2: Form for configuration control: Software change request 64
Annex M-3: Form for configuration control: Software change register 65
Annex M-4: Form for configuration control: Configuration status report ... 66
Annex M-5: Form for configuration control: Document register 67
Annex N: Verification and validation activities applied to a typical
software life-cycle 68
Annex P: Possible additional items for configuration control 69
Annex Q: Outline of a typical software verification and validation plan .. 70
Annex R: Verification record form 71
Annex S: Typical design review check-list 72
Annex T-l: Non-conformance report form 79
Annex T-2: Non-conformance report log form 80
Annex U: Validation record form 81
Annex V-l: Typical check-list for audit of computer programs 82
Annex V-2: Typical check-list for configuration management audits 83
Annex V-3: Typical check-list for audit of software testing 85

BIBLIOGRAPHY 87

LIST OF PARTICIPANTS 103

 1; INTRODUCTION

1.1. OBJECTIVES OF THE MANUAL

The objective of the Manual is to provide guidance in the assurance of quality

of specification, design, maintenance and use of computer software related to items
and activities important to safety (hereinafter referred to as safety related) in nuclear
power plants. This guidance is consistent with, and supplements, the requirements
and recommendations of Quality Assurance for Safety in Nuclear Power Plants:
A Code of Practice, 50-C-QA, and related Safety Guides on quality assurance for
nuclear power plants. Annex A identifies the IAEA documents referenced in the
Manual.
The Manual is intended to be of use to all those who, in any way, are involved
with software for safety related applications for nuclear power plants, including
auditors who may be called upon to. audit management systems and product software.

1.2. NEED FOR SOFTWARE QUALITY ASSURANCE

Software is playing an ever increasing role in the design, manufacture and

operation of nuclear power plants, and the potential for error and poor quality is very
real. This potential for error exists with computer software at all stages of interaction
with computers and the software that runs on them.
Assurance is needed that the software to be run on a computer installation is
compatible with the.hardware. The software must be appropriate for the intended
purpose. Where a mathematical model is used in a computer program it must be an
appropriate representation of the physical problem. The algorithms used must main-
tain the required numerical precision and must be free of numerical instabilities.
It has been established that the effort in finding and correcting software errors
and making changes during the operating phase can be many times that incurred at
the design and implementation stages. Also, there may be associated safety and hard-
ware problems resulting from inadequate software. Taking these facts into account,
an orderly planned approach is needed to ensure the high quality software required
for safety related computer applications.

1.3. APPLICABILITY OF THE MANUAL

The Manual is applicable to the following types of software, which may

directly or indirectly influence the safety of the plant:

1
(1) Systems — includes software which provides the following services, enabling
application software to function:

(a) Input and output management

(b) Resource allocation
(c) Mathematical and system libraries
(d) Data management.

(2) Application — includes software which contributes towards:

(a) Operating the plant (for example, software for plant monitoring and
control systems, plant protection systems, process diagnostic and opera-
tor support systems, etc.)
(b) Recording and storing data, and producing reports, data banks and
statistics
(c) Modelling and calculating for design, safety and reliability analyses
(d) Planning maintenance, fault diagnosis, operational use and verification
(e) Training plant operators with computer based simulators.

(3) Support — includes all software used in the development, testing and main-
tenance of the above applications as computer programs.

The Manual covers the development of new software, and the maintenance and
use of existing software.
The terms used in the Manual are explained in the Glossary (see Annex B).

Note; Although the principles contained in the Manual are applicable in all cases,
some of the details may hot be required.

1.4. SOFTWARE QUALITY

Software quality may be defined as the degree to which the software conforms
to the specified requirements and expectations of the prospective user. A list of
attributes which may affect software quality is given in Annex C as an indication of
some of the many characteristics which need to be considered when software quality
is being evaluated during its life-cycle.
Many of these characteristics may lead to conflicting requirements, the solu-
tion of which is not always clear or easy. For example, added efficiency is often
purchased at the price of simplicity, portability, accuracy, understandability and
maintainability; added accuracy often conflicts with portability; conciseness can
conflict with readability. Users of software generally find it difficult to quantify their
preferences in such conflicting situations, and in these cases software suppliers may
produce a code to their standards. Every effort should be made to assess the relative

2
importance of all the attributes when specifying software for a particular application.
This assessment may be achieved more easily through the use of appropriate check-
lists and associated priorities.
To summarize these considerations, the measurement of quality of software
products will vary with the needs and priorities of the prospective user. It is therefore
necessary to specify the precise requirements, including quality, at the onset of a
software project, and to develop the requirements formally in a structured and
controlled manner through the ensuing project phases.

1.5. REFERENCES

In preparing the Manual existing literature was reviewed. The Bibliography

contains a list of some available literature.

2. SOFTWARE LIFE-CYCLE

The whole life, from concept to cessation of use, of the software has been
described as the software life-cycle and Figs 1-3 delineate a systematic approach
to the development, commissioning, use and formal withdrawal from use (decom-
missioning). The software life-cycle diagrams are examples showing the basic stages
through which software grows and develops from the, conceptual stage through to its
operation and withdrawal from use. At each of these stages, appropriate controls
need to be applied to the associated activities in order that the status of the software
is known and controlled and, where necessary, verified and approved. However, for
any particular project the whole of the software life-cycle as shown may not apply,
or it may be necessary to identify further stages.
Figure 1 is a very basic diagram, whereas Figs 2 and 3 are examples showing
life-cycle diagrams, together with related software documentation, and management
control activities. It is seen, therefore, that there are many different ways of describ-
ing the software life-cycle and it is a matter of project preference which is used. In
all cases it is necessary to ensure that the software life-cycle is mutually compatible
with the hardware life-cycle; together, they constitute the combined system life-
cycle. Provision of a detailed life-cycle diagram will help to ensure that software
development progresses in a planned and orderly manner.
Each of the software life-cycle activities should be formally defined and should
have a formal conclusion, usually with a document that provides tangible evidence
that the activity is completed. The following activities of the software life-cycle, as
shown in Fig. 2, have been chosen as examples for the Manual.

3
 i commissioning
ano
handover

use a n d
maintenance

decommission

DESIGN
STUDY DESIGN SPECIFICATION DESIGN VERIFICATION OPERATION AND MAINTENANCE

SOFTWARE LIFE-CYCLE PHASES

FIG. I. Simplified software life-cycle diagram.

 1
• user requirements
REQUIREMENTS
SPECIFICATION validation t e s t
- facilities to be requirements

\ T3
provided

\ FUNCTIONAL

\ SPECIFICATION
subsystems structure
\ programs, d a t a o r g a n i z a t i o n
s y s t e m verification
ARCHITECTURAL t e s t specification
system
\ DESIGN p r o g r a m and
extension,
replacement X d a t a b a s e design

\ DETAILED DESIGN

\\ source programs,
s y s t e m building,
object programs
CODING AND
project management
V IMPLEMENTATION
test results,
-programme of work/progress/resources \ f a u l t / m o d . record
-standards/methods \ INTEGRATION
- d e v e l o p m e n t facilities
- c o n f i g u r a t i o n and document control \ TESTING AND
COMMISSIONING fault reports,
\
r
- u s e r acceptance/formal h a n d o v e r
mod. r e q u e s t s ,

\ OPERATION, release notices

\ MAINTENANCE AND - dispose of

obsolete
\ \
ENHANCEMENT
programs

FIG. 2. Software life-cycle diagram showing project activities.

 GUIDELINES MANAGEMENT
SOFTWARE DEVELOPMENT
PROJECTS

ABBREVIATIONS

COR - CRITICAL DESIGN REVIEW

FCA - FUNCTIONAL CONFIGURAT'ON AUDIT
PCA - PHYSICAL CONFIGURATION AUDIT
GUIDELINES
POR - PRELIMINARY DESIGN DESCRIPTION TECHMCAL REVIEW
SDD - SOFTWARE DESIGN DESCRIPTION • OF SOFTWARE
SOS - SOFTWARE DESIGN SPECIFICAT'ON PRODUCTS
SRR - SOFTWARE R E T I R E M E N T S REVIEW
SRS - SOFTWARE REQUIREMENTS SPECIFICATION
SVR - SOFTWARE VERIFICATION REVIEW
GUIDELINES
VRR - VERIFICATION READINESS REVIEW SOFTWARE
CONFIGURATION
MANAGEMENT

FIG. 3. Software life-cycle diagram showing documents to be produced.

2.1. REQUIREMENTS SPECIFICATION

The requirements that the computer program must satisfy are identified from
the overall system requirements and ultimately documented in the requirements
specification. The requirements specification activity is the most significant part of
the overall project in terms of its influence on the quality of the final product; it
should be as complete, unambiguous and verifiable as possible.
The requirements specification may take many forms. However, it should
contain enough information to totally identify and define the required functions so
that the top level of software design can address all these functions in a traceable
manner. To ensure that the requirements specification is read and understood, its
contents should be simply and concisely presented. It should, however, be
sufficiently detailed to show how all the relevant system requirements are being
satisfied. The requirements specification should follow a recognized standard (see
Section 3(d) of the Bibliography), should deal with what the software is intended
to do, and should also address both quality and general testing requirements.
Compatibility between software and hardware should be addressed at this and
appropriate subsequent phases of the life-cycle.

2.2. FUNCTIONAL SPECIFICATION

The functional specification activity determines the high level preliminary

design for the software, the result of which is to break down the software system
into functional parts and to document the results in the software system design. Each
part should be a cohesive unit that carries out, as independently as possible, the
associated function stated in the requirements specification, and develops the modu-
lar structure of the software. A modular approach enables a large and complex
problem to be divided into a set of smaller, independently testable, less complex
elements. Such a division of the problem into tiers of related modules represents a
major step in the completion of the final design. The degree to which software can
be made modular will have far reaching effects on its ultimate quality. There are
several design methods available to carry out this phase of the operation, references
to which are given in Section 6 of the Bibliography.

2.3. ARCHITECTURAL AND DETAILED DESIGNS

The software design activities specify the software architecture and continue
the breakdown of the functions identified in the requirements into the necessary
detail. The detailed design will include definition of the actual algorithms and equa-
tions as well as the detailed control logic and data operations that are to be

7
performed. They provide the basis for the coding and implementation activity that
will follow and take into consideration all the elements which will make up that
activity. The primary output of this activity is a design specification which is usually
designated the detailed functional specification, consisting of, for example, texts,
program specifications or decision tables. The detailed functional specification and
the software specification can be issued together as the program design document.

2.4. CODING AND SOFTWARE IMPLEMENTATION

During this activity, the detailed software design is translated into an appropri-
ate level programming language. Compilation and assembly errors are corrected,
after which preliminary program check-out begins by executing the individual
program modules to remove the more obvious errors. Although some testing is done
at this point in the life-cycle by the developer, it does not formally include the testing
phase of the software life-cycle. There are a number of guides that have been
prepared for this activity. These include text books as well as formal standards, many
of which are listed in Sections 3(d) and 6 of the Bibliography. The product of this
activity is usually a source program, which is available in computer readable and
computer processable form. A printout of the program listing should be produced.

2.5. INTEGRATION TESTING AND COMMISSIONING

Testing is carried out on modules, integrated subsystems and final software.

It includes final testing, acceptance testing arid commissioning, where required, and
is conducted in accordance with the documented test plans established in the prelimi-
nary and detailed design activities, in compliance with the requirements
specification.
The processes which should be followed durihg the testing phase can be
obtained from a study of the documents listed in Section 8 of the Bibliography. The
test requirements and results are recorded and evaluated, and reports are prepared
to describe the outcome of the tests. Successful completion of the testing means
obtaining acceptable test results, thus leading tb formal acceptance of the software
by the user.

2.6. OPERATION, MAINTENANCE AND ENHANCEMENT

The final activities in the software life-cycle (see Fig. 2) are operation, main-
tenance and enhancement. During these activities the software is being operated
according to user documentation; and further activity consists of applying modifica-
tions to remove latent errors, or is in response to new or revised requirements or

8
to improve its operation (i.e. enhancement). Such enhancements may be due to hard-
ware changes. Any modifications resulting from these activities should be retested
and approved.

3. QUALITY ASSURANCE PROGRAMME

3.1. GENERAL

The general quality assurance programme requirements are to be found in

Establishing the Quality Assurance Programme for a Nuclear Power Plant Project:
A, Safety Guide, 50-SG-QA1.
The purpose of the quality assurance programme during the software life-cycle
is to identify, to all concerned, a basis for the control of all activities affecting
quality, and to ensure that the specified quality is achieved. Performance of these
activities in accordance with the defined and documented procedures and work
instructions for software is an important contribution towards the ultimate success
of the nuclear power plant project.
Implementation of this programme requires that all the individuals involved
understand what is going on, why it is being done, how they will benefit from it,
how the organization will benefit from it, and exactly what is expected of them. In
particular, individuals should be convinced that if a systematic software engineering
approach embodying quality principles is used, it will help rather than hinder their
software development activities.

3.2. SCOPE OF THE QUALITY ASSURANCE PROGRAMME

The quality assurance programme should cover the relevant activities of the
software life-cycle and the associated management controls, e.g. quality planning,
configuration control, organization and training, and should provide auditing of all
the functions. Organizational responsibilities and authorities for the conduct and
approval of activities affecting quality should also be defined. For the full scope of
the programme, reference should be made to Sections 2.1 and 2.2 of Code of
Practice 50-C-QA.

3.3. RESPONSIBILITY

To define the responsibilities for the establishment and implementation of an

effective quality assurance programme, reference should be made to Code of
Practice 50-C-QA and related Safety Guides.

9
3.4. QUALITY ASSURANCE PROGRAMME ESTABLISHMENT

3.4.1. General

To establish the quality assurance programme, reference should be made to

Section 2 of Safety Guide 50-SG-QA1. However, it should be noted that all the
quality assurance criteria should be contained in the quality assurance programme,
each being applied to the degree appropriate to the project; for safety related soft-
ware the highest degree of quality assurance is recommended.

3.4.2. Quality assurance activities

To determine the quality assurance activities, reference should be made to

Section 2.1.3 of Safety Guide 50-SG-QA1.

3.5. QUALITY ASSURANCE PROGRAMME DOCUMENTATION

Reference should be made to Sections 2.1 and 2.2 of Code of Practice

50-C-QA and to Sections 4.2.1 and 4.2.2 of Safety Guide 50-SG-QA1 in the prepara-
tion of quality assurance programme documentation. In addition to the documenta-
tion covered by the above references, documentation of the quality assurance
programme should include the software quality plan and the related work procedures
and instructions. Corporate standards should be generated or adopted for each
project, for example, the preferred languages, coding standards, documentation
formats, etc.

3.5.1. Software quality plan

It is necessary at the outset of the software project to prepare a software quality

plan which should incorporate, as appropriate, a flow chart or sequential narrative
listing of all processes, procedures, work instructions, tests and inspections to be
performed and tools to be used in the development and acceptance of software.
Typical software quality plans are shown in Figs 4 and 5. The software quality plan
should cross-relate to any quality plan for associated hardware, examples of which
are contained in Annexes 1 and 2 of Quality Assurance in the Manufacture of Items
for Nuclear Power Plants: A Safety Guide, 50-SG-QA8.
The software quality plan for any part of the project should be issued prior to
the commencement of any work on that part of the project and should be subject to
review and approval before issue.

10
Note: PEC'S . RCaUmEHENIS
SPEC, * SPECIFICATION
FIG. 4. Simple quality plan showing system hardware interfaces.
FIG. 5. Typical software quality plan showing all modules and documents to be produced.
3.5.2. Work procedures and instructions

Work procedures and instructions are detailed directives on how general or

specific activities are to be performed. They delineate the explicit step by step actions
to be taken in order to ensure that the work performance and recorded results will
comply with the predetermined requirements.
Where an existing standard is to be adopted, the work procedures should
indicate which sections are applicable.
In addition to the procedures and instructions required by the usual require-
ments contained in the referenced IAEA documents (e.g. for collection and main-
tenance of records), procedures and instructions should be written to describe the
contents and format of those project documents listed in Annex D. The procedures
should also indicate who is responsible for the preparation and authorization of the
documents.
Procedures and instructions should also be prepared for all activities which
affect software quality, i.e. as a minimum for: the software design process; design
reviews; software coding; software module verification; software integration and
testing; preparation and content of user documentation; handling and storage of
media; software library functions; auditing and reporting; commissioning and accep-
tance; software operation and maintenance; error reporting and correction; configu-
ration management and reporting; system testing and validation; training; and use
of computing facilities. The contents of a typical procedure are shown in Annex E - l ;
a typical procedure is shown in Annex E-2.

4. ORGANIZATION

4.1. STRUCTURE

The requirements for organizations participating in activities affecting the

quality of nuclear power plants are contained in Code of Practice 50-C-QA. These
requirements include the establishment of a documented organizational structure,
with clearly defined responsibilities, levels of authority and lines of communication.
The design, implementation, maintenance and use of computer software in safety
related applications are activities which could affect the quality and safe operation
of nuclear power plants and as such are subject to these requirements and the recom-
mendations contained in Quality Assurance Organization for Nuclear Power Plants:
A Safety Guide, 50-SG-QA7.
The position of the quality assurance function in relation to the project organi-
zation should be clearly indicated, including lines of reporting to higher management

13
and communication with other organizational units. The typical responsibilities of
the quality assurance function are outlined in Annex F.
Project management ultimately has control of the quality of the software as it
determines the resources the project will use in its development. If the resources are
inadequate, it may be difficult to ensure product quality.
The functions of project management should be clearly defined and the quality
assurance related responsibilities of the project manager outlined (see Annex G).

4.2. INTERFACES

For the control of interfaces, reference should be made to Sections 5 and 6

of Quality Assurance in the Design of Nuclear Power Plants: A Safety Guide,
50-SG-QA6, and to Section 2 of Safety Guide 50-SG-QA7. Care should be taken to
ensure that these controls also encompass interfaces between software and hardware
organizations as well as specialized software houses, etc.

4.2.1. Transfer of responsibility

This is a particularly important interface involving the handover of software

where the responsibility for the maintenance of the software is transferred from the
developer to the user. The transfer usually takes place at the end of system testing
or commissioning. It is necessary to establish that all the requirements which the user
specified have been met and that any exceptions are formally recorded and agreed'
upon by both parties. Once the responsibility has been formally transferred to the
user, his software quality assurance programme is then applied for subsequent phases
of the life-cycle.

4.3. DUTIES, AUTHORITIES AND RESPONSIBILITIES

The duties, authorities and responsibilities of all members of the software

project must be clearly identified and documented by project management for each
phase of the project. It is not unusual, particularly in smaller organizations, for
individuals to assume multiple roles, for example, a software designer may be the
tester for the software produced by another designer, provided this tester has not
participated in the design of the software being tested. This may be acceptable as
long as the respective responsibilities of the designers and verifiers are clearly
defined. The typical duties, authorities and responsibilities for some key activities
in a project are given in Annex H.

14
4.4. STAFFING AND TRAINING

Reference should be made to Section 3.3 of Code of Practice 50-C-QA for

staff selection and training requirements, and to Section 2.5.4 of Safety Guide
50-SG-QA7 for qualification requirements of inspection, testing and audit personnel.
Training of personnel is not usually thought of as a software quality assurance
activity. However, quality of the software product is directly related to the compe-
tence of the individuals developing the product. Therefore, training in appropriate
elements of software engineering and software quality assurance of all personnel
performing and verifying the activities affecting software quality should be given.
Also, there must be a strong commitment to and support of training by upper
management. Appropriate company policy statements should be issued proclaiming
commitment and support for the training.
Classes and seminars should be conducted to train software development,
operations and quality assurance personnel on aspects of the software standards and
software engineering techniques. These training activities should be designed to
increase the competence of the personnel on the project, thereby improving the
quality of the work. There are various methods which can be used for training of
personnel in the concepts of software engineering and quality assurance. The most
prevalent methods are the use of seminars or short courses and recorded seminars
and computer aided instructions.
It is recommended that a detailed training programme be laid out for the soft-
ware developers, verifiers and users and for the quality assurance personnel to help
them obtain the necessary competence.
As with all educational processes, knowledge of software methods should be
maintained fresh in the software developers' and quality assurance personnels' minds
by follow-up training in subject headings that are important.
Records should be kept on each of the individuals who are involved in software
development, software maintenance, software testing and software quality
assurance, of the training they have received and the dates when they received it.
This information will be useful in establishing which individual should be given
future training or retraining. It also identifies those individuals who are competent
to carry out the various functions. Records of the performance of individuals in the
training course may also be useful.
Proper training of staff in the use and maintenance of a software product is
equally vital. It is essential that users are fully familiar with the proper preparation
of inputs and that they are competent in the correct interpretation of the output. They
should be familiar with all the features of a software package so that they can apply
it with maximum effectiveness. The training programme for users should place
particular emphasis on identifying the limitations of the software package in ques-
tion. Training should cover, for example, the applications for which the software has
been designed, the steps required to avoid numerical instabilities, an understanding

15
of the mathematical models used to approximate physical processes, the limits of
applicability of empirical correlations, etc. The recommendations for training of the
developer's staff apply equally to the user's staff.
Annex J shows a typical training record form.

5. SOFTWARE DOCUMENT, CONFIGURATION,

MEDIA AND SERVICES CONTROL

5.1. SOFTWARE CONTROL AND COMPUTING FACILITIES

5.1.1. Software document control

For the control of documents, reference should be made to Section 4 of Code

of Practice 50-C-QA.
Annexes K - l and K-2 indicate the typical responsibilities for preparing and
verifying some documents.
Provisions should be made to ensure that only approved versions of operating
computer codes are distributed and that only these are being used. The system should
have the ability to capture all the information essential for producing any distribution
records and status reports on the software.
When software is accessed from the central library for use in analysis,
monitoring, or any other system, a means should be established for demonstrating
that the software was transmitted correctly. This can be accomplished by a number
of standard practices such as using check sums, parity checking and multiple
transmissions, with ensuing file comparisons. Along with the software, appropriate
test cases should be transmitted which can then be run upon the host computer to
verify that the software is performing correctly.

5.1.2. Computing facilities control

Internal and external computing facilities should be controlled to the degree

necessary to distribute, protect and ensure the validity of the operating software and
the associated documentation. Operating software and associated documentation
should be placed under software configuration management and stored in a central-
ized computer program library. The quality assurance programme should require
that adequate controls and security measures have been established to ensure that
valid results are produced and that the software is protected from any unauthorized
alteration. Rights of access to external facilities should be obtained in order that
controls can be monitored.

16
5.2. CONFIGURATION MANAGEMENT

5.2.1. General

Configuration management is an extension of the established quality assurance

practices referred to in Section 5.1. It is applied to software and its documentation
throughout the whole life-cycle to ensure that the software system, its component
software, e.g. modules stored on magnetic tape or other media, specifications, verifi-
cation evidence, and descriptive documentation are all mutually identified and at a
known status (status is usually defined by allocating issue/revision/version identi-
fiers). Software configuration must be checked against the appropriate hardware
configuration to ensure that the software and hardware used in the integrated system
are mutually compatible.

5.2.2. Software identification, labelling, cataloguing

A project should establish controls and procedures to ensure that all versions
of the computer software are accurately identified. Controls should also be estab-
lished to record the changing of the configuration status of the software. Mechanisms
should be provided in the software library to assign and track the identification of
computer programs and documentation, including their revisions. Proper release
authorization documentation should be provided within the library media. An autho-
rized signature list should be in place for the released documentation. The software
library function should assist with the arrangements for marking, labelling and pack-
ing for shipment of all deliverable material. The software library should maintain
logs and records relating to the distribution, inventory, configuration control and
status accounting for all deliverable items. The typical information to be recorded
is shown in Annexes L - l to L-5.

5.2.3. Documentation

Documentation for operating software should be subjected to software configu-

ration management procedures. Provisions should be made for informing all person-
nel of the latest changes to software documentation. This includes documentation
such as user manuals, design manuals, requirements specifications/or any other
documents, including the software listing or code itself whenever changes are made.
An effective way of implementing the distribution of documentation is to maintain
an on-line system for documentation. The responsibility is placed upon the user to
access the on-line documentation whenever he is using the computer code in order
to determine that he indeed does have the latest documentation. Appropriate notifica-
tion procedures may be implemented on-line to alert users that the code or documen-
tation has been modified.

17
SOFTWARE LIFE-CYCLE PHASES

FIG. 6. Configuration control.

 New version implementation should follow procedures similar to those
mentioned above for documentation. It is beneficial to use appropriate tools in the
documentation of new versions in order to identify wherever the code or the
documentation has been modified. This can be done by bars in the margins, or lists
of pages which have been modified, or a list of lines where modifications have been
made to the software. Superseded versions should not be accessible unless under
controlled conditions.

5.2.4. Configuration identification

All items which are required to be subjected to configuration control should

be identified and should, as a minimum, include those items referenced in Annex D
and those activities similarly identified in Fig. 6. All utility software, e.g. tools and
harnesses, should be subject to configuration control. The purpose of each item,
e.g. master copy, archive, or object code, should be recorded, together with the date
generated, in a description of the item. A baseline which uniquely identifies all the
items should be established, and all changes should be recorded and incorporated
into a new baseline.

5.2.5. Configuration control

A comprehensive configuration control system will include the following

activities:

(1) Establishment of configuration reference points. This can vary from project to
project, but it is essential that item and system configuration should be estab-
lished during the software life-cycle, as shown for example in Fig. 6. After
handover, the configuration should be maintained and it is necessary for a soft-
ware user to define the configuration reference points which are usually estab-
lished upon upgrading of the software or hardware system.
(2) Establishment of configuration baseline. A configuration baseline is a datum
established at a reference point in order that all subsequent changes be recorded
and incorporated into the configuration at the next reference point. Baselines
should be frozen.
(3) Introduction of new items. Before a configured item is made available for use
by someone other than its originator, it should be brought under configuration
control.
(4) Introduction of changes. All changes to configured items should be
documented and controlled, and all related configuration documentation should
be reviewed and amended as appropriate. The nature of the change and effect
on the performance of other configured items and systems should be assessed
and recorded. Any retest requirement should be assessed and recorded. All

19
 changes should be approved by the appropriate authority and communicated to
all those persons involved. Patching of safety related software should be
strongly discouraged.
(5) Establishment of a master configuration index. This index should contain a list
of all the configured items, together with their current configuration status and
identification of all the associated documentation. To provide backward trace-
ability, copies of indexes for previous baselines, together with the changes
applied, should be kept.
(6) Configuration status accounting. This activity consists of recording a detailed
history of each configured item and baseline to enable the dates of configura-
tion and change to be reported, together with the current issue and change
status. Any known deficiencies, e.g. from error reports or configuration
audits, should also be recorded against a configured item.

Samples of configuration control documentation are shown in Annexes M - l

to M-5.

5.2.6. Tools, techniques and methods

The extent to which management of configuration techniques is employed

should be determined by the importance of the software. The methods of approving
software and documentation and the quality requirements for control of individual
configured items in terms of approval, change control and introduction are similar,
to the methods used for conventional document control; they are as follows:

(1) The preparation, review, approval and issue of software and configuration
management reports should be controlled.
(2) The parties responsible for item (1) should be identified'.
(3) A software release system should be established and measures provided to
ensure that users are aware of, and use, appropriate and correct software and
information.
(4) Software changes should be documented, reviewed and approved. Review and
approval should be made by the same authorities as the original item.
(5) All software and documentation revisions should be reported in a timely
manner to all those persons involved.
(6) Configuration records should provide objective evidence that all configuration
items have been identified, that configuration baselines have been established
and that software changes are reviewed, approved, controlled and
implemented. Management operating procedures for all the activities are
necessary, and software tools for configuration management are available
commercially. Further information on tools is given in Section 5.2 of the
Bibliography.

20
5.2.7. Software library

The software library is an important concept in the control of configuration.

Its prime function is to maintain and report on the state of the software and associated
documents, and to control their issue and recall. The library should also contain and
control all configuration records and copies of media. (See Annex N for documents
which should be subject to configuration control and Annex P for possible additional
items, which should be retained under configuration control.) No configuration item
should be admitted to the library until;; it has been brought under configuration
control.
The software library may also contain and control all those international,
national, company and project standards and Codes of Practice for languages,
programming and testing, unless these documents are controlled elsewhere.

5.3. MEDIA AND SERVICES CONTROL

For storage control, reference should be made to Section 5 of the Quality

Assurance Records System: A Safety Guide, 50-SG-QA2.
The physical media used for computer software storage include all means by
which computer data can be stored. Control of the physical media and associated
services are all those functions which are carried out to ensure that the data or soft-
ware so stored are physically retrievable and cannot be lost or compromised due to
any day to day operations or possible catastrophic events.

5.3.1. Physical media utilized

The typical storage media currently.in use are magnetic disks, magnetic tapes,
large scale integrated circuits, punched paper tape, program cards, diskettes, or
computer listings. As technology changes, the media will probably also include such
articles as video cassette tapes, laser disks, compact disks, and any other future
media which will be used in the industry.

5.3.2. Backup copy location and maintenance

It is strongly recommended that at least two backup copies of software should

be stored in relatively remote locations which have a very low probability of being
subjected to the same catastrophic event. One common practice is to implement a
common storage facility for use by many different organizations, with each organiza-
tion having its own additional local facility for storage of.the computer software.
A second operating copy may also be provided to allow ease of access to the
user in case the first copy is degraded. jThe second copy is maintained on the same

21
central machine so that the user can access it readily if it becomes evident that the
primary copy has somehow not given a correct result. Periodic comparisons should
be made between the two copies to ensure that no degradation has taken place, and
magnetic media should be rewritten since magnetic domains on tapes and disks tend
to deteriorate.

5.3.3. Security

A means should be provided for controlling the physical media to ensure that
the stored software is accessible only to those authorized persons having need of
access. Several methods are available which will provide adequate protection from
unauthorized access to computer program media. Section 5.3 of the Bibliography
contains references on standards which have been developed for the physical security
of computer media. This area of knowledge has increased as many computer systems
have been subjected to violation by 'unauthorized' individuals. The primary method
is by effective password control or hardware access protection. Other methods
include limited access program libraries and computer rooms, encryption, external
markings and proprietary statements identifying the controlled programs. Modern
computer operating systems are being designed to include extensive security
considerations, especially when access by means of telephone lines and modems is
permitted.

6. DESIGN CONTROL

Reference should be made to Safety Guide 50-SG-QA6 for the control of

design activities and to Section 4.2 of Safety Guide 50-SG-QA1 for the form and
content of documentation.
Software development is mainly a design process and, as for hardware, a
disciplined professional approach is needed to ensure that the software will meet the
specified requirements.
It is necessary to use well defined design standards and practices, starting with
an organized disciplined approach to design management and leading to a planned
and structured design which can be verified and validated. Annex N clarifies the
relationships between the life-cycle phases and the related verification and validation
activities.
To ensure an organized disciplined approach to the design development
process, particular attention should be paid to the planning of activities. In addition
to a specific software quality plan for each project, a specific manual of work
procedures should be established, including the basic technical, design and program-
ming activities involved, such as documentation naming and coding, programming

22
languages and testing. For example, the following information should be provided:
documentation standards; logic structure standards; coding standards; and com-
mentary standards.

6.1. DESIGN SPECIFICATIONS

t
All software designs require specification of all the elements. The following
sections provide examples of a possible hierarchy of specifications with typical
contents. The division of requirements between the various specifications is not rigid
and may be adjusted as required.

6.1.1. Requirements specification

This specification is produced ideally before a contract is entered into as a basis

for agreement between the client and the supplier and should contain sufficient infor-
mation for a work schedule and cost estimates to be produced to the accuracy
required.
The document should provide the basis for development of the system and
typically should cover the following: functional requirements; performance require-
ments; design requirements; human engineering requirements; interface require-
ments; expansion requirements; special test requirements; general test requirements;
system development environment; subcontracted requirements; special require-
ments; validation requirements; acceptance requirements; maintenance require-
ments; and documentation requirements.

6.1.2. Functional specification

This specification provides sufficient information for software design work to

commence and should typically contain sections on the following: design philosophy
and method, applicable standards and codes, language, standard values and
algorithms, programming methods, standard forms of output, methods of specifying
programs and production of program listings; software architecture; hardware and
software environment; performance criteria; data structures and formats; control
logic; interface requirements; utility software; and functional test requirements.

6.1.3. Detailed design specifications

These specifications detail the design information for each elemental program
in sufficient depth to enable programmers not having a knowledge of the overall
system to produce these programs. The detailed design document should typically
contain the following information: a detailed logic structure; internal data structures;

23
 SPECIAL REQUESTS DEFINED
INTERFACES SPECIFIED
DELIVERABLE SOFTWARE LISTED

- FORMAT AND ADEQUACY OF DOCS

- IDENTIFICATION OF CRITICAL AREAS
- IDENTIFICATION OF SPECIAL RECORDS
TESTING OVERLOAD, SECURITY, ALSO
- TRANSFER OF DOCS TO CONFIGURATION CONTROL
- ESTABLISH NEED FOR TEST RECORD
- GIVE APPROVAL FOR NEXT STAGE

- FORMAT AND ADEQUACY OF DOCUMENTS

/»tdmdiau\ - REQUIREMENT FOR TEST AND SUPPORT SOFTWARE
DESIGN > . ENSURE DOCUMENTATION IS IN CONFIGURATION CONTROL
\ REVIEW _ D | V E A P P R O V A L FOR NEXT STAGE '

CONT.

* TYPICAL ITEMS FOR REVIEW

FIG. 7. Documentation and design review;

24
 FROM

* TYPICAL ITEMS FOR REVIEW

FIG. 7 (cont.)

25
cross-references to the library routines to be used; the routines and conditions
specific to the program; functions; memory allocation requirements; a program iden-
tifier; priority requirements; interface requirements; and performance requirements.

6.2. DESIGN VERIFICATION

6.2.1. Verification

Verification is a planned ongoing activity and provides an independent assess-

ment of the software and its associated documentation throughout the life-cycle. It
establishes the degree of conformance to the specified requirements at each planned
stage.
Verifiers should include representatives from any part of the project which
interfaces with the work being verified and, where appropriate, a hardware represen-
tative to ensure compatibility between the software and hardware. Verification
activities should be performed in accordance with a predetermined verification and
validation plan (see Annex Q) established independently of software development
and should be formally recorded, stating the names of the verifiers and the scope of
their verification, e.g. scope or format of verified documents, technical content and
compatibility of interfaces. Verifiers should provide written comments and evidence
that the originator has considered; these comments should be included in the verifica-
tion record.
Typical verification methods are described in the documents referenced in
Section 6.2 of the Bibliography. See also Annexes R (verification record form), K - l
(typical records and verification responsibility table), Q (outline of a typical verifica-
tion and validation plan) and N (verification and validation activities applied to a
typical software life-cycle).

6.2.2. Design reviews

Design reviews are one form of verification. They make use of the same
principles, irrespective of whether they are performed on software or hardware
systems. A typical design review check-list is shown in Annex S.
In reporting the review, the reviewers are to identify the configuration items
and status. Actions resulting from the review should be agreed upon and recorded
in software non-conformance reports (see Annex T - l ) , deposited in the software
library and recorded in the configuration status reports (see Annex M-4) for manage-
ment monitoring. As a design review should be a project hold point, all actions are
to be cleared before further work proceeds.
Documents subject to design review are shown in Fig. 7, which also shows
how the reviews are phased into the design process.

26
6.3. VALIDATION

Validation of software should be a planned activity and should establish,

independently of the developer, that the final software meets the original intent as
defined in the requirements specification (see Section 2.1). For example, it may be
possible to establish confidence in the performance of stand alone codes by compar-
ing the results from code computation against the results from physical experiments
or against empirical data. Validation may also be possible by comparing the
computed results with less complex models or alternative calculations. In some real
time control and instrumentation systems it is often necessary to use full or partial
plant process simulation. The integrated software/hardware should be exercised
through simulation of input data (signals) present during normal operating condi-
tions, anticipated operational occurrences and accident conditions requiring adequate
system action. This approach basically includes:

(1) Real time testing of the integrated software implemented on a target computer
system under simulated conditions
(2) Testing to demonstrate that the software meets the computer software require-
ments under operational conditions.

Software validation tests might be performed as a part of total computer system

acceptance testing to demonstrate the ability of the system to meet the system
requirements specification (see Annex N, verification and validation activities
applied to a typical software life-cycle).
A record of validation should be kept, indicating the method of validation and
the configuration status of the software used in the validation. The validation record
should be approved before the software is used. The typical documents used in
validation activities are shown in Annexes Q and U and in Section 3.2 of Annex E-2.

6.4. TOOLS, TECHNIQUES AND METHODS

Software tools are computer programs used by the programmer for develop-
ment of software products as aids to productivity and quality. These tools include
design aids, testing tools, program analysers, translators and editors. It is possible
for a programmer to work entirely within a tool system; therefore it is important that
a tool which could directly affect the product quality be subject to the same kind of
controls that apply to product software and that the functions of the tool are known
to have been verified and recorded. Tool documentation should detail information
on what constitutes an acceptable input to the tool, how the information is accepted,
manipulated and analysed, and also what form of output is produced. If uncontrolled
tools are used, additional verifications shall show that the resultant output code is
correct.

27
 For information on tools, techniques and methods, reference should be made
to Section 6.4 of the Bibliography.

6.5. DESIGN CHANGES

For design changes, reference should be made to Section 9 of Safety Guide

50-SG-QA6 and to Section 5.2.3 of this Manual.

7. PROCUREMENT CONTROL

For control of procurement activities, reference should be made to Quality

Assurance in the Procurement of Items and Services for Nuclear Power Plants:
A Safety Guide, 50-SG-QA3.

8. TESTING

For control of testing activities; reference should be made to Section 9 of Code

of Practice 50-CQ-A and to Section 7 of Safety Guide 50-SG-QA8.
The objectives of the testing activities in the software development process are
to expose latent defects and to provide assurance that the software performs to its
technical and operational requirements, as specified in the requirements specification
and the design documentation, e.g. a module against its design specification. The test
activities should therefore be aimed at ensuring that these objectives are achieved in
an orderly, cohesive, clear and controlled fashion and are formally recorded.
Effective test planning should start with the requirements specification and
address the totality of the testing to be performed throughout the applicable phases
of the software life-cycle, including the operation and maintenance phases where
changes can be made to the software.
Part of maintenance involves what is known as regression testing (that function
which is required to determine that the software has not been affected by any
enhancements). Therefore, the results of previous tests that have been carried out
during the testing and commissioning phases and the relevant test procedures must
be available in order that they can then be used- for a direct comparison (either
automatically by an appropriate tool or by hand) to determine that the software still
correctly performs the originally specified tasks.
Testing of software is a very extensive and complicated subject and much liter-
ature has been written on the subject, including several books. A Bibliography is

28
attached to this document and the reader is referred to the references given in
Section 8 therein in order to establish an appropriate and high quality programme
of software testing.

8.1. TEST PLANNING

Test planning should include:

(1) Delineation and description of the purpose and scope of each level of testing
to be conducted on each deliverable item or support item
(2) Identification of the responsibilities for each level of testing of each item
(3) Identification and description of the pre- and post-test documentation that will
be generated for each level of testing, including test specifications, procedures,
logs and records
(4) The test methods which will be used to establish compliance, i.e. test by func-
tion or test by structure
(5) Identification and use of the test support software and computer hardware to
be used in testing
(6) The test standards and quality criteria for acceptance that will be employed
(7) Identification and schedule for the user's manuals and operator handbooks
(8) Test reviews.

It is recommended that test planning should follow the procedures implied by

one of the many standards on test documentation, resulting in the production of an
approved test plan and test procedures.

8.2. TEST PERFORMANCE

Performance of the actual testing should follow the test plan that has been
developed. During the course of performance, the test procedures should be followed
in detail and appropriate records kept. When modules and subsystems have been
tested by the software developer and independently verified, the elements of the
system should be passed to an independent group responsible for testing the complete
system. This may involve software from third parties; acceptance tests for such
external software should be devised. System integration testing should be carried out
using the operating hardware environment. All the test data should be compiled
independently of those data used by the software developer, which should have been
designed and verified in accordance with the same principles as those for the
designed software.
Several national and international standards are listed in Section 8 of the
Bibliography for guidance when planning tests for the software. The design of

29
appropriate individual tests can be obtained by referring to the text books identified
in the same section of the Bibliography.

8.3. TEST REVIEWS

The test plan should identify all the review type functions related to software
testing, including;

(1) A review of the software requirements to determine their testability

(2) A review of the test plans and procedures to ensure compliance with appropri-
ate standards and satisfaction of any contractual and statutory requirements
(3) A review of the test requirements and criteria to ensure the adequacy, feasibil-
ity and satisfaction of the relevant requirements specification
(4) Monitoring of the test and verification processes to establish that the test results
are the actual findings of the test
(5) Review and verification of the test reports
(6) Ensuring that the test related documentation is retained for comparison
purposes if the tests are repeated.

The review procedures should follow the guidance given in the texts cited in
Section 8 of the Bibliography.

8.4. ACCEPTANCE TESTING AND CERTIFICATION

Software acceptance testing carries a degree of formality that is not present

during development testing in that the acceptance tests are approved by the user who
may also formally certify his acceptance. Many of the tests used in the course of this
testing may be identical to those which are performed during development testing.
To carry out adequate and complete acceptance testing it is advisable to iden-
tify each software requirement according to some numerical or other scheme, so that
requirements can be related to appropriately recorded results.

Note: Software acceptance testing may be a part of total system testing.

9. NON-CONFORMANCE CONTROL

For control of non-conformances, reference should be made to Section 10 of

Code of Practice 50-C-QA.
The recording of non-conformances should commence as early as possible in
the life-cycle.

30
 Copies of all reports on software non-conformances should be deposited with
the software library and feature in configuration status reports.
All amended (repaired) software should be retested against existing or new test
specifications and reconfigured. A typical software non-conformance report form
and a non-conformance report log are shown in Annexes T - l and T - 2 .
Reports for verification, validation and audit activities should contain details
of the non-conformances raised during these activities.
The project manager should be responsible for the review, approval and dispo-
sition of the reported non-conformances.

10. CORRECTIVE ACTION

For corrective action activities, reference should be made to Section 11 of

Code of Practice 50-C-QA.
Consideration should be given to analysis of the non-conformance reports to
provide a history from which information can be used for reliability modelling and
to improve standards and practices in the design and use of software. (Reliability
modelling is not covered by the Manual.)

1 1 . RECORDS

Reference should be made to Safety Guide 50-SG-QA2 for the generation,

identification, collection, indexing, filing, storing, maintenance and disposition of
quality assurance records, and to Section 5 of this Manual.
Annexes L - l to L - 5 indicate some of the records which should be retained.

' 12. AUDITING

Reference should be made to Section 13 of Code of Practice 50-C-QA, Quality

Assurance Auditing for Nuclear Power Plants: A Safety Guide, SG-QA-10, and the
Manual on Quality Assurance Programme Auditing, Technical Reports Series
No. 237, for the auditing requirements.
Some aspects which should be audited are:

(1) Coding implementation: to determine compliance with the detailed design

specification and standards

31
(2) Tested software: to determine compliance with the requirements specification
(3) Consistency between software and documentation before delivery
(4) Configuration.

Ideally, configuration audits should be carried out when the configuration item
is first brought under configuration control, and whenever any changes are made.
Checks should be made to verify that the configuration control procedures have been
applied, and that the item is uniquely identified and mutually traceable to the
associated documentation.
Configuration audits should also include verification of the accuracy of the
master configuration index and the functioning of the software library.
Annexes V - l to V - 3 contain examples of the check-lists that can be used for
auditing.

32
 ANNEXES

Annex A

IAEA DOCUMENTS
REFERRED TO IN THE MANUAL

50-C-QA: Quality Assurance for Safety in Nuclear Power Plants: A Code of

Practice (1978) (Chinese, English, French, Russian, Spanish)
50-SG-QA1: Establishing the Quality Assurance Programme for a Nuclear Power
Plant Project: A Safety Guide (1984) (English, French, Russian,
Spanish)
50-SG-QA2: Quality Assurance Records System: A Safety Guide (1979) (Chinese,
English, French, Russian, Spanish)
50-SG-QA3: Quality Assurance in the Procurement of Items and Services for
Nuclear Power Plants: A Safety Guide (1979) (Chinese, English,
French, Russian, Spanish)
50-SG-QA6: Quality Assurance in the Design of Nuclear Power Plants: A Safety
Guide (1981) (Chinese, English, French, Russian, Spanish)
50-SG-QA7: Quality Assurance Organization for Nuclear Power Plants: A Safety
Guide (1983) (English, French, Russian, Spanish)
50-SG-QA8: Quality Assurance in the Manufacture of Items for Nuclear Power
Plants: A Safety Guide (1981) (Chinese, English, French, Russian,
Spanish)
50-SG-QA10: Quality Assurance Auditing for Nuclear Power Plants: A Safety
Guide (1980) (Chinese, English, French, Russian, Spanish)
Technical Reports Series No. 237: Manual on Quality Assurance Programme
Auditing (1984) (English)

33
 Annex B

GLOSSARY

The glossary defines some of the terms used in software engineering.

acceptance testing: Formal testing conducted to determine whether or not a system

satisfies its acceptance criteria and to enable the customer to determine whether
or not to accept the system. (See also system testing.)
algorithm: A finite set of well defined rules for the solution of a problem in a finite
number of steps or, in the cases of recursion and iteration, with a specified
condition.
application software: Software specifically produced for the functional use of a
computer system to perform specific tasks in the user's operational environ-
ment (for example, software for reactor control and instrumentation, reactor
core performance calculations and monitoring, fuelling machine control, plant
simulation, etc).
architectural design: The process of defining a collection of hardware and software
components and their interfaces to establish a framework for the development
of a computer system,
assemble: To translate a program expressed in an assembly language into a machine
language. Assembling is usually accomplished by substituting machine
language operation codes for assembly language operation codes and by
substituting absolute addresses, immediate addresses, relocatable addresses, or
virtual addresses for symbolic addresses,
assembler: A computer program used to assemble. (See compiler, interpreter.)
Synonymous with assembly program,
assembly language: A computer oriented language whose instructions are usually in
one to one correspondence with the computer's internal instructions. (See
machine code, assemble, assembler.)
backup: Provisions made for the recovery of data files or software, for restart of
processing, or for use of alternative computer equipment after a system failure
or a disaster.
baseline: A configuration identification document or a set of such documents
formally designated and fixed at a specific time during a configuration item's
life-cycle. Baselines, plus approved changes from these baselines, constitute
the current configuration identification,
certification: A written guarantee by a responsible body that a system or computer
program complies with its specified requirements,
code: Data or a program in a form that can be accepted by a computer system.

34
code audit: An independent review of source code done by a person, team or tool
to verify compliance with software design documentation and programming
standards. Correctness and efficiency may also be evaluated. (See also static
analysis, inspection, walk-through.)
code inspection: See inspection,
code walk-through: See walk-through.
compile: To translate a higher order language program into its relocatable or
absolute machine code equivalent,
compiler: A computer program used to compile. (See assembler, interpreter.)
component: A basic part of a system, computer program, or module,
computer: A functional programmable unit that consists of one or more associated
processing units and peripheral equipment that is controlled by internally
stored programs and that can perform computation, including arithmetic opera-
tions or logic operations, without human intervention,
computer program: A sequence of instructions suitable for processing by a
computer. Processing may include the use of an assembler, an interpreter,
or a translator (assembler or compiler) to prepare the program for execution
as well as to execute it.
computer program abstract: A brief description of a computer program,
providing sufficient information for potential users to determine the
appropriateness of the computer program to their needs and resources,
computer program configuration identification: See configuration identifi-
cation.
computer software: See software.
computer system: A functional unit, consisting of one or more interconnected
computers and associated software, that uses common storage for all or part
of a program and also for all or part of the data necessary for the execution
of the program; executes user written or user designated programs; performs
user designed data manipulation, including arithmetic operations and logic
operations; executes programs that modify themselves during their execution.
A computer system may be a stand alone unit or may consist of several inter-
connected units.
computing facility: A service which is available to several users, enabling them to
run their own software or that provided by the service,
configuration: The complete description of a product, which is a collection of
its descriptive and governing characteristics that can be expressed in functional
and physical terms. The functional terms express the performance that the item
is expected to achieve; the physical terms express the physical appearance and
composition of the item,
configuration audit: The process of verifying that all the required configuration
items have been produced, that the current version agrees with the specified
requirements, that the technical documentation completely and accurately

35
 describes the configuration items, and that all the change requests have been
resolved.
configuration baseline: A specific reference point of product definition to which
changes or proposed changes may be related,
configuration control: The process of evaluating, approving or disapproving and
co-ordinating changes to configuration items after formal establishment of
their configuration identification,
configuration identification: The process of designating the configuration items in
a system and recording their characteristics,
configuration item: A collection of hardware or software elements treated as a unit
for the purpose of configuration management,
configuration management: The process of identifying and defining the configura-
tion items in a system, controlling the release and change of these items
throughout the system life-cycle, recording and reporting the status of configu-
ration items and change requests, and verifying the completeness and correct-
ness of configuration identification, configuration control, configuration
status accounting and configuration audit,
configuration status accounting: The recording and reporting of the information
that is needed to manage a configuration effectively, including a listing of the
approved configuration identification, the status of proposed changes to the
configuration and the implementation status of approved changes,
database: (1) A set of data items, together with the defined relationships between
them
(2) A collection of data fundamental to a system
(3) A collection of data fundamental to an enterprise,
defect: See error.
design language: A language with special syntax and sometimes verification
protocols used to develop, analyse and document a software design,
design level: The design decomposition of the software item (for example, system,
subsystem, computer program or module),
design methodology: A systematic approach to designing software consisting of
the ordered application of a specific collection of tools, techniques and
guidelines.
design requirement: Any requirement that impacts or constrains the design of
a software system or software system component, for example, functional
requirements, physical requirements, performance requirements, software
development standards, software quality assurance standards. (See also
requirements specification.)
design review: The formal review of an existing or proposed design for the purpose
of detection and remedy of design deficiencies that could affect fitness for use
and environmental aspects of the product, process or service and/or for iden-

36
 tification of potential improvements of performance, and safety and economic
aspects.
desk checking: The manual simulation of program execution to detect faults
through step by step examination of the source code for errors in logic or
syntax. (See also static analysis),
detailed design: The process of refining and expanding the preliminary design
to contain more detailed descriptions of the processing logic, data structures
and data definitions to the extent that the design is sufficiently complete to be
implemented.
development methodology: A systematic approach to the creation of software
that defines development phases and specifies the activities, products, verifica-
tion procedures and completion criteria for each phase,
dynamic analysis: The process of evaluating a program based on execution of the
program. (See static analysis.)
enhancement: The process adopted by the user to charge software to improve
or add to its performance,
error: A discrepancy between a computed, observed or measured value or
condition and the true, specified or theoretically correct value or condition.
(See also non-conformance.)
error analysis: The process of investigating observed software errors with the
purpose of tracing each error to its source and of determining quantitative
rates and trends.
flow chart: A graphic representation of the definition, analysis or solution of a
problem in which symbols are used to represent operations, data, flow and
equipment.
formal testing: The process of conducting testing activities and reporting results
in accordance with an approved test plan,
functional requirement: A requirement that specifies a function that a system
or system component must be capable of performing,
functional specification: A specification that defines the functions that a system
or system component must perform,
implementation: The process of translating a design into a correct usable code,
independent verification and validation: Verification and validation of a soft-
ware product by individuals or groups other than those who performed the
original design, but who may be from the same organization,
inspection: An evaluation technique in which the software requirements, design
or code are examined in detail by a person or group other than the author to
detect errors. (See also walk-through and code audit.)
interface: (1) An interaction or communication with another system component
or another system across a shared boundary.
(2) See Section 4.2 of text for organization interfaces.

37
interpreter: A program that interprets and executes source language statements at
run time in a computer,
machine code: A representation of instructions and data that is directly executable
by a computer. (See assembly language.)
maintenance: Any change that is made to the software, either to correct a deficiency
in performance, as required by the original requirements specification, or to
improve its operation (i.e. enhancement),
methodology: A system of methods and rules applicable to research or work in a
given science or art.
module: A logical separable part of a program.
non-conformance: A deficiency in characteristics, documentation or procedure
which renders the quality of an item unacceptable or indeterminate. For the
purpose of the Manual, the terms 'error' and 'non-conformance' are
synonymous,
non-conformance analysis: (See error analysis.)
object program: A program in machine code that is the output after translation
from the source program and is ready to be executed in the computer. This
is also known as executable code. (See source program.)
operating system: Software that controls the execution of computer programs,
provides hardware resource allocation input/output control and related
services. The most important part of the system software is sometimes called
supervisor, monitor, executive or master control program. Although operating
systems are predominantly software, partial or complete hardware implemen-
tations are possible. (See also system software.)
operational: Pertaining to the status given a software product once it has entered the
operation and maintenance phase,
operational reliability: The reliability of a system or software subsystem in its
actual use environment. Operational reliability may differ considerably from
reliability in the specified or test environment,
patch: The modification made to an object program without recompiling the source
program.
Note: The practice of patching is strongly discouraged for safety related
software.
program instrumentation: The process of preparing and inserting probes, such as
instructions or assertions into a computer program to facilitate execution
monitoring, to prove correctness and to aid resource monitoring or other
activities.
real time: The processing of data by a computer in connection with another process
outside the computer according to the time requirements imposed by the
outside process. This term is also used to describe systems operating where
operation can be influenced by human intervention.

38
requirements specification: A specification that sets forth the requirements for
a system component, for example, a software configuration item. Typically
included are functional requirements, performance requirements, interface
requirements, design requirements and development standards,
routine: A set of instructions arranged in proper sequence to enable the computer
to perform a desired task,
simulation: The representation of simulated characteristics of the behaviour of one
physical or abstract system by another system,
software: Computer programs, procedures, rules and possibly associated
documentation and data pertaining to the operation of a computer system. (See
also application software, system software, utility software and computer
software.)
software database: A centralized file of data definitions and present values
for data common to, and located internal to, an operational software system,
software documentation: Data or information, including computer listings and
printouts, in human readable form, that describe or specify the design or
details, explain the capabilities, or provide operating instructions for using the
software to obtain the desired results from a software system. (See also system
documentation, user documentation.)
software engineering: The systematic approach to the development, operation,
maintenance and decommissioning of software,
software item: Source code, object code, job control code, control data, or a
collection of these items,
software librarian: The person responsible for establishing, controlling and
maintaining a software library,
software library: A controlled collection of software and related documentation
available to aid in software development, use or maintenance. Categories of
software held in the library may include development software, released soft-
ware and archive software,
software life-cycle: The period of time that starts when a software product
is conceived and ends when the product is decommissioned,
software maintenance: Modification of a software product after delivery to correct
faults, to improve performance or other attributes, or to adapt the product to
a changed environment,
software non-conformance: See error.
software product: A software entity designated for delivery to a user,
software tool: A computer program or device used to help develop, test, analyse
or maintain another computer program or its documentation, for example, an
automated design tool, translator, test tool, maintenance tool,
source language: A language used to write source programs,
source program: A computer program that must be translated or interpreted before
being executed by a computer. (See object program.)

39
static analysis: The process of evaluating a program without executing the program.
(See also desk checking, code audit, dynamic analysis, inspection,
walk-through.)
structured design: A disciplined approach to software design that adheres to
a specified set of rules based on principles such as top-down design, stepwise
refinement and data flow analysis,
structured program: A program constructed of a basic set of control structures,
each having one entry point and one exit. The set of control structures typically
includes: a sequence of two or more instructions, conditional selection of one
of two or more instructions, or sequences of instructions and repetition of an
instruction or a sequence of instructions,
structured programming: Any well defined software development technique that
incorporates top-down design and implementation and strict use of structured
program control constructs. (See top-down.)
support software: See utility software.
system: A collection of people, machines and methods organized to accomplish
a set of specific functions,
system architecture: The structure and relationship among the components of
a system. It may also include the system's interface with its operational
environment.
system design: The process of detailing the hardware and software architectures,
components, modules, interfaces and data for a system to satisfy specified
system requirements,
system documentation: Documentation conveying the requirements, design
philosophy, design details, capabilities, limitations and other characteristics of
a system (see user documentation), for example, quality records such as
quality assurance reports and non-conformance reports,
system software: Software designed for a specific computer system or family of
computer systems to facilitate the operation and maintenance of the computer
hardware and application programs. Typically included are operating
systems, translators and utility software. (See application software.)
system testing: The process of testing an integrated hardware and software system
to verify that the system meets its specified requirements. (See also
acceptance testing.)
test plan: A document prescribing the approach to be taken for intended testing
activities. The plan typically identifies the items to be tested, the testing to be
performed, test schedules, personnel requirements, reporting requirements,
evaluation criteria and any risks requiring contingency planning,
top-down: Pertaining to an approach that starts with the highest level component
of a hierarchy and proceeds through progressively lower levels, for example,
top-down design, structured programming, top-down testing.

40
top-down design: The process of designing a system by identifying its major
components, decomposing them into their lower level components and iterat-
ing until the desired level of detail is achieved,
top-down testing: The process of checking out hierarchically organized programs
progressively, from top to bottom, using simulation of lower level
components.
translator: A computer program used to convert programs from one programming
language (source language) to another (object language). (See compiler or
assembler.)
user documentation: Documentation conveying to the end user of a system instruc-
tions for using the system to obtain the desired results, for example, a user's
manual. (See system documentation.)
utility software: Computer programs or routines designed to perform some
general support function required by other application software, by the
operating system, or by system users. It also includes tools,
validation: Term that has a more global connotation than verification. It is an
activity that is performed at the end of development and integration to check
that the final software meets the original intention. The activity is usually test-
ing/exercising the system (hardware and software) functions within the bounds
of the system requirements specification(s). User documentation is also
applied during execution of the validation test plan by qualified personnel.
Validation also checks the resolution of all anomalies noted during all software
project verification activities,
verification: An activity which assures that the results of each successive step in the
software development cycle correctly encompass the intentions of the previous
step. This activity is performed by qualified personnel at convenient quality
verification points, e.g. between project milestones such as the completion of
a module and its release to the library, etc. Examples of verification activities
are testing, checking, analysing, design review, structured walk-throughs,
quality measurements, code inspections, etc.
walk-through: A review process in which a designer or programmer leads one or
more other members of the development team through a segment of design or
code that he or she has written, while the other members ask questions and
make comments about technique, style, possible errors, violation of develop-
ment standards and other problems. (See inspection.)

41
 Annex V-3

SOFTWARE QUALITY ATTRIBUTES

Accountability refers to the ability to which usage of the computer resources

by a module or program can be measured. This means that critical segments
of the code can be instrumented with probes to measure timing, whether
specific branches are exercised, etc. The code used for probes is preferably
invoked by conditional assembly or compilation.
Accuracy refers to the extent that the code's outputs are sufficiently precise to
satisfy their intended use.
Augmentability refers to the extent that the code can accommodate expansion
in component computational functions or data storage requirements.
Communicativeness refers to the extent that the form and content of the code's
inputs and outputs are clearly annotated and readily understood. Communica-
tiveness also includes those attributes of the software that provide the standard
protocols and interface routines required to couple the system with another
independent system.
Completeness refers to the extent that all the code's required functions are
present and fully developed. This implies, for example, that external reference
documents are available and that all the required functions are coded and
present as designed.
Conciseness is the absence of redundant or excessive coding and that the required
functions are implemented with a minimum amount of coding. This implies
that the program is not excessively fragmented into modules, overlays, func-
tions and subroutines, nor that the same sequence of coding is repeated in
numerous places, rather than defining a subroutine.
Consistency refers to the extent that the code contains uniform notation, terminol-
ogy, comments, symbology and implementation techniques.
Correctness means the ability of the software to produce the specified outputs when
given the specified inputs and the extent to which a program satisfies its
specification.
Device independence is the ability of the code to be unaffected by changes
made to the computer hardware or peripheral equipment. This means that
coding which is directly related to a specific hardware device should be
minimized, isolated and specifically identified.
Efficiency refers to the extent that the code performs its required functions
without waste of resources. This implies that choices of source code construc-
tions must be made in order to produce the minimum number of words of
object code, or that where alternative algorithms are available, those taking the
least time are chosen; or that the information packing density in the core is
high, etc.

42
Error handling capability refers to the code's ability to handle errors due to
hardware or software failures or operator input in such a way that the resulting
system performance degrades gracefully rather than catastrophically.
Human engineering (ergonomics) refers to the extent that the code fulfils its
purpose without wasting the user's time and energy, or degrading his morale.
Inputs and outputs should be self-explanatory, easy to learn and understand,
unambiguous and designed to avoid misinterpretation. This attribute implies
robustness and communicativeness.
Integrity refers to the extent to which access to software or data by unauthorized
persons can be controlled.
Interoperability refers to the effort required to couple the code system to
another independent code system.
Maintainability refers to the extent that the code facilitates updating to satisfy
new requirements, to correct deficiencies or to move to a different but similar
computer system. This implies that the code is understandable, testable and
modifiable.
Modifiability refers to the characteristics of the design and implementation of
the code which facilitates the incorporation of changes, once the nature of the
desired change has been determined.
Portability refers to the extent that the code can be operated easily and well
on computer configurations other than its current one.
Readability refers to the extent that the code's function and design can easily be
read and understood. (Example: complex expressions have meaningful
mnemonic variable names, and the code is extensively annotated.) Readability
is necessary for understandability.
Reliability refers to the extent that the code can be expected to continue to
perform its intended functions.
Reusability refers to the extent to which a program, or pieces of it, can be used
in other applications. This is related to the packaging and scope of the functions
that programs perform.
Robustness refers to the extent that the; code can continue to perform despite
some violation of the assumptions in its specification. This implies, for
example, that the program will handle inputs or intermediate calculated
variables which are out of range, or in a different format or type than specified,
without degrading its performance of functions not dependent on the inputs or
variables.
Self-containedness refers to the extent that the code performs all its explicit and
implicit functions within itself, e.g. initialization, input checking, diagnostics,
etc.
Self-descriptiveness refers to the extent that the code listing contains enough
information for a reader to determine or verify its objectives, assumptions,
constraints, inputs, outputs, components and revision status.

43
Simplicity refers to those attributes of the software that provide implementation of
functions in the most understandable manner, and usually requires avoidance
of practices which increase complexity.
Structuredness refers to the extent that the code possesses a definite pattern of
its interdependent parts. This implies that the program design has proceeded
in an orderly and systematic manner (e.g. top-down design or functional
decomposition) and that standard control structures have been defined and fol-
lowed in coding the program, e.g. DO-WHILE, IF-THEN-ELSE, resulting in
well structured paths through the program.
Testability refers to the extent that the code facilitates the establishment of test plans,
specifications, procedures and their implementation, e.g. modular construction
with well defined interfaces that can be independently tested.
Traceability refers to those attributes of the software that provide a thread
from the requirements to the implementation with respect to the specific
development and operational environment.
Understandability refers to the extent that the code's functions are clear to the
reader. This implies that variable names or symbols are used consistently, that
modules of code are self-descriptive, and that the control structure is simple
or in accordance with a prescribed standard, etc. There should be no hidden
meanings or operating characteristics that come to light only after months of
use.
Usability refers to the effort required to learn, operate, prepare input and interpret
output of a program.

44
 Annex K-2

LIST OF DOCUMENTS

Programmatic documents

Quality assurance programme document

Software quality plan
Configuration control management
Control of other administrative functions

Work oriented documents

Development plan
Configuration plan
Documentation plan
Test plan
Verification plan
Requirements specification 1
Software system design 1
Detailed functional specification 1
Program design document 1
Program test document 1
System test document 1
Acceptance test document 1
Commissioning and handover documents 1
Operation and maintenance documents 1

1
Indicates documents which, as a minimum, are subject to configuration control.
 Annex V-3

CONTENTS OF A TYPICAL PROCEDURE:

SOFTWARE ARCHIVING AND CONTROL PROCEDURE

1. Introduction and scope

2. Contents of tapes
3. Maintaining the catalogue of tapes
3.1. Used tapes
3.2. New/surrendered tapes
3.3. Faulty tapes
3.4. Off-site tapes
4. Formal software issue procedure
5. Tape rewriting procedures
5.1. System volume with bootstrap area
5.2. System volume without bootstrap area
5.3. Fileset
5.4. File/masterfile
5.5. Hardware test programs
6. . Disk to tape archiving procedures
6.1. Use of fileset
6.2. Daily archive
6.3. Weekly archive
6.4. Monthly archive
6.5. General archives
7. Software control procedures
7.1. Archiving of software modules
7.2. Retrieving software modules from archive
7.3. Purging the software control masterfile
7.4. Masterfile recovery
7.5. Software module recovery
8. System reports procedure
9. Computer booking procedure
10. Forms in use for software archiving and control procedures
11. Archive/rewrite timetable
12. References

46
 Annex V-3

TYPICAL PROCEDURE:
PROCEDURE FOR THE USE OF COMPUTER PROGRAMS
FOR SAFETY RELATED DESIGN PURPOSES

Contents

(1) Scope
(2) Responsibilities
(3) Requirements
(4) Records

1. SCOPE

This document defines the requirements for controlling the use of computer
programs for design purposes. The objectives are to ensure that:

(1) Computer programs which are used for design calculations produce results
which are valid for the actual physical problems under study.
(2) A comprehensive record is kept of the modelling assumptions and the data
used, so that a recalculation is possible at a later date using whatever tools are
available at the time.

2. RESPONSIBILITIES

It is the responsibility of the design manager to ensure that:

(1) Indexes of computer programs are maintained.

(2) Appropriate reports are produced where required by the documentation section
of this procedure. These reports shall conform to procedures for review and
approval.
(3) Software records are produced, including the computer program input data and
results needed to substantiate design decisions, together with details of the
model.

It is the responsibility of all computer users to ensure that the programs which
they use for design calculations are included on the index of programs.

47
3. REQUIREMENTS

3.1. Program index

An index of programs used for design purposes shall be produced and shall
include:

(1) Program name and identifier. The identifier shall be sufficiently detailed so as
to indicate a specific version and/or an amendment of revision of the program;
each change must be recorded. Programs which are no longer used should not
be removed from the index.
(2) Installation used.
(3) Organization originating the program.
(4) Validation report references.
(5) Details of the program control arrangements. It is essential that the control
arrangements should be sufficient to ensure that programs are not inadvertently
modified from the validated and documented version. Any change in the pro-
gram identifier, as new releases of a program become available, shall be
recorded as a new entry on the index and an amendment issued to the computer
program report.

3.2. Validation report

T h i s r e p o r t m u s t include, o r q u o t e r e f e r e n c e s to, other r e p o r t s w h i c h c o v e r t h e

following points:

(1) An explanation as to how this program is applicable to the specific type of

problem under study.
(2) The modelling assumptions necessary to enable data to be prepared.
(3) Examples of results which demonstrate their consistency with known solutions
or experimental results; however, if this is impractical, a comparison of results
from an independent computer program should be presented.

3.3. Computer program report

The report shall include details of the mathematical model used and the method
of solution, and quote the test cases which have been used. All routes through the
program which are tested should be identified. If possible, analytical results should
be included for comparison purposes. User instructions should be given. Each
change in the program (which will automatically require a change of identification),
will require an associated amendment to the computer program report.

48
3.4. Use of computer programs

Each use of a computer program for design purposes will require the retention
of the following information:

(1) The associated job control language in order to demonstrate that the program
used is contained in the appropriate index.
(2) The associated input data and all or part of the output results.
(3) Any associated user supplied routines which have been linked with the
protected program at run time, unless these have already been separately
included on the index.

4. RECORDS

The following, documents should be retained in design files:

(1) Program index.

(2) Validation reports.
(3) Computer program reports.
(4) The information described in Section 3.4.

49
 Annex V-3

TYPICAL RESPONSIBILITIES
OF THE QUALITY ASSURANCE FUNCTION

The quality assurance function should be responsible for:

(1) Verifying the effective implementation of the software quality assurance

programme.
(2) Co-ordinating the quality assurance activities of the software project team,
maintaining the quality assurance programme description and issuing all
revisions of the programme.
(3) Evaluating and issuing the approval of main suppliers' quality assurance
programmes and relevant documents such as quality assurance manual and
procedures, and recommending their inclusion on the approved suppliers list.
(4) Reviewing quality assurance requirements for the procurement of services and
equipment.
(5) Reviewing software quality plans and schedules as well as the work procedures
to be used in software activities.
(6) Co-ordinating or initiating the preparation of non-conformance reports when
deficiencies are identified, and receiving users' non-conformance reports.
(7) Organizing audits of the quality assurance programme of the main contractors
and suppliers identifying deficiencies in the programme and verifying the
implementation of corrective action.
(8) Recommending corrective action and re-auditing of the contractors when cor-
rective action has been implemented.
(9) Organizing and conducting quality assurance programme orientation courses
for the project team personnel.
(10) Identifying significant conditions adverse to quality, and verifying that the
corrective action is taken, including action to prevent repetition.
(11) Co-ordinating with the quality assurance groups of all contractors and
suppliers.
(12) Planning, preparing and performing the internal quality assurance programme
audits.
(13) Reporting to the management of the responsible organization of the status of
quality assurance activities.
(14) Authorizing the release of deliverables.

50
 Annex V-3

TYPICAL RESPONSIBILITIES OF PROJECT MANAGEMENT

There are many guides and standards available for project management. An
extensive list is given in Section 4(a) of the Bibliography. Reading and implementing
the concepts that are detailed in these guides and tutorials will greatly enhance the
quality and ultimately the reliability of the software.
Project management should be responsible for:

(1) Determining the responsibility, providing plans and identifying internal and
external interfaces for the project organization, lines of directions and commu-
nications among groups and external interfaces.
(2) Controlling preparation, submittal and review of all quality assurance records,
and overseeing the effective implementation of all quality assurance functions
by the software project team.
(3) Ensuring the definition of quality assurance requirements for the software
project team and other contractors, as well as the approval of their quality
assurance manual documents such as quality assurance manual and programme
procedures.
(4) Approving the software quality plans and schedules.
(5) Ascertaining standards, codes, guides and procedures applicable to the soft-
ware project and directing the development and approval of specific proce-
dures and instructions to be used during the development of the activities.
(6) Establishing the software project file system.
(7) Ensuring the management of the configuration control system.
(8) Reporting on an established schedule, to the management of the responsible
organization, on the regular development of the software project as well as
when problems and deficiencies are identified in the process and providing the
approach and schedule for the resolution of problems and deficiencies.
(9) Originating the recommendation for the resolution of non-conformance
reports.
(10) Proposing, reviewing and approving corrective actions.

51
 Annex V-3

TYPICAL DUTIES, AUTHORITIES AND RESPONSIBILITIES

FOR SOME KEY ACTIVITIES IN A PROJECT

(1) Project management has overall responsibility for the project. These respon-
sibilities include ensuring the preparation, implementation and maintenance of
the project quality assurance programme and all referenced plans and proce-
dures. Project management also has the responsibility for ensuring that the
quality assurance programme is reviewed and audited, and that any resulting
corrective actions are implemented adequately and in a timely manner.

(2) Software designers are responsible for:

(a) Interpreting the system requirements specification and plans, resulting in

a well coded software system integrated into the computer system.
(b) Producing all software support documentation such as program descrip-
tions and user manuals.
(c) Assessing and taking action on the results of all verification activities
such as design reviews, code tests, system tests and, where appropriate,
formally approving these before the next activity commences.

(3) Software users have a responsibility for specifying their requirements to

ensure that the software, particularly in the case of scientific software, has
been validated by an approved method for their particular applications. It is the
user's responsibility to ensure that the software is used in accordance with the
supplier's instructions, and that any software errors are formally reported,
either back to the supplier when the software is being maintained, or internally
within the user's organization when the user is maintaining the software. It is
the user's responsibility to formally control any subsequent changes to soft-
ware or associated documentation.

52
 Annex J

STAFF T R A I N I N G R E C O R D F O R M

SHEET OF

PROJECT
S T A F F TRAINING RECORD REVISION N*

IN-HOUSE TRAINING BATES EXTERNAL TRAINING DATES

' NAME OF TRAINEE
COURSE A COURSE B COURSE C COURSE D COURSE 1 COURSE 2 COURSE 3 COURSE t

OBTAINED

OBTAINED
OBTAINED

OBTAINED
OBTAINED

OBTAINED
OBTAINED

OBTAINED

MARK
MARK

MARK

MARK
MARK
MARK

MARK

DATE

DATE
DATE

MARK

DATE
DATE

OATE
DATE

OATE

Ul
 Annex K-2

TYPICAL RECORDS AND VERIFICATION RESPONSIBILITY TABLE

Record Purpose of record Initiated by Verified by

Requirements To establish functional, performance, interface, Software user Independent verifier within user
specification design, validation and acceptance requirements organization

Functional Completion of preliminary design; establishment of Software designer Design team leader
specification test plan (approved by the software user)

Software Completion of detailed design, code and develop- Software design/team Integration team leader after
configuration ment testing stages leader inspection of documents,
submission forms verification of development tests
and successful production of new
version of software

Software Recording of errors observed in the software Anyone working with Librarian maintains these reports;
non-conformance (employing) the integration team member will
reports configured software analyse the significance of the
errors and file a software change
request form

Software Request design team to correct known software Integration team Senior software designer
change request form errors leader
Design change To request project management to make available Design team leader Senior software designer
order additional resources to modify existing software
because of a design change

Integration test To document the detailed description of the tests to Integration test team Design team leader
procedures be performed, the purpose of each test, the evalua-
tion and acceptance criteria to be used, the methods
and the basis for generation of test inputs

Verification and' To document the criteria, techniques and tools to be Verification/validation Independent verifier
validation plan utilized and to describe the activities to be per- team
formed for software verification and validation

Verification test To establish tests which will demonstrate the cor- Verification/validation Design team leader
specifications rectness of the software at each stage and between team
each stage of the development life-cycle

Validation test To establish tests which will demonstrate the cor- Verification/validation Design team leader
specifications rectness of the final software with respect to the team
user needs and requirements

Verification/ To document the detailed description of the tests to Verification/validation Independent verifier
validation test be performed, the purpose of each test, the evalua- test team
procedures tion and acceptance criteria to be used, the methods
and the basis for generation of the test inputs

Integration and To record test events and test results of integration Integration team Integration team leader
test log book" tests member
 Annex K - l (cont.)

Record Purpose of record Initiated by Verified by

System test T o record test events and test results of the System test team System test team leader and user
log book computer system

Final test report Documents the results of all the tests described in System test team Test team leaders
the software test specifications in terms of whether
or not the software has achieved the performance
requirements given in the software functional
requirements

a
See Annex K - 2 for an example of a test record log.

Note: The verification test specifications and the validation test specifications are often combined into one document, the verification and validation
plan. Also, one team, the verification and validation team, can carry out verification and validation activities independent of the design team.
 Annex K-2

TYPICAL FORM FOR TEST RECORD LOG

TEST RECORD LOG Sheet of

Test Software
Record Module
Number Name Release Issue Version Date Originator

TR/

TR/

TR/

TR/

TR/

TR/

TR/

TR/

TR/

TR/

TR/

TR/

TR/

TR/

TR/
Lh Annex K-2
00
FORM FOR MEDIA CONTROL:
REGISTER OF MEDIA
SHEET OF

PROJECT
REGISTER OF MEDIA REVISION OATE

REFERENCE N ' MEDIA TYPE ISSUE N* DATE TITLE

 Annex L-2
FORM FOR MEDIA CONTROL:
PROGRAM LOCATION RECORD
SHEET OF

PROJECT
PROGRAM LOCATION RECORD REVISION DATE

PROGRAM SOURCE LOCATION OBJECT LOCATION DOCUMENTATION

VERSION NOTES
IDENTIFIER MASTER COPT MASTER COPY LOCATION

Ui
VO
 Annex K-2
FORM FOR MEDIA CONTROL:
ARCHIVES RECORD

SYSTEM

PROJECT
ARCHIVES RECORO
PROGRAM IDENTIFIER TITLE

SUBMITTED
ITEM ARCHIVED VERSION LOCATION
DATE BY

MASTER

LISTING

SPECIFICATION DOCUMENTS

TEST DOCUMENTS

USER DOCUMENTS

ACCESS AUTHORIZED TO STAfF NAMED MOVEMENT RECORD

NAME SIGNATURE WITHORAWN BY RETURNED BY

ITEM (SIGNATURE! DATE DATE
(SIGNATURE!
a

AUTHORIZED BY DATE
 Annex K-2

FORM FOR MEDIA CONTROL:

RECORD OF FAULTY MEDIA

PROJECT MEDIA DATE OF RETURNEO

OETAILS OF FAULT DATE
REFERENCE N* FAULT TO
 Annex K-2
FORM FOR MEDIA CONTROL:
PROGRAM MODIFICATION RECORD
SHEET OF
PROGRAM MODIFICATION RECORO
TO
O
REVISION DATE
—1

PROGRAM IDENTIFIER NAME

SOURCE FILE TYPE LOCATION OF FILE

LOCATION OF SOURCE LISTING

ACCESS INFORMATION

LINK INFORMATION

OUTSTANDING CHANGE
VERSION DATE COMMENTS NOTES
REQUESTS (AUTHORIZED)
 Annex V-3

FORM FOR CONFIGURATION CONTROL:

MODULE RELEASE

PROJECT
MODULE R E L E A S E

SYSTEM NAME

MODULE NAME IDENTIFIER

MODULE VERSION / REVISION

MODULE SPECIFICATION N*

DATE OF MOOULE LISTINGS

PROGRAM SOURCE REFERENCE

DATE OF TEST TEST SPECIFICATION ..

Revision:

DATE CONFIGURED

COMMENTS

PROGRAMMER DATE TESTER DATE APPROVER DATE

63
 Annex K-2

FORM FOR CONFIGURATION CONTROL:

SOFTWARE CHANGE REQUEST

PROJECT SCR N*
S O F T W A R E CHANGE R E Q U E S T
SHEET OF

SOFTWARE IDENTIFIER SOFTWARE VERSION

DETAILS OF CHANGE REQUESTEO

REASON FOR CHANGE

CHANGES REQUIRED TO

SOURCE MASTER Y/N FLOW CHART Y/N

OBJECT MASTER Y/N TEST SPECIFICATION Y/N

DESIGN SPECIFICATION Y/N TEST SCHEDULE Y/N

LISTING Y/N HARDWARE Y/N

SOFTWARE AFFECTED

DISTRIBUTE TO

ORIGINATOR DATE APPROVER DATE

 Annex M - 3
FORM FOR CONFIGURATION CONTROL:
SOFTWARE CHANGE REGISTER

SHEET OF

PROJECT
SOFTWARE CHANGE REGISTER REVISION OATE

CHANGE DATE OATE NEW DATE OF OATE

DESCRIPTION VERSION COMMENTS
N* RAISED IMPLEMENTED RETEST RECONFIGURED
N'-
 Annex K-2
FORM FOR CONFIGURATION CONTROL:
CONFIGURATION STATUS REPORT
SHEET OF

PROJECT
CONFIGURATION S T A T U S REPORT REVISION OATE

CONFIGURATION OATE SPECIFICATION TEST SPECIFICATION TEST SCHEDULE DOCUMENTATION

VERSION CONFIGURED
ITEM IDENTIFIER N* REVISION N"' REVISION N* REVISION REVISION
 Annex K-2

FORM FOR CONFIGURATION CONTROL:

DOCUMENT REGISTER

PROJECT SHEET OF
DOCUMENT REGISTER
REVISION DATE

REV DATE ORIG, DATE OF

DOCUMENT NOTES
N". APPROVED LAST REV.

QUALITY ASSURANCE
V/L PROGRAMME
H- SOFTWARE QUALITY
Z .
UJ PLAN
X REQUIREMENTS
=3 CUSTOMER APPROVAL
SPECIFICATION
0 FUNCTIONAL
SPECIFICATION

UJ DETAILED SOFTWARE
DESIGN
cc. TEST PLAN
CL
\
x SYSTEM TEST
UJ SPECIFICATION
T/1 SYSTEM TEST
V
1/1 SCHEDULE
ACCEPTANCE TEST CUSTOMER APPROVAL
SPECIFICATION
ACCEPTANCE TEST CUSTOMER APPROVAL
SCHEDULE

SUBSYST. OR MODULE N '

M B I B l l l l I f i i i i i l B W
DETAIL
SPEC.
TEST
SPEC.
TEST
SCHEDULE
1/1
h-
Z
DETAIL
SPEC
X TEST
SPEC-
0 TEST
0 SCHEDULE
U
_ JJ DETAIL
SPEC.
0 TEST
0 SPEC.
TEST
0 SCHEDULE
< DETAIL
SPEC.
X
UJ TEST
1— SPEC.
L/> TEST
>-
SCHEDULE
CA DETAIL
RS
1/1 SPEC
TEST
SPEC
TEST
SCHEOULE
 Annex K-2

VERIFICATION AND VALIDATION ACTIVITIES

APPLIED TO A TYPICAL SOFTWARE LIFE-CYCLE

COMPUTER SYSTEM
REQUIREMENTS

REQUIREMENTS
SPECIFICATION
INTEGRATION
HARDWARE AND TEST
REQUIREMENTS REQUIREMENTS

|
FUNCTIONAL
SPECIFICATION

PLAN
ARCHITECTURAL
DESIGN S
t—
<
a
DETAILED
DESIGN DESIGN

COOING
BUILD AND
INTEGRATION

HARDWARE
SOFTWARE
INTEGRATION

COMPUTER
SYSTEM
TESTING
I
• - J
 Annex K-2

POSSIBLE ADDITIONAL ITEMS

FOR CONFIGURATION CONTROL

Results of design reviews

Software requirements review 1

Preliminary design review 1
Critical design review 1
Verification readiness review 1
Software verification review 1
Functional configuration audit 1
Physical configuration audit 1

Individual software modules (or subsystems)

Source code listing

Test plans
Test procedures
Test results
Load module — affected by:
Translator
Linking loader
Operating system
Mathematics library
System utilities
System parameters

Data

Module parameters
Input data
Databases

1
See Fig. 3 in text-
 Annex V-3

OUTLINE OF A TYPICAL SOFTWARE

VERIFICATION AND VALIDATION PLAN 1

1. Purpose
2. Definitions
3. Verification and validation summary
3.1. Organization
3.2. Schedule
3.3. Resources summary
3.4. Responsibilities •
3.5. Tools, techniques and methods
4. Life-cycle verification and validation (V&V)
Each subsection shall address the following topics as appropriate:
Verification and validation tasks
Methods and criteria
Inputs/outputs
Schedule
Resources
Assumptions and reservations
Responsibilities
4.1. Management of V&V
4.2. Design specification V&V
4.3. Design V&V
4.4. Implementation V&V
4.5. Testing V&V
4.6. Operation and maintenance V&V
5. Software verification and validation reporting
5.1. Required reports
5.2. Optional reports
6. Verification and validation administrative procedures
6.1. Non-conformance reporting and resolution
6.2. Corrective action policy
6.3. Control procedures
6.4. Standards, practices and conventions
7. Referenced documents

1
Based on Draft Standard for Software Verification and Validation Plans, IEEE
Draft STD P1012/D6.0, Institute of Electrical and Electronics Engineers, Inc., New York
(6 Jun. 1986)

70
 Annex K-2

VERIFICATION RECORD FORM

PROJECT SHEET OF
VERIFICATION RECORD
VR N '

SOFTWARE IDENTIFIER VERSION

METHOD OF VERIFICATION

VERIFICATION COMMENTS

VERIFICATION BY TEST

TEST SPECIFICATION N - VERSION

TEST SCHEDULE N* VERSION

ASSOCIATED. SOFTWARE IDENTIFIER VERSION

HARDWARE SYSTEM USED VERSION

TEST LOCATION

TESTED B r WITNESSED BY

TEST N* PASS/FAIL COMMENT ACTION REQUIRED

VERIFIEO BY OATE APPROVED BY DATE

 Annex V-3

TYPICAL DESIGN REVIEW CHECK-LIST

General

(A) Is the detailed Junctional specification (DFS) or the program design document
(PDD) in conformance with the design documentation guideline and any other
project guidelines?

(1) Does the basic design comply with the established design guidelines?
(2) Does the detailed design comply with the established design guidelines?
(3) Have the standard naming, interface, etc. conventions been followed?
(4) Does the design as documented (in the DFS or PDD) conform to existing
documentation standards?
(5) Assuming the design standards call for structure design, does the detailed
design conform to these precepts?
(6) Do the flow charts, narratives, code diagrams, data flow diagrams, or other
form of the design description agree with the established structural and
documentation standards?
(7) Has the hierarchical structure been defined to the point where the scope of each
identified module or subsystem is narrowed down to a set of specific
operations?
(8) Does the modular decomposition reflect intermodular or intersubsystem
independence?
(9) If not stipulated earlier in the requirements specification (RS), has the
programming language that will be used been identified?
(10) Will global constants be expressed in terms of parameters? For example, will
the design lead to code of the form AREA = 3.14159*R**2 or AREA =
PI*R**2 where 'PI' is defined globally?
(11) Are parameters consistently named and referenced only by name throughout
the program?

(B) Is the translation of the RS into the design correct?

(1) Does the design (DFS or PDD) decompose the overall system requirements
into the hardware and software components which are correct and such that
there are no ambiguities or deficiencies?
(2) Does the design give an accurate, consistent, complete translation of the RS?
(3) Do the software module structures, interfaces and layout conform to the RS?
(4) Are all the support software requirements properly referenced for proper
execution of the design?

72
 (5) Does each subsystem (or module) described in the DFS (or PDD) correspond
to a previously defined element? Is it properly identified?
(6) Do module specifications contain information from which traceability back to
the software requirements can be made and in turn verified?
(7) Have all software requirements been addressed in the design?
(8) Does the design (DFS or PDD) only comply with each and every requirement
in the RS and not introduce extraneous functions?
(9) Is the design internally consistent, complete and consistent with the previous
documents (DFS with RS, DFS with PDD)?

(C) Have all the overall considerations in the design been dealt with?

(1) Have all the necessary preliminary technical design reviews been made? Have
they been recorded and the recommendations incorporated into the design?
(2) Does the DFS (or PDD) meet product assurance requirements? (Allow for
review and audits, detail facilities and procedures for testing.)
(3) Is there agreement between the project manager and user management that the
alternative selected is the best one? Are there unanswered questions concerning
trade-offs?
(4) Is each design structure chart (or equivalent) being reviewed uniquely identi-
fied by name, version and date?
(5) Has the software verification and validation plan been updated to reflect any
changes in the test schedules, test facilities and test requirements resulting from
information developed during the design activity?
(6) Is there agreement from quality assurance representatives that the DFS (or
PDD) is ready for approval?
(7) Is there agreement between the developer and user that the design will fulfil
the user needs?
(8) Has sufficient attention been paid to external (user oriented) requirements?
(9) Is the design expansible?
(10) Will it be necessary to acquire any additional software/hardware resources to
carry out futher development consistent with design? If so, have they been
specified in sufficient detail to permit their acquisition to proceed?
(11) Has each design alternative been clearly identified?
(12) Have all assumptions been identified?
(13) Were the design alternatives objectively compared and objectively selected?

(D) Is the design (DFS or PDD) complete?

(1) Database

(a) Is the refinement of all data storage and timing allocations complete?
(b) Is the detailed definition of the database content, structure and control
complete?

73
 (c) Are the methods of accessing all database elements defined?
(d) Is the database design consistent with the overall design?
(e) Is each data structure referenced within a module defined within the data
dictionary?
(f) Has the definition of database organization and content been completely
generated?
(g) Are the control structures and data linkages consistently defined?
(h) Have all major parameters been adequately described?
(i) Does the database specify the structure of the tables and, to the extent
possible, the format, range and scale of the table entries?
(j) The data structure should be examined for inconsistency or
awkwardness.

(2) Interfaces

(a) Are the identification and specification of parameters, entries and normal
and abnormal exits for all common subroutines complete?
(b) Is the detailed definition and documentation of software internal and
external interfaces completed?
(c) Are all external interfaces well defined (control, support, test)?
(d) Is the design compatible with external interfaces?
(e) Does each subsystem (or module) have well defined outputs and inputs?
Is its function clearly understood?
(f) Have the methods of sequencing and controlling module execution and
interfaces between modules been completely analysed?
(g) Has each interface between the module and the submodules been
completely specified?
(h) Have all human factors been properly addressed?
(i) Has hardware/telecommunications/software compatibility been assured?
(j) Have all the interface requirements between the specified computer
program and other system elements or other computer systems as speci-
fied in the RS been addressed?
(k) Does the planned user interface to the computer program provide for use
of all the computer program functions and control of the computer
program in its operational environment?
(1) Are all functional interfaces with exterior system equipment and
communication links correct?
(m) Are all functional interfaces within the system correct, i.e. soft-
ware/software and software/hardware? (Analyse word format, transfer
rates, etc. for incompatibilities.)
(n) Will the design when implemented interface with other system.elements
or other systems in accordance with the interface specifications?

74
 (o) Have all module interfaces been examined to assure that calling routines
provide the information and environment required by the called routines
in the forms, format and units assumed?
(p) Are the functional interfaces between all modules (subsystems)
complete, correct and named, and have they been referenced only by
name throughout the program?

(3) Control logic

(a) Is the logical interrelation of parts of the DFS (or PDD) clear? Are there
any major flaws?
(b) Is the control flow clear and acceptable?
(c) Is the program control system clearly outlined?
(d) Does the control logic trap infinite loops?
(e) Does the logic prevent self-destruction? (Overflow of a data table should
be diagnosed and reported to the user with appropriate recovery action.)

(4) Input/output (I/O)

(a) Are the specified data rates, accuracies, response time and the like
reflected in the design?
(b) Does the design avoid potential snags caused by defective external
devices?
- (c) Have input data validity checks been.stipulated? Can corrective action be
taken by the module/subsystem that traps the error?
(d) Does the design assume the same form, format and units for input and
output that are stated in the requirements?

(5) Algorithms/models/functions

(a) Is the development and documentation of all required computational

algorithms completed?
(b) Is software explicitly described in terms of either: (i) algorithms/equa-
tions, or (ii) in a narrative form with logic and timing diagrams,
memory/timing and I/O budgets?
(c) Do the algorithms satisfy accuracy, interface, timing and response
requirements?
(d) Have the numerical techniques (e.g. polynomial approximations) been
analysed for accuracy and range? Are they consistent with module
specification and with the end purpose of software?
(e) Has there been a technical review to check the models and algorithms?
Have the results been recorded and the recommendations incorporated
into the preliminary design?

75
 (f) Do the individual algorithms or special techniques really work for the
given problem and data? (This could involve hand calculations for
representative test cases.)
(g) Is the logic of individual algorithms clear and acceptable?

(6) DFS alone

(a) Has a functional description and design been prepared for each module?
(b) Have the specifications of each modular/subsystem (in the DFS) been
specified to a level of detail sufficient to permit the detailed design (lead-
ing to the PDD) to begin?

(7) PDD alone

(a) Does every function identified in the DFS appear either as a module or
is it embedded identifiably within some module in the PDD?
(b) Do all submodules of a given module perform actions identifiable as
subfiinctions of the stated function of the given module? (All those
subfunctions that seem to be missing or extra should be noted as a
discrepancy.)
(c) Are module functions allocated to the lowest level of individually iden-
tifiable computer program components?
(d) Is the design representation from which the software will be coded
prepared in sufficient detail to permit direct translation to a computer
program code and to derive requirements for developmental testing of
the design?
(e) Are the functions allocated to each subsystem further allocated to the
modules comprising the subsystem?
(f) Is the design detailed to the point where each design quantum (e.g. flow
chart symbol, design language statement) clearly indicates the scope of
the code that will implement it?
(g) Has each of the functions that were defined in the module specification
(DFS) been addressed?
(h) Do all decisions within each module test explicit determinable condition
flags, either defined within that module, passed to it as an argument (or
globally), or returned to it by one of its submodules?

(E) Does the design have the desired quality?

(1) Does the design satisfy the attributes of quality: correctness, reliability, effi-
ciency, integrity, usability, maintainability, testability, flexibility, portability,
reusability and interoperability?
(2) Does the design specification reflect the quality criteria required in the RS?

76
(3) Is the design modular?
(4) Is the design free from error, as determined by simulations or walk-throughs.

(F) Does the design have the required performance?

(1) Does the design contain adequate provisions for error corrections, testing and
maintenance?
(2) Are corrective actions for potentially invalid conditions (e.g. dividing by zero,
or the 'noise' remaining after the subtraction of two similar values) reasonable
with regard to minimizing the effect of system or input failures?
(3) Has sufficient attention been paid to abnormal conditions or exceptions and
have they been handled adequately?
(4) Does the design approach for all the functions satisfy the performance require-
ments of the RS?
(5) Is the design compatible with the timing requirements?
(6) Are the test requirements and test tolls sufficient to determine if all the func-
tions meet the requirements of the RS?

(G) Is the design complete and clear?

(1) Are the specific module capabilities defined?

(2) Is the design detailed enough to begin coding?
(3) Is the hierarchy of submodules defined?
(4) Are all the software functions clearly delineated?
(5) Are all interfaces clearly defined?
(6) Are there areas in the design (DFS or PDD) that are not understandable? Are
there any major flaws?
(7) Does a specification exist for each module?
(8) Have all the computer program subsystems, modules and data groups needed
to process user input and generate resulting responses been identified?
(9) Have all the computer program subsystems, modules and data groups critical
to meeting the performance criteria specified in the RS been identified?
(10) Are there superfluous computer program subsystems, modules or data groups
planned in the design (DFS or PDD)?
(11) Does the design have any missing cases, faulty logic, module interface
mismatches, data structure inconsistencies, erroneous I/O consumption and
user interface inadequacies?
(12) Have all the functions listed in the requirements been included in the design?
Have any steps or special cases been overlooked?

(H) Is the design feasible, i. e. achievable within allocated resources, schedules and
acceptable risks?

920
(I) Does the design appropriately meet all the constraints?

(1) Have time critical functions been analysed to verify that the system operations
timing requirements can be met?
(2) Have adequate time and memory space been correctly allocated to individual
software items?
(3) Are data handling limitations of the hardware and resource utilization reason-
able and are they consistent/compatible with interface and accuracy control
specifications, hardware/system specifications and resource allocations?
(4) Have the overall memory required, computation speed and other resource
consumptions been estimated?
(5) Do timing and sizing margins exist?
(6) Has critical timing, if any, been analysed? If a module execution load budget
applies, are the timing estimates consistent with it?
(7) Does the module design imply previously unstipulated limitations or
constraints? Are they acceptable with regard to the user's needs?
(8) Is there a memory budget allocating estimated storage requirements for each
module? Does it have adequate reserve?
(9) Is there an execution load budget by module or subsystem functions? If the
latter, is the grain fine enough to be credible? Does it leave adequate reserve?
(10) Does the design predicate any constraint or limitation beyond those found in
the specification? Are they compatible with the end and systems specifications?
( I I ) Have all constraints specified by the requirements been met?

78
 Annex K-2

NON-CONFORMANCE REPORT FORM

PROJECT
NON - CONFORMANCE REPORT NCR N*

IDENTIFIER OF ITEM AFFECTEO VERSION

DETAILS OF NON-CONFORMANCE

RAISED BY OATE

CAUSE OF NON-CONFORMANCE

CORRECTIVE ACTION TO BE TAKEN

CORRECTIVE ACTION VERIFIED BY

RETEST N ' DATE

NEW ITEM VERSION N -

DISTRIBUTE TO

APPROVED BY DATE
 Annex K-2

NON-CONFORMANCE REPORT LOG FORM

SHEET OF

PROJECT
NON-CONFORMANCE REPORT LOG REVISION OATE

OATE OATE
NCR N* IDENTIFIER BRIEF DESCRIPTION COMMENTS
RAISED CLEARED
 Annex K-2

VALIDATION RECORD FORM

PROJECT SHEET OF
V A L I D A T I O N RECORD
VL N*

SOFTWARE IDENTIFIER VERSION

METHOD OF VALIDATION

VALIDATION COMMENTS

VALIDATION BY TEST

TEST SPECIFICATION N* VERSION

TEST SCHEDULE N - VERSION

ASSOCIATED SOFTWARE IDENTIFIER VERSION

HARDWARE SYSTEM USED VERSION

TEST LOCATION

TESTED BY WITNESSED BY

TEST N*- PASS/FA! COMMENT ACTION REQUIRED

VALIDATED BY DATE APPROVED BY DATE

 Annex V-3

TYPICAL CHECK-LIST
FOR AUDIT OF COMPUTER PROGRAMS

(1) Does the program have a unique reference and status identification?
(2) Does the identifier identify the program, and its version and revision status?
(3) Do all outputs include the identifier?
(4) Is the responsibility for recording the program name and identifier defined?
(5) Is the responsibility for program security defined?
(6) Do the defined methods of program security prevent inadvertent modification
from the verified and approved version?
(7) Is the responsibility for program verification defined?
(8) Has a program verification report been produced?
(9) . Has the verification report been reviewed for scope and adequacy by someone
other than the originator?
(10) Has the program been validated by the user for his application?
(11) Has a validation report been produced?
(12) Is there a set of program documentation which includes: user guides; descrip-
tion of program, including theoretical basis; verification report; and validation
report?
(13) Has the program documentation been subject to review and approval?
(14) Do records of program use contain: listings of the data used for the run; and
associated job control language?
(15) Are changes to the program recorded and verified?
(16) Is documentation revised when program changes are made?
(17) Are changes communicated to all users of the program?
(18) Is the program maintained by an outside organization?
(19) If the program is maintained by an outside organization has the user ascer-
tained that: security procedures are applied; and that the program operates as
the user intended?
(20) Is there a computer program index containing all the programs used by the
user?
(21) Does the computer program index include all applicable versions of the
program?

82
 Annex V-3

TYPICAL CHECK-LIST
FOR CONFIGURATION MANAGEMENT AUDITS

(1) Have configuration reference points been established for: requirement specifi-
cation; system description; system test; and handover?
(2) Do configuration reference points for in-service use include: major enhance-
ments: of operational or application software; upgrading of hardware; and
introduction of new versions of software?
(3) Has a configuration baseline been established?
(4) Are all changes to the configuration baseline subject to configuration control
procedures and recorded?
(5) Have all changes from previous configuration baselines been incorporated into
succeeding baselines?
(6) Have all configuration items been identified and recorded?
(7) Does each configuration item have a complete description which includes:
unique identification; type of software (i.e. master, archive, object, etc.);
nature and identification of machine readable form of software; and applica-
tion details, i.e. where used?
(8) Are all introductions or changes to system configuration controlled by proce-
dure to produce documentation to: specify nature of change; identify all config-
uration items involved or affected; uniquely identify new configuration items;
and identify the effects of the change on system performance?
(9) Do the configuration control procedures specify the level of authority for
approving a change?
(10) Do the procedures require communication of details of the change to all who
need to know?
(11) Do the procedures ensure that all relevant configuration documentation is
updated and cross-referenced?
(12) Do the procedures provide for backward traceability from an approved
changed configuration to that which existed before the change?
(13) Do the configuration procedures provide for tests or retests which verify the
change and retest any aspects which may have been affected by the changes?
(14) Do the procedures provide control of implementation of changes to define: the
responsibility for making changes, progressing approval of changes and main-
taining records of the status of changes?
(15) Do all changes have an individual identification number?
(16) Does configuration documentation conform to the prescribed project standard?
(17) Is there a correspondence between documentation and the subject configuration
items?

83
(18) Does configuration documentation contain adequate cross-references, e.g.
between hardware and software?
(19) Does configuration documentation identify dependencies and interrelationships
between configuration items?
(20) Is the identity and issue status of all configuration documentation established?
(21) Is there a master configuration index which identifies and lists all configuration
items, together with their documentation and respective issue status?
(22) Is there a security copy of the master configuration index?
(23) Are copies of all previous issues of the master configuration index provided?
(24) Do the historical records of the master configuration index contain details of
all changes between issues?
(25) Is a copy of the master configuration index available in the software library
for issue and return control of software and data?
(26) Is the master configuration index structured and organized to reflect the
systems and subsystems it comprises?
(27) Does a system of configuration status accounting exist?
(28) Does the accounting contain the following elements: dates when baselines and
changes were established; date of introduction of each configuration item;
current issue and change status (including pending changes); deficiencies
identified by quality auditing, configuration auditing, design reviews and non-
conformance reports?

84
 Annex V - 3

TYPICAL CHECK-LIST
F O R AUDIT O F SOFTWARE TESTING

(1) Has a test strategy been defined and documented for the project?
(2) Has the strategy been approved and issued?
(3) Does the strategy include: methods of test; testing aids and monitoring
techniques; methods for constructing and maintaining test data; communication
links between designers, testers and users; methods of submitting the tests; test
duration; organization and responsibilities for activities; special testing
arrangements; library control; and document and record filing?
(4) Has a test plan been established and approved?
(5) Does the test plan identify the sequence of testing and the test specifications
and procedures to be used?
(6) Has the adequacy of the specifications and procedures been reviewed and the
documents approved and issued?
(7) Does the test procedure contain the following: definition of what is to be tested;
objectives of the test; test conditions; test data to be used; test date and
duration; methods of test; versions of test aids and tools to be used; test output
expected; analysis of results to be carried out; test records required; and
actions required if test fails?
(8) Have all test facilities and data been identified and provided?
(9) Does the test log provide evidence that the test was conducted according to
procedure?
(10) Do the recorded results verify that the test acceptance criteria have been met?
(11) Are all deviations from the test plan, procedure and specification recorded?
(12) Was the test date recorded?
(13) Did the following activities take place after the test: all non-conformances
recorded and analysed for cause; documentation corrected; repeat tests carried
out; test documentation completed; and test documentation and results
reviewed and approved?
(14) Was a test summary prepared which included the following points: non-
conformances which are to be corrected or are subject to concession; and any
actions to be taken and time-scales for completion?

85
 BIBLIOGRAPHY

The following list of standards, books and papers are included as useful
references for the benefit of the reader who wishes to study them in more detail than
is possible to provide in the Manual.
The subjects of software engineering in general and software quality assurance
in particular are covered and indexed by main text chapter and section titles.

2. SOFTWARE LIFE-CYCLE

EUROPEAN WORKSHOP O N INDUSTRIAL COMPUTER SYSTEMS, TECHNICAL

COMMITTEE 7, Development of Safety Related Software, Position Paper No. 1 (Oct. 1981).

E U R O P E A N WORKSHOP ON INDUSTRIAL COMPUTER SYSTEMS, TECHNICAL

COMMITTEE 7, System Requirements Specification for Safety Related Systems, Position
Paper No. 6 (Jan. 1985).

KEROLA, P., F R E E D M A N , P., " A comparison of life-cycle models", Proc. 5th Int. Conf.
on Software Engineering, IEEE Computer Society Catalog No. 81CH1627-9, Institute of
Electrical and Electronics Engineers, Inc., N e w York (1981) 90.

2.6. Maintenance

BARIKH, G., Techniques of Program and Systems Maintenance, Ethnotech Inc., Lincoln,
NB (1980).

GLASS, R.L., NOISEX, R.A., Software Maintenance Guide Book, Prentice-Hall,

Englewood Cliffs, NJ (1981).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial on

Software Maintenance, IEEE Computer Society Catalog No. EH0201-4, IEEE, N e w York
(1983).

N A T I O N A L B U R E A U OF S T A N D A R D S , Guidance on Software Maintenance, Special

Publication NBS-SP-500-106, NBS, Washington, D C (Dec. 1983).

3. QUALITY ASSURANCE PROGRAMME

(a) Software quality assurance techniques

ALBIN J.L., FERREOL, R . , Collecte et analyse des mesures du logiciel, Tech. Sci. Inf. 1
4 (1982).

ANSI/IEEE, Standard for Software Quality Assurance Plans, Standard 730, Institute of Elec-
trical and Electronics Engineers, Inc., N e w York (1984).

87
ASSOCIATION FRANCAISE DU CONTROLE INDUSTRIEL DE LA QUALITE,
Demonstration de la fiabilite des logiciels, Section quality du logiciel, AFCIQ, Paris.

ASSOCIATION OF COMPUTING MACHINERY, Software Quality Assurance (Proc.

Workshop San Diego, 1978), A C M , New York (Nov. 1978).

BRITISH S T A N D A R D S INSTITUTION, Total Quality Assurance Programme for Nuclear

Installations (Specification) BS-5882, BSI, London.

C A N A D I A N S T A N D A R D S ASSOCIATION, Software Quality Assurance Program. Part 1.

CSA Preliminary Standard Q396.1 (1982).

CANADIAN STANDARDS ASSOCIATION, Software Quality Assurance Program,

Standard Q396.1 (1984).

CENTRE N A T I O N A L D ' E T U D E S SPATIALES, Exigences quality pour le ddveloppement

des logiciels, AQ-059, CNES-Direction des lancjeurs,' Paris.

COOPER, J.D., FISHER, M.J. (Eds), Software Quality Management, Petrocelli Books, N e w
York and Princeton (1979).

Datafrance 8 (1984) 3 7 - 4 1 .

D U N N , R., U L L M A N , R., Quality-Assurance for Computer Software, McGraw Hill,

N e w York (1982).

G U S T A F S O N G.G., KERR, R.J., Some practical experience with a software quality

assurance program Commun. A C M 25 1 (Jan. 1982).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Guide for Soft-

ware Quality Assurance Plans, IEEE Guide 983-1986, IEEE, New York (1986).

IVES, K . A . , Computer Software Quality Assurance, Research Report, Prior Data Sciences
Ltd, Rep. INFO-0201, Atomic Energy Control Board, Ottawa, Canada.

L'industrie face & la qualite du logiciel, C.R. Congrfes Toulouse, 1983.

MEEKEL, J., TROY, R., "Comparative study of standards for software quality assurance
plans", Proc. Software Engineering Standards Workshop (SESAW-III), Computer Society,
Institute of Electrical and Electronics Engineers, Inc., New York (Oct. 1984).

MILITARY S T A N D A R D , Defense System Software Development, Rep. D O D - S T D - 2 1 6 7 ,

United States Department of Defense, Washington, D C (4 Jun. 1985).

MILLER, E., Software quality assurance, Computer 12 8 (Aug. 1979) 7.

NERSON, J.M., Panorama des outils d'amelioration de la quality des logiciels (Atelier
Logiciel No. 40), EDF-DER-Service Informatique' et math6matiques appliqu&s, Clamart
(1983).

NORTH ATLANTIC TREATY ORGANIZATION, Software Quality Control System

Requirements, Allied Quality Assurance Publication No. AQAP-13, N A T O , Brussels.

88
NORTH ATLANTIC TREATY ORGANIZATION, Guide for the Evaluation of Contractors'
Software Quality Control System for Compliance with AQAP-13, Allied Quality Assurance
Publication No. AQAP-14, N A T O , Brussels.

PERRY, W . F . , Effective Methods of EDP Quality Assurance, Q E D Information Sciences,

Inc., Wellesley, M A (1981).

Software Quality Assurance Program Requirements, Military Specification MIL-S-52779A

(1 Aug. 1979).

TAUSWORTHE, R . C . , Standardized Development of Computer Software. Part I. Methods,

Prentice-Hall, Englewood Cliffs, NJ (1977).

TAUSWORTHE, R.C., Standardized Development of Computer Software. Part II.

Standards, Prentice-Hall, Englewood Cliffs, NJ (1979).

(b) Software non-conformances

BASILI, V . R . , PERRICONE, B . T . , Software errors and complexity: An empirical investiga-

tion, Commun. A C M 2 7 1 (Jan. 1984).

BRITISH S T A N D A R D S INSTITUTION, Performance Monitoring of Computer Based

Systems (Code of Practice) BS-6238, BSI, London.

E N D R E S , A . , An analysis of errors and their causes in system programs, IEEE Trans. Soft-
ware Eng. SE-1 2 (Jun. 1975).

G A N N O N , C., "Software error studies", Proc. Nat. Conf. on Software Test and Evaluation,
National Security Industrial Association (Feb. 1983) 1-1.

GLASS, R X . , Persistent software errors, IEEE Trans! Software Eng. SE-7 2 (Mar. 1981)
162-168.

H O W D E N , W . E . , "Errors, design properties and functional program tests", Computer

Program Testing, North-Holland, Amsterdam and N e w York (1981) 115.

SCHNEIDEWIND, N . S . , H O F F M A N , H . M . , An experiment in software data collection and

analysis, IEEE Trans. Software Eng. SE-5 3 (May 1979) 276.

SHEN, V . Y . , Y U , T.J., T H E B A U T , S . M . , P A U L S E N , L.R., Identifying error-prone soft-

ware — An empirical study, IEEE Trans. Software Eng. SE-11 3 (Mar. 1985) 302.

S H O O M A N , M . L . , BOLSKY, M.I., "Types, distribution, and test and correction times for
programming errors", Proc. Int. Conf. on Reliable Software, Computer Society, Institute of
Electrical and Electronics Engineers, Inc. (Apr. 1975) 347.

(c) Software quality attributes

ALBIN, J.L., FERREOL, R., Collecte et analyse des mesures du logiciel, Tech. Sci. Inf. 1
4 (1982).

89
BOEHM, B . W . , et al., Characteristics of Software Quality, North-Holland, Amsterdam and
N e w York (1978).

BRITISH S T A N D A R D S INSTITUTION, Performance Monitoring of Computer Based

Systems (Code of Practice) BS-6238, BSI, London.

C A V E N O , J.P., McCALL, J.A., " A framework for measurement of software quality",

Proc. Workshop on Software Quality and Assurance, Association of Computing Machinery,
N e w York (Nov. 1978) 133.

COOPER, J . D . , FISHER, M . I . , Software Quality Management, Petrocelli Books, N e w York

and Princeton (1979).

LIPOW, H., WHITE, B . B . , BOEHM, B.W., Software Quality Assurance, An Acquisition

Guidebook, Rep. TRW-SS-77-07, TR"W Systems Group, Redondo Beach, CA (Nov. 1977).

McCALL, J.A., " A n introduction to software quality metrics", Software Quality Manage-
ment, Petrocelli Books, N e w York and Princeton (1979) 127.

Software Acquisition Management Guidebook, Software Quality Assurance, Rep.

ESD-TR-77-255 (or NTIS/AD-A047318), National Technical Information Service, Spring-
field, V A (Aug. 1977).

(d) Standards, practices, and conventions

AMERICAN NUCLEAR SOCIETY, Guidelines for Considering User Needs in Computer

Program Development, Rep. ANSI/ANS-10.5-1979, A N S , La Grange Park, IL (1979).

AMERICAN N U C L E A R SOCIETY, Standard Criteria for the Application of Programmable

Digital Computer Systems in Safety Systems of Nuclear Power Generating Stations, Rep.
A N S I / A N S - 4 . 3 . 2 , Proposed American National Standard, Draft 8, A N S , La Grange Park, IL
(3 Jun. 1980).

AMERICAN N U C L E A R SOCIETY, Guidelines for the Verification and Validation of

Scientific and Engineering Computer Programs, Draft Standard, Rep. A N S - 1 0 . 4 , A N S ,
La Grange Park, IL (Aug. 1983).

ANSI/IEEE, Standard for Software Quality Assurance Plans, Standard 730, Institute of
Electrical and Electronics Engineers, Inc., New York (1984).

ANSI/IEEE, Standard for Software Unit Testing, Standard P1008-1987, Institute of Electrical
and Electronics Engineers, Inc., N e w York (1987).

ASSOCIATED TECHNOLOGY C O M P A N Y , A Fortran Coding Standard, ATC, Estill

Springs, T N (30 May 1983).

ASSOCIATED TECHNOLOGY C O M P A N Y , A Product Level Software Documentation

Guide, ATC, Estill Springs, T N (Feb. 1985).

BRITISH S T A N D A R D S INSTITUTION, Configuration Management of Computer Based

Systems (Code of Practice) BS-6488, BSI, London.

90
BRITISH S T A N D A R D S INSTITUTION, Control of the Operation of a Computer (Code of
Practice) BS-6650, BSI, London.

BRITISH S T A N D A R D S INSTITUTION, Design Structure Diagrams for Use in Program

Design and Other Logical Applications (Guide) BS-6224, BSI, London.

BRITISH S T A N D A R D S INSTITUTION, Documentation of Computer Based Systems (Code

of Practice) BS-5515, BSI, London.

BRITISH S T A N D A R D S INSTITUTION, Performance Monitoring of Computer Based

Systems (Code of Practice) BS-6238, BSI, London;

BRITISH S T A N D A R D S INSTITUTION, Specifying User Requirements for a Computer

Based System (Guide) BS-6719, BSI, London.

BRITISH S T A N D A R D S INSTITUTION, Testing of Computer Based Systems (Code of

Practice) BS-5887, BSI, London.

BRITISH S T A N D A R D S INSTITUTION, Total Quality Assurance Programme for Nuclear

Installations (Specification) BS-5882, BSI, London.

C A N A D I A N S T A N D A R D S ASSOCIATION, Software Quality Assurance Program. Part I.

CSA Preliminary Standard Q396.1 (1982).

C H E C K L A N D , P., Systems Thinking, Systems Practice, Wiley, N e w York.

COATS, R.B., Software Engineering for Small Computers, Arnold, London.

DOGGETT, R.B., CAREY, Z . E . , WILBURN, N . P . , Guidelines — Software Configuration

Management, Rep. HEDL-TC-2263, Westinghouse-Hanford Company, Richland, WA
(Jan. 1983).

F O R E M A N , J.J., Implementing software standards, IEEE Computer 13 6 (Jun. 1980) 67.

GLASS, R . L . , "Standards for standards writers", Software Engineering Standards Applica-

tions (SESAW-I) (Proc. Workshop San Francisco, 1981), IEEE Computer Society Catalog
No. 81CH1633-7, Institute of Electrical and Electronics Engineers, Inc., N e w York (Aug.
1981) 144.

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, I N C . , Guide for Soft-

ware Configuration Management, IEEE Computer Society Approved Draft Guide P1042,
IEEE, N e w York (1986).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, I N C . , IEEE Guide to

Software Requirements Specification, IEEE Standard 830-1984, IEEE, N e w York (1984).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, I N C . , IEEE Standard

for Software Configuration Management Plan, IEEE Standard 828-1983, IEEE, N e w York
(1983).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, I N C . , Standard for

Software Engineering Standards Taxonomy, IEEE Computer Society Approved Draft
Standard P1002, IEEE, N e w York (1986).

91
INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., IEEE Standard
for Software Test Documentation, IEEE Standard 829-1983, IEEE, N e w York (1983).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Standard for

Software Verification and Validation Plans, IEEE Computer Society Approved Draft Standard
P1012, IEEE, N e w York (1986).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Standard

Glossary of Software Engineering Terms, IEEE Standard No. 729, IEEE, N e w York (1983).

INTERNATIONAL ELECTROTECHNICAL COMMISSION, Standard 880, Software for

Computers in the Safety Systems of Nuclear Power Stations, Geneva (1986).

JENSON, R.W., TONIES, C.C., Software Engineering, Prentice-Hall, Englewood

Cliffs, NJ.

NORTH ATLANTIC TREATY ORGANIZATION, Software Quality Control System

Requirements, Allied Quality Assurance Publication No. AQAP-13, N A T O , Brussels.

NORTH ATLANTIC TREATY ORGANIZATION, Guide for the Evaluation of Contractors'

Software Quality Control System for Compliance with AQAP-13, Allied Quality Assurance
Publication No. AQAP-14, N A T O , Brussels.

POSTON, R . M . , Determining a complete set of software development standards, IEEE,

Software 1 3 (Jul. 1984) 87.

POSTON, R . M . , Software Standards, IEEE Software 2 1 (Jan. 1985) 83.

PRESSMAN, R.S.j Software Engineering: A Practitioners Approach, McGraw Hill,

N e w York.

SOMMERVILLE, I., Software Engineering, Addison-Wesley, Reading, MA.

TAUSWORTHE, R.C., Standardized Development of Computer Software. Part I. Methods,

Prentice-Hall, Englewood Cliffs, NJ (1977).

TAUSWORTHE, R.C., Standardized Development of. Computer Software. Part II.

Standards, Prentice-Hall, Englewood Cliffs, NJ (1979).

W I L B U R N , N . P . , Guidelines — Software Requirements Specification (SRS) Document

Preparation, Rep. HEDL-TC-2159, Westinghouse-Hanford Company, Richland, WA
(Aug. 1982).

WILBURN, N . P . , Standards and Guidelines Applicable to Scientific Software Life Cycle,

Rep. HEDL-TC-2314, Westinghouse T Hanford Company, Richland, W A (Jan. 1983).

4. ORGANIZATION

(a) General

BRUCE, P., PEDERSON, S . M . , The Software Development Project — Planning and

Management, Wiley, New York (1982).

92
D E MARCO, T . , Controlling Software Projects — Management Measurement and Estima-
tion, Yourdon Press, N e w York (1982).

FIFE, D . W . , Computer Science and Technology: Computer Software Management, A Primer

for Project Management and Quality Control, National Bureau of Standards Special Publica-
tion NBS-SP-500-11, NBS, Washington, D C (Jul. 1977).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial:

Software Management, IEEE Computer Society Catalog No. EH0146-1, IEEE, N e w York
(1979).

TAUSWORTHE, R . C . , Standardized Development of Computer Software. Part I. Methods,

Prentice-Hall, Englewood Cliffs, NJ (1977).

TAUSWORTHE, R.C., Standardized Development of Computer Software. Part II.

Standards, Prentice-Hall, Englewood Cliffs, NJ (1979).

Y O U R D O N , E., Managing the Structured Techniques, Yourdon Press, N e w York (1979).

(b) Training

ACKERMAN, A.F., ACKERMAN, A.S., E B E N A U , R.G., " A software inspections

training program", Computer Software and Applications (COMPSAC-82) (Proc. Conf.
N e w York, 1982), IEEE Computer Society Catalog No. 82CH1810-1, Institute of Electrical
and Electronics Engineers, Inc., N e w York (Nov. 1982) 443.

BARNES, K., Doing it yourself — A blueprint for training, PC Mag.(6 Aug. 1985) 147.

CAIN, J.T., L A N G D O N , G . G . , V A R A N A S I , M . R . , "The IEEE computer society model

program", Computer Science and Engineering, Computer 17 4 (Apr. 1984) 8.

INSTITUTE OF ELECTRICAL AND ELECTRONICS ENGINEERS, INC., Model

Program in Computer Science and Engineering, IEEE Computer Society, IEEE, N e w York
(1984).

M A L S B U R Y , S., "Educational support for the standards process", Software Engineering

Standards Application (SESAW-II) (Proc. Workshop, N e w York, 1983), IEEE Computer
Society Catalog No. 83CH1884-6, Institute of Electrical and Electronics Engineers, Inc.,
N e w York (May 1983) 119.

McGILL, J.P., The software engineering shortage, IEEE Trans. Software Eng. SE-10 1
(Jan. 1984) 42.

MILLS, H . D . , "Software engineering education", Proc. IEEE '68, N o . 9, Institute of

Electrical and Electronics Engineers, Inc., N e w York (1980) 1158.

RASKIN, R., Individual training: A matter of style, PC Mag. (6 Aug. 1985) 121.

WASSERMAN, A.I., FREEDMAN, P., "Software engineering education: Status and

prospects", Tutorial on Software Design Techniques, 3rd edn, IEEE Computer Society
Catalog No. EH0161-0, Institute of Electrical and Electronics Engineers, Inc., N e w York
(1980) 445.

93
 5. SOFTWARE DOCUMENT, CONFIGURATION,
MEDIA AND SERVICES CONTROL

5.1.1. Software document control

A M E R I C A N N U C L E A R SOCIETY, Guidelines for the Documentation of Digital Computer

Programs, Rep. A N S I / A N S 10.3-1986, A N S , La Grange Park, IL (1986).

ASSOCIATED TECHNOLOGY C O M P A N Y , A Product Level Software Documentation

Guide, ATC, Estill Springs, T N (Feb. 1985).

BRITISH S T A N D A R D S INSTITUTION, Documentation of Computer Based Systems (Code

of Practice) BS-5515, BSI, London.

E U R O P E A N WORKSHOP O N INDUSTRIAL COMPUTER SYSTEMS, TECHNICAL

COMMITTEE 7, Guidelines for Documentation of Safety Related Computer Systems,
Position paper No. 4 (Oct. 1984).

INTERNATIONAL ELECTROTECHNICAL COMMISSION, Standard 880, Software for

Computers in the Safety Systems of Nuclear Power Stations, Geneva (1986).

NATIONAL BUREAU OF STANDARDS, Proceedings of the NBS/FIPS Software

Documentation Workshop, Special Publication NBS-SP-500-04, NBS, Washington, DC
(Oct. 1982).

N E U M A N N , A.J., Management Guide For Software Documentation, National Bureau of

Standards Special Publication NBS-SP-500-87, N B C , Washington (Jan. 1982).

UNITED STATES D E P A R T M E N T OF COMMERCE, Guidelines for Documentation of

Computer Programs and Automated Data Systems, Federal Information Processing Standard
Publication 38, National Bureau of Standards, Washington, D C (15 Feb. 1976).

5.2. Configuration management

BERSOFF, E . R . H . , H E N D E R S O N , V . , SIEGEL, S . E . , Software Configuration Manage-

ment — An Investment in Product Integrity, Prentice-Hall, Englewood Cliffs, NJ (1980).

BERSOFF, E . R . H . , H E N D E R S O N , V . , SIEGEL, S . E . , Software configuration manage-

ment: A tutorial, IEEE Computer 11 1 (Jan. 1979) 6.

BRITISH S T A N D A R D S INSTITUTION, Configuration Management of Computer Based

Systems (Code of Practice) BS-6488, BSI, London.

DOGGETT, R.B., CAREY, Z . E . , WILBURN, N . P . , Guidelines - Software Configuration

Management, Rep. HEDL-TC-2263, Westinghouse-Hanford Company, Richland, WA
(Jan. 1983).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Guide for

Software Configuration Management, IEEE Computer Society Approved Draft Guide P1042,
IEEE, New York (1986).

94
INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Standard for
Software Configuration Management Plan, IEEE Standard 828-1983, IEEE, N e w York
(1983).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial

Software Configuration Management, IEEE Computer Society Catalog No. EH0169-3, IEEE,
N e w York (1980).

5.3. Media and services control

ASSOCIATION FRANCAISE D E NORMALISATION, Recueil des normes frangaises infor-

matiques. Tome 3. Supports magnetiques, A F N O R , Paris-La Defense (1985).

N A T I O N A L B U R E A U OF S T A N D A R D S , Guidelines for Automatic Data Processing Physi-

cal Security and Risk Management, Federal Information Processing Standards Publication 31,
NBS, Washington, D C (Jun. 1974).

N A T I O N A L B U R E A U OF S T A N D A R D S , Guidelines for Automatic Data Processing Risk

Analysis, Federal Information Processing Standards Publication 65, NBS, Washington, D C
(Aug. 1979).

N A T I O N A L B U R E A U OF S T A N D A R D S , Guidelines for Security of Computer Applica-

tions, Federal Information Processing Standards Publication 73, NBS, Washington, D C
(Jun. 1980).

R U D E R , B., M A D D E N , J . D . , An Analysis of Computer Security Safeguards for Detecting

and Preventing Intentional Computer Misuse, National Bureau of Standards Special Publica-
tion NBS-SP-500-25, NBS, Washington, D C (Jan. 1978).

SHANKAR, K.S., The total computer security problem: An overview, IEEE Computer 10
6 (Jun. 1977) 50.

STEINAUER, D . D . , Security of Personal Computer Systems: A Management Guide,

National Bureau of Standards Special Publication NBS-SP-500-120, NBS, Washington, D C
(Jan. 1985).

6. D E S I G N CONTROL

AMERICAN N U C L E A R SOCIETY, Guidelines for Considering User Needs in Computer

Program Development, Rep. ANSI/ANS-10.5-1979, A N S , La Grange Park, IL (1979).

BARIKH, G., Techniques of Program and Systems Maintenance, Ethnotech Inc., Lincoln,
NB (1980).

D E MARCO, T . , Structured Analysis and System Specification, Yourdon Press, New York
(1978).

ENOS, J.L., V A N TILBURG, R . L . , Tutorial series 5: Software design, Computer 14 2

(Feb. 1961) 61.

95
EUROPEAN WORKSHOP O N INDUSTRIAL COMPUTER SYSTEMS, TECHNICAL
COMMITTEE 7, Development of Safety Related Software, Position Paper No. 1 (Oct. 1981).

G A N E , C., SARSON, T., Structured Systems Analysis: Tools and Techniques, McDonald
Douglas Automation Company, St. Louis, M O (1977).

GLASS, R.L., NOISEX, R.A., Software Maintenance Guide Book, Prentice-Hall,

Englewood Cliffs, NJ (1981).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial on

Software Design Techniques, IEEE Computer Society Catalog N o . EH1061-0, IEEE,
N e w York (1980).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial on

Software Maintenance, IEEE Computer Society Catalog No. EH0201-4, IEEE, N e w York
(1983).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial

JSP & JSD: The Jackson Approach to Software Development, IEEE Computer Society
Catalog No. EH0206-3, IEEE, N e w York (1983).

INTERNATIONAL ELECTROTECHNICAL COMMISSION, Standard 880, Software for

Computers in the Safety Systems of Nuclear Power Stations, Geneva (1986).

JACKSON, M . A . , Principles of Program Design, Academic Press, N e w York (1975).

KERNIGHAN, B . W . , PLAUGER, P.J., The Elements of Programming Style, McGraw Hill,

N e w York (1978).

WILBURN, N . P . , Guidelines — Software Requirements Specification (SRS) Document

Preparation, Rep. HEDL-TC-2159, Westinghouse-Hanford Company, Richland, WA
(Aug. 1982).

Y O U R D O N , E., CONSTANTINE, L.L., Structure Design: Fundamentals of a Discipline of

Computer Program and Systems Design, Yourdon Press, New York (1978).

6.2. Design verification

ADRION, W . R . , BRANSTAD, M . A . , CHERNIAVSKY, J.C., Validation, Verification and

Testing of Computer Software, National Bureau of Standards Special Publication
NBS-SP-500-75, NBS, Washington, D C (Feb. 1981).

AMERICAN N U C L E A R SOCIETY, Guidelines for the Verification and Validation of Scien-

tific and Engineering Computer Programs, Draft Standard, Rep. A N S 10.4, A N S , La Grange
Park, IL (Aug. 1983).

DEUTSCH, M . S . , Software Verification and Validation — Realistic Approaches, Prentice-

Hall, Englewood Cliffs, NJ (1982).

EUROPEAN WORKSHOP ON INDUSTRIAL COMPUTER SYSTEMS, TECHNICAL

COMMITTEE 7, Guidelines for Verification and Validation of Safety Related Software,
Position Paper No. 3 (Jun. 1983).

96
INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, I N C . , Draft Standard
for Software Verification and Validation Plans, IEEE Computer Society Draft Standard
P 1 0 1 2 / D 4 . 0 , IEEE, N e w York (31 Jul. 1985).

POWELL, P.B. (Ed.), Planning for Software Validation, Verification, and Testing, National
Bureau of Standards Special Publication NBS-SP-500-98, NBS, Washington, DC
(Nov. 1982).

POWELL, P.B., (Ed.), Software Validation, Verification and Testing, Technique and Tool
Reference Guide, National Bureau of Standards Special Publication NBS-SP-500-93, NBS,
Washington, D C (Sep. 1982).

WILBURN, N.P., Guidelines - Software Verification, Rep. HEDL-TC-2425,

Westinghouse-Hanford Company, Richland, W A (Aug. 1983).

6.4. Tools, techniques and methods

BROWN, J.R., "Programming practices for increased software quality" Software Quality
Management, Petrocelli Books, N e w York and Princeton (1979) 197.

EUROPEAN WORKSHOP O N I N D U S T R I A L COMPUTER SYSTEMS, TECHNICAL

COMMITTEE 7, Techniques for Verification and Validation of Safety Related Software,
Position Paper No. 5 (Jan. 1985).

FISHER, C . F . , "Software quality assurance tools: Recent experience and future require-
ments", Software Quality Assurance (Proc. Workshop San Diego, 1978), Association of
Computing Machinery, N e w York (1978) 116.

HOUGHTON, R . C . , Jr., Features of Software Development Tools, National Bureau of

Standards Special Publication 500-74, NBS, Washington, D C (Feb. 1981).

H O U G H T O N , R . C . , Jr., Software Development Tools, National Bureau of Standards Special

Publication NBS-SP-500-88, NBS, Washington, D C (Mar. 1982).

H O U G H T O N , R . C . , Jr., Software development tools: A profile, Computer 16 5 (May 1983)

63.

HOUGHTON, R . C . , Jr., OAKLEY, K . A . , NBS Software Tools Data Base, National Bureau
of Standards Publication NBS-IR2159, NBS, Washington, D C (Oct. 1980).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial:

Automated Tools for Software Engineering, IEEE Computer Society Catalog No. EH0150-3,
IEEE, N e w York (1979).

N A T I O N A L B U R E A U OF S T A N D A R D S , Proc. NBS/IEEE/ACM Software Tool Fair,

Special Publication NBS-SP-500-80, NBS, Washington, D C (Oct. 1981).

NERSON, J.M., Outils d'aide a l'amdlioration de la qualite des logiciels, Application a

l'editeur structurel, Ecole nationale sup^rieure des telecommunications, N o . 13 (1983).

97
NERSON, J.M., Panorama des outils d'amelioration de la qualite des logiciels (Atelier
Logiciel No. 40), EDF-DER-Service Informatique et mathematiques appliquees, Clamart
(1983).

OSTERWEIL, L.J., "TOOLPACK — An experimental software development environment

research project", Proc. 6th Int. Conf. on Software Engineering, IEEE Computer Society
Catalog No. 82CHI795-4, Institute of Electrical and Electronics Engineers, Inc., N e w York
(Sep. 1982) 166.

POWELL, P.B. (Ed.), Software Validation, Verification and Testing, Technique and Tool
Reference Guide, National Bureau of Standards Special Publication NBS-SP-500-93, N B S ;
Washington, D C (Sep. 1982).

REIFER, D.J., "Software quality assurance tools and techniques", Software Quality
Management, Petrocelli Books, N e w York and Princeton (1979) 209.

RIDDLE, W.E., FAIRLEY, R.E., Software Development Tools, Springer-Verlag,

N e w York (1980).

SOFTAIR, A Conference on Software Development Tools, Techniques, and Alternatives,

IEEE Computer Society Catalog No. 83CH1919-0, Institute of Electrical and Electronics
Engineers, Inc., N e w York (Jul. 1983).

7. PROCUREMENT CONTROL

CENTRE N A T I O N A L D ' E T U D E S SPATIALES, Exigences quality applicables aux fourni-

tures des logiciels standard, AQ060, CNES-Direction des langeurs, Paris.

LIPOW, H., WHITE, B . B . , BOEHM, B . W . , Software Quality Assurance, An Acquisition

Guidebook, Rep. TRW-SS-77-07, TRW Systems Group, Redondo Beach, CA (Nov. 1977).

NORTH ATLANTIC TREATY ORGANIZATION, Guide for the Evaluation of Contractors'

Software Quality Control System for Compliance with AQAP-13, Allied Quality Assurance
Publication No. AQAP-14, NATO,. Brussels.

Software Acquisition Management Guidebook, Software Quality Assurance, Rep.

ESD-TR-77-255 (or NTIS/AD-A047318), National Technical Information Service, Spring-
field, V A (Aug. 1977).

. 8. TESTING .

A D R I O N , W . R . , B R A N S T A D , M . A . , CHERNIAVSKY, J.C., Validation, Verification

and Testing of Computer Software, National Bureau of Standards Special Publication
NBS-SP-500-75, NBS, Washington, D C (Feb. 1981).

A M E R I C A N N U C L E A R SOCIETY, Guidelines for the Verification and Validation of

Scientific and Engineering Computer Programs, Draft Standard, Rep. ANSI/ANS-10.4,
A N S , La Grange Park, IL (1987).

98
ANSI/IEEE, Standard for Software Unit Testing, Standard P1008-1987, Institute of Electrical
and Electronics Engineers, Inc., N e w York (1987).

BEIZER, B., Software System Testing and Quality Assurance, Van Nostrand-Reinhold,
N e w York (1984).

BEIZER, B., Software Testing Techniques, Van Nostrand-Reinhold, N e w York (1983).

B R A N S T A D , M . A . , (NBS), CHERNIAVSKY, J.C., (NBS), A D R I O N , W . R . , (NSF),

Validation, verification and testing for the individual programmer, Computer 13 12
(Dec. 1980) 24.

BRITISH S T A N D A R D S INSTITUTION, Testing of Computer Based Systems (Code of

Practice) BS-5887, BSI, London.

Computer Program Testing, North-Holland, Amsterdam and N e w York (1981).

GLASS, R.L., Software Reliability Guide Book, Prentice-Hall, Englewood Cliffs, NJ (1979).

INFOTECH, State of The A r t . Report — Software Testing, Vol.1, Analysis and

Bibliography, Infotech International Limited, Berkshire, UK (1979).

INFOTECH, State of the Art Report — Software Testing, Vol. 2, Invited Papers, Infotech
International Limited, Berkshire, UK (1979).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Standard for

Software Test Documentation, IEEE Standard 829-1983, IEEE, N e w York (1983).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial:

Software Testing and Validation Techniques, IEEE Computer Society Catalog No. EH0138-8,
IEEE, N e w York (1978).

INTERNATIONAL ELECTROTECHNICAL COMMISSION, Standard 880, Software for

Computers in the Safety Systems of Nuclear Power Stations, Geneva (1986).

McCABE, T.J., Structured Testing, IEEE Computer Society Catalog No. EH0200-6,
Institute of Electrical and Electronics Engineers, Inc., N e w York (1982) 90.

MYERS, G.J., Software Reliability Principles and Practices, Wiley, N e w York (1976).

MYERS, G.J., The Art of Software Testing, Wiley, N e w York (1979).

POWELL, P.B. (Ed.), Planning for Software Validation, Verification, and Testing, National
Bureau of Standards Special Publication NBS-SP-500-98, NBS, Washington, DC
(Nov. 1982).

POWELL, P.B. (Ed.), Software Validation, Verification and Testing, Technique and Tool
Reference Guide, National Bureau of Standards Special Publication NBS-SP-500-93, NBS,
Washington, DC (Sep. 1982). .

99
 10. CORRECTIVE ACTION

BRITISH S T A N D A R D S INSTITUTION, Performance Monitoring of Computer Based

Systems (Code of Practice) BS-6238, BSI, London.

FAG A N , M . E . , "Design and code inspection to reduce errors in program development",

Tutorial on Structured Programming: Integrated Practices, IEEE Computer Society Catalog
No. EH0178-4, Institute of Electrical and Electronics Engineers, Inc., N e w York (1980) 216.

FOSTER, K . A . , Error sensitive test case analysis (ESTCA), IEEE Trans. Software Eng.
SE-6 3 (May 1980) 258.

G A N N O N , C., Error detection using path testing and static analysis, Computer 12 8
(Aug. 1979) 26.

H O W D E N , W . E . , "Errors, design properties and functional program tests", Computer

Program Testing, North-Holland, Amsterdam and N e w York (1981) 115.

H U A N G , J.C., "Error detection through program testing", Current Trends in Programming

Methodology, Vol. 2, Program Validation, Prentice-Hall, Englewood Cliffs, NJ (1977) 16.

OSTERWEIL, L.J., FOSDICK, L . D . , ' ' D A V E — A validation error detection and documen-
tation system for Fortran programs", Tutorial: Software Testing and Validation Techniques,
IEEE Catalog No. EH0138-8, Institute of Electrical and Electronics Engineers, Inc.,
N e w York (1978) 473.

SHEN, V . Y . , Y U , T.J., T H E B A U T , S.M., P A U L S E N , L.R., Identifying error-prone

software — An empirical study, IEEE Trans. Software Eng. SE-11 3 (Mar. 1985) 302.

11. RECORDS

BARIKH, G., Techniques of Program and Systems Maintenance, Ethnotech Inc., Lincoln,
N B (1980).

BRITISH S T A N D A R D S INSTITUTION, Documentation of Computer Based Systems (Code

of Practice) BS-5515, BSI, London.

GLASS, R.L., NOISEX, R.A., Software Maintenance Guide Book, Prentice-Hall,

Englewood Cliffs, NJ (1981).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Standard for

Measurements to Produce Reliable Software, IEEE Computer Society Draft Standard P982,
IEEE, N e w York (1986).

INSTITUTE OF ELECTRICAL A N D ELECTRONICS ENGINEERS, INC., Tutorial on

Software Maintenance, IEEE Computer Society Catalog No. EH0201-4, IEEE, N e w York
(1983).

100
N A T I O N A L B U R E A U OF S T A N D A R D S , Guidelines for Automatic Data Processing
Physical Security and Risk Management, Federal Information Processing Standards Publica-
tion 31, NBS, Washington, D C (Jun. 1974).

N A T I O N A L B U R E A U OF S T A N D A R D S , Guidance on Software Maintenance, Special

Publication NBS-SP-500-106, NBS, Washington, D C (Dec. 1983).

12. AUDITING

F R E E D M A N , D . P . , WEINBERG, G . M . , Ethnotech Review Handbook, Ethnotech, Inc.,

Lincoln, NB (1977, 1979).

WILBURN, N.P., Guidelines for Technical Reviews of Software Products, Rep.

HEDL-TC-2132, Westinghouse-Hanford Company, Richland, W A (Oct. 1982).

WILBURN, N.P., Guidelines — Software Verification, Rep. HEDL-TC-2425,

Westinghouse-Hanford Company, Richland, W A (Aug. 1983).

Y O U R D O N , E., Structured Walk-throughs, 2nd edn, Yourdon Press, N e w York (1978).

101
 LIST OF PARTICIPANTS

MEMBERS OF THE ADVISORY GROUP

AND EXPERTS PARTICIPATING IN THE
ADVISORY GROUP MEETINGS (AGM)
AND CONSULTANTS MEETING (CM)

I AGM 17-21 June 1985

II CM 30 September to 4 October 1985
IH AGM 2-6 December 1985
IV AGM 26-30 May 1986
V AGM 6-10 April 1987

Blot, T. Direction de l'equipement, Departement informatique,

(Ill, IV, V) Electricity de France,
Tour Atlantique, Cedex 6, F-92080 Paris-La Defense, France

H o m e , G.S. Nuclear Installations Inspectorate,

(I, III, IV, V) Baynards House,
1 Chepstow Place, Westbourne Grove,
London W 2 4TF, United Kingdom

Ielasi, R. Comitato Nazionale per la Ricerca e lo Sviluppo

(Ill, IV, V) dell'Energia Nucleare e delle Energie Alternative (ENEA),
Via Vitalino Brancati 48,
1-00144 Rome, Italy

Krcek, V. Institute for Rationalization of Power Enterprises,

(HI, V) Orgrez Brno,
P.O. Box 251, CS-65697 Brno Radlas 18, Czechoslovakia

Leviston, J.T. Quality System Department,

(I, II, III, IV, V) National Nuclear Corporation Limited,
Risley, Warrington,
Cheshire W A 3 6BZ, United Kingdom

Lorenzelli, G. Ente Nazionale per l'Energia Elettrica (ENEL),

(V) Viale Regina Margherita 137,
1-00198 Rome, Italy

Massicotte, M. Atomic Energy Control Board,

(I, III, IV, V) P.O. Box 1046,
Ottawa, Ontario K I P 5S9, Canada

103
Miani, G. Ente Nazionale per l'Energia Elettrica (ENEL),
(Ill, IV) Viale Regina Margherita 135,
1-00198 Rome, Italy

Raisic, N. Division of Nuclear Power,

(Scientific Secretary International Atomic Energy Agency,
for QA) P.O. Box 100, A-1400 Vienna, Austria
d)

Roggenbauer, H. Institut fur Reaktorsicherheit,

(I, III, IV, V) Osterreichisches Forschungszentrum in Seibersdorf,
Lenaugasse 10, A-1082 Vienna, Austria

Wald, F. Division of Nuclear Power,

(Scientific Secretary) International Atomic Energy Agency,
(I, II, III, IV, V) P.O. Box 100, A-1400 Vienna, Austria

Wilburn, N . P . Analysis and Integration

(I, III, V) Westinghouse-Hanford Company,
P.O. Box 1970, MSC 86,
Richland, Washington 99352, United States of America

104
 HOW TO ORDER IAEA PUBLICATIONS
An exclusive sales agent for IAEA publications, to whom all orders
and inquiries should be addressed, has been appointed
in the following country:

U N I T E D STATES OF A M E R I C A U N I P U B , 4611-F Assembly Drive, Lanham, M D 20706-4391

In the following countries IAEA publications may be purchased from the

sales agents or booksellers listed or through
major local booksellers. Payment can be made in local
currency or with UNESCO coupons.

ARGENTINA Comision Nacional de Energfa Atomica, Avenida del Libertador 8250,

RA-1429 Buenos Aires
AUSTRALIA Hunter Publications, 58 A Gipps Street, Collingwood, Victoria 3066
BELGIUM Service Courrier UNESCO, 202, Avenue du Roi,B-1060 Brussels
CHILE Comisi6n Chilena de Energi'a Nuclear, Venta de Publicaciones,
Amunategui 95, Casilla 188-D, Santiago
CHINA I A E A Publications in Chinese:
China Nuclear Energy Industry Corporation, Translation Section,
P.O. Box 2103, Beijing
I A E A Publications other than in Chinese:
China National Publications Import & Export Corporation,
Deutsche Abteilung, P.O. Box 88, Beijing
CZECHOSLOVAKIA S.N.T.L., Mikulandska 4, CS-11686 Prague 1
Alfa, Publishers, Hurbanovo namestie 3.CS-815 89 Bratislava
FRANCE Office International de Documentation et Librairie,48, rue Gay-Lussac,
F-75240 Paris Cedex 05
HUNGARY Kultura, Hungarian Foreign Trading Company,
P.O. Box 149, H-1389 Budapest 62
INDIA O x f o r d Book and Stationery Co., 17, Park Street, Calcutta-700 016
O x f o r d Book and Stationery Co.,Scindia House, New Delhi-110 001
ISRAEL Heiliger & Co. Ltd.
23 Keren Hayesod Street, Jerusalem 94188
ITALY Libreria Scientifica, D o t t . Lucio de Biasio "aeiou",
Via Meravigli 16, 1-20123 Milan
JAPAN Maruzen Company, L t d , P.O. Box 5050,100-31 T o k y o International
PAKISTAN Mirza Book Agency, 65, Shahrah Quaid-e-Azam, P.O. Box 729, Lahore 3
POLAND Ars;Polona-Ruch, Centrala Handlu Zagranicznego,
Krakowskie. Przedmiescie 7, PL : 00-068 Warsaw '.
ROMANIA l l e x i m . P O . Box 136-137, Bucharest
SOUTH A F R I C A Van Schaik Bookstore (Pty) Ltd, P.O. B o x 7 2 4 , Pretoria 0001
SPAIN Diaz de Santos, Lagasca 95, E-28006 Madrid .
Diaz de Santos, Balmes 417, E-08022 Barcelona
SWEDEN AB Fritzes Kungk Hovbokhandel, Fredsgatan 2, P.O. Box 16356,
S-103 27 Stockholm;
UNITED KINGDOM Her Majesty's Stationery Office, Publications,Centre, Agency Section,
51 Nine Elms Lane, London SW8 5 D R
USSR Mezhdunarodnaya Kniga, Smolenskaya-Sennaya 32-34, Moscow G-200
YUGOSLAVIA Jugoslovenska Knjiga, Terazije 27, P.O. Box 36, YU-11001 Belgrade 1

Orders from countries where sales agents have not yet been appointed and
requests for information should be addressed directly to:
Division of Publications •
International Atomic Energy Agency
Wagramerstrasse 5, P.O. Box 100, A-1400 Vienna, Austria
INTERNATIONAL SUBJECT GROUP: V
ATOMIC ENERGY AGENCY Reactors and Nuclear Power/Reactor T e c h n o l o g y
V I E N N A , 1988