THE McGRAW-HILL

INTERNATIONAL
SERIES IN SOFTWARE
∎ ENGINEERING

Software Quality Assurance (SQA) is now a major factor in the

Software
success of any software development project - a fact reflected in the
content of today's undergraduate and postgraduate courses . This
book is a truly student-oriented introduction to the subject, applying
SQA to all the key stages of development - requirements analysis,
project planning, system design, detailed design, programming and
Quality
finally testing . Pitfalls and remedies in the application of SQA are
illustrated by anecdotes, often taken from the author's own
experience, and a practical case study shows how to implement the
Quality Assurance Process from start to finish .

Special features
Assurance
• Self-assessment questions at the end of each chapter to
measure understanding A STUDENT
• A chapter devoted to ISO 9001
13 How to improve an existing QA system
M QA and the human-computer interface
INTRODUCTION
The author
Darrel Ince is the Series Editor of the McGraw-Hill International
Software Engineering and International Software Quality Series .
He is the author of 18 books including An Introduction to Software
Quality Assurance and its Implementation, over 90 papers and
has edited three collected works on the subject of software
engineering . He writes frequently in the national press on computing
matters and holds a chair in computing at the Open University .

ISBN 0-07-709096 - 9

DARRLEIN(E
THE McGRAW-HILL
INTERNATIONAL
SERIES IN SOFTWARE
McGRAW-HILL BOOK COMPANY 9 780077 090968 ® ENGINEERING U
THE McGRAW-1111L INTERNATIONAL SERIES IN SOFTWARE ENGINEERING

Consulting Editor SOFTWARE QUALITY

Professor D . Ince
The Open University
ASS URANCE A STUDENT
INTRODUCTION
Titles in this Series

SSADM : A Practical Approach Ashworth and Goodland

SSADM Version 4 : A User's Guide 2/c Eva
An Introduction to SSADM Version 4 Ashworth and Slater
Object-Oriented Databases :
Applications in Software Engineering Brown
Danel
1
Ince
Object-Oriented Software Engineering with C++ Ince Professor of Computer Science
Introduction to Software Project Management and Quality
Assurance Ince. Sharp and Woodman Open University
Software System Development : A Gentle Introduction Britton and Doake
Introduction to VDM Woodman and Heal
Discrete Event Simulation in C Watkins
Objects and Databases Krohn
Object-Oriented Specification and Design with C++ Ilcndcrson
Software Engineering Metrics
Volume 1 : Measures and Validation Shepperd
Software Tools and Techniques for Electronic Engineers Jokes
Reverse Engineering and Software Maintenance :
A Practical Approach Lano and Haughton
Coding in Turbo Pascal Sargent
A Primer on Formal Specification Turner and McCluskey
Specification and Design of Concurrent Systems Mctt, Crows and Strain-Clark
Introducing Specification Using Z Rateliff
A Programming Approach to Formal Methods Casey
An Introduction to Formal Specification with Z and VDM Sheppard
Provably Correct Systems He Jifcng
Safer C Hatton
System Requirements Engineering Loucopoulos and Karakostas
Developing Object-Oriented Data Structures Using C++ McMonnics and McSporran
People and Project Management for IT Craig and Jassim
Concurrent Systems : Formal Development in CSP Hinchey and Jarvis
Standardizing SSADM Bryant
Sizing and Estimating Software in Practice : Making MKII
Function Points Work Treble and Douglas

The McGraw-Hill Companies

London • New York St Louis • San Francisco - Auckland
Bogoi,A • Caracas • Lisbon • Madhid • Mexico
Milan - Montreal • New Delhi Panama - Paris - San Juan
Further titles in this Series ore listed at the hack of the book Sao Paulo • Singapore • Sydney • Tokyo . Toronto
Published by
McGRAW-Hill Publishing Company
SHOPPENHANGERS ROAD . MAIDENHEAD . BERKSHIRE SL62QL-ENGLAND
TELEPHONE 01628 23432
FAX 01628 770224
CONTENTS

British Library Cataloguing In Publication Data

Ince, Darrel
Software Quality Assurance : Student
Introduction . - (McGraw-Hill
International Software Quality Assurance
Preface X

Series)
I . Title II . Series 1 What is software quality? 1
005 .10685 Aims 1
1 .1 Introduction 1
ISBN 0-07-709096-9
1 .2 The meaning of quality 3
1 .3 Some users of a quality system 9
Library or Congress Cataloging-in-Publication Data 1 .4 Some principles 16
Ince, D. (Darrel) 1 .5 Quality and the quality system 21
Software quality assurance : a student introductionIDarrel Ince . 1 .6 Standards and procedures 24
p. cm.
1 .7 Summary 25
Includes index .
1 .8 Further reading 25
ISBN 0-07-709096-9
1 . Computer software-Quality control . I . Title .
Problems 25
QA76 .76 .Q351536 1995
~~Eyra,t
005 . I068'5-dc20 95-7040 2 Software tasks and quality assurance 27
CIP 27
V Aims
~'AIV751 tti . m
2 .1 Introduction . 27
e 1G "
c 6 g1.3 . 0 6 .0 04. f'zi(o.s ) 2 .2 A software development model 27
2 .3 Another development model 31

l= C'cr;t~C /~'t ) h 2 .4 Verification and validation

2 .5 Software problems and quality assurance
34
38
2.6 Summary 40
2.7 Further reading 41
Problems 41
McGraw-Hill
A Division of77%eMcGraw-Hill Companies
3 Project planning 42
Aims 42
Copyright O 1995 McGraw-Hill International (UK) Limited . All rights reserved . No pan of this publication may be
reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, 3 .1 Introduction 42
photocopying, recording, or otherwise, without the prior permission of McGraw-Hill International (UK) Limited . 3 .2 Risk analysis 45
3 .3 Outline design 48
2345 CUP 9876 3 .4 The identification of quality factors 48
Typeset by the author
3 .5 Developing the quality plan 50
3 .6 Determining the project organization 52
Printed and bound in Great Britain at the University Press, Cambridge 3 .7 Determining the configuration management system 53

Printed on permanent paper in compliance with ISO Standard 9706 V

rt CONTENTS

3 .8 The evaluation of subcontractors 54 7 Testing

3 .9 Project costing 55 Aims
3 .10 Task identification 58 7 .1 Introduction
3 .11 The project plan 60 7 .2 Testing and quality assurance
3 .12 Summary 62 7 .3 Unit testing
3 .13 Further reading 62 7 .4 Integration testing
Problems 62 7 .5 System and acceptance testing
7 .6 Quality manual requirements
7 .7 Summary
4 Requirements analysis 64 7 .8 Further reading
Aims 64
Problems
4 .1 Introduction 64
4 .2 The requirements specification 65
4 .3 Specifying functionality 67 8 Configuration management
4 .4 Data specification 69 Aims
4 .5 Modes of operation 71 8 .1 Introduction
4 .6 Interfaces 72 8 .2 The process of configuration management
4 .7 Customer-analyst interaction 73 8 .3 Configuration management activities
4 .8 The requirements specification and the statement of requirements 74 8 .4 Documentation for configuration management
4 .9 Validating the requirements specification 75 8 .5 Quality manual requirements
4 .10 Developing the verification requirements 79 8 .6 Summary
4.11 The structure of the requirements specification 83 8 .7 Further reading
4 .12 Quality manual requirements 84 Problems
4 .13 Summary 84
4 .14 Further reading 85
9 QA and the new technologies
Aims
5 System design 86 9 .1 Introduction
Aims 86 9 .2 Prototyping
5 .1 Introduction 86 9 .3 Object-oriented programming languages
5 .2 Design documentation 87 9 .4 Formal methods of software development
5 .3 Data specification 88 9 .5 Structured methods and CASE technology
5 .4 Design procedure 89 9 .6 Summary
5 .5 Design validation 91 9 .7 Further reading
5 .6 The structure of the system design specification 95 Problems
5 .7 Quality manual requirements 97
5 .8 Summary 97
97 10 QA and the human-computer interface
5.9 Further reading
98 Aims
Problems
10.1 Introduction
10.2 The user/environment questionnaire
6 Detailed design and programming 99 10 .3 The task specification
Aims 99 10 .4 HCI design
6 .1 Introduction 99 10.5 HCI evaluation
6 .2 Detailed design 100 10 .6 Postscript
6 .3 Programming 101 10 .7 Quality manual requirements
6 .4 Quality manual requirements 104 10 .8 Summary
6 .5 Summary 104 10 .9 Further reading
Problems 105 Problems
viii CONTENTS
CONTENTS iX

11 Software metrics 157 16 Improving a quality system 222

Aims 157 Aims 222
11 .1 Introduction 157 16 .1 Introduction
11 .2 An example 222
157 16.2 The quality improvement team 223
11 .3 Definitions 159 16.3 Summary 225
11 .4 The uses of a metric 159 16 .4 Further reading
11 .5 A tour of software metrics 225
160
11 .6 The derivation of metrics 161 Appendix A Some QA questions 226
11 .7 Further reading 163 A .I Introduction
Problems 226
163 A .2 The 25 questions 226
12 Process modelling and process assessment 164
Bibliography
Aims 164 233
12 .1 Introduction 164
Index
12 .2 Processes and process models 165 236
12 .3 Notations for process modelling 167
12 .4 Individual process improvement 169
12-5 An example of process modelling in action 170
12.6 Process assessment 171
12 .7 Summary 172
12 .8 Further reading 172
Problems 173

13 Standards and procedures 174

Aims 174
13 .1 Introduction 174
13 .2 The starting point 175
13 .3 Unit programming 176
13 .4 Design reviews 184
13 .5 Summary 188
13 .6-Further reading 189
Problems 189

14 ISO 9001 190

Aims 190
14 .1 Introduction 190
14 .2 The elements of ISO 9001 192
14 .3 Main summary points 205
14 .4 Further reading 210
Problems 211

15 Case study 212

Aims 212
15 .1 Background 212
15 .2 First steps 215
15 .3 The design study 217
15 .4 Implementation 219
15 .5 The current position 220
15 .6 Summary 221
 1
WHAT IS SOFTWARE QUALITY?

AIMS

• To describe the nature of software quality assurance .

• To introduce the idea of a quality factor .
• To describe some common quality factors .
• To show how consideration of quality factors drives the quality planning process .
• To show how a quality system affects different members of staff associated with a
software project .
• To describe what exactly a quality system is and how it is used in software develop-
ment .

1 .1 INTRODUCTION

The building that I visited was very quiet, considering that software was developed in
it-normally programmers are quite a boisterous bunch and the offices had the ambience
of a funeral parlour . The reason I was in the building was that I had been called in
as a consultant to carry out a post-mortem on three software projects that had failed
dismally . The quietness arose from the fact that morale in the company was low after the
three disasters, and also from the fact that quite a large number of staff had been made
redundant-including many of the senior managers responsible for software development .
A revealing question to ask a programmer who has participated in a disaster, in
order to get some idea about the reasons for it, is : 'What tasks do you enjoy doing,
and what tasks do you not enjoy doing?' One programmer told me that he enjoyed
programming, and even testing, but what he hated was the process of retesting . He used
the programming language C to produce coded functions (a function is a module in C)
which, he claimed, were well tested and were then consigned to a project library ready
for subsequent development . This part of his job he thoroughly enjoyed doing . However,

i
2 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 3

he often found that the functions which he produced were returned to him a few weeks This is a very minor example of how a component-albeit a small component-of a
later with a change request . These change requests arose for two reasons . The first was quality system, in this case a procedure which instructs a programmer to store test data
that the company's marketing department, who was, in effect, the internal customer for on a file, is able to not only save a company money, but also make a software developer's
the software, continually generated new requirements . Members of the department often life easier .
talked to external customers who would buy the software only if it did something extra I hope that this anecdote might act as an antidote to the view of quality assurance
and, being a good marketing department, they promised the changes in order to generate which many people still have . It is a view exemplified by the story related to me by a
potential revenue . senior programmer who, at the start of his career, joined a company which developed
The second category of change came from modifications to hardware . The software real-time software . On his first day he was befriended by a helpful senior colleague who
developer produced systems which were intended to run on interface hardware ; this was pointed out to him where the best bar was in the area, where the best sandwiches were
often being enhanced in parallel with the software development . Each time a new feature to be bought, which project manager to work for, which project manager not to work
was added to the hardware, or a feature modified, it resulted in a large number of software for, and which secretary to ask for stationery. During the process of communicating
change requests . this advice on a tour of the building the senior colleague pointed at a door and said :
The programmer explained to me that when he received one of his original functions 'Whatever you do, don't go through that door, the people in there have been given the
back from the project library he would have major problems trying to remember his job of stifling our creativity .' The door, of course, belonged to the quality assurance
original test data, and would end up spending a considerable amount of time doing this, department . I hope that this book will go some way towards overcoming the prejudice
together with thinking up new test data which checked out the requested modifications which I still encounter-albeit in less extreme forms than that just described .
to the function .
I asked him whether he had thought about storing his original test data in a file and,
perhaps, storing the test outcomes in the same way . These files could easily be retrieved 1 .2 THE MEANING OF QUALITY
whenever a modification was requested, and the only work required from the programmer
would have been the derivation of additional tests which checked out the modifications . If you look at many of the books that have already been published on quality assurance
When I suggested the use of these files the programmer looked at me as if I had solved you will see that they generally agree on the meaning of quality . Usually, their agreement
every problem in his life . All I had done was to suggest something that a good quality is enshrined in the phrase `fitness for purpose' ; that is, a quality product-be it a car,
system should have insisted he do anyway . refrigerator, hair dryer or any other item-is one which does what the customer expects
The reason for the three software disasters that I had to investigate was an inade- it to do . Fitness for purpose is an important concept, and forms a central tenet of quality
quate quality system . The company had what it called a `quality manual' . This was a assurance . However, later in this chapter you will see that it is not the only property of
sort of rudimentary quality system, but it suffered from two problems : one managerial a quality product .
and one technical . The managerial problem was that it was not mandatory for project Before examining the full story it is worthwhile looking at a particular implication of
managers to use the system . Software projects in the company had achieved the status the phrase `fitness for purpose' . This implies that somewhere there will be a description
of medieval Italian city states, with individual project managers having a huge degree of of the purpose for which an item is intended . For very simple objects this description
independence even over the pitifully small quality assurance department . might mainly be contained in the name of the article ; for example, the term `hair dryer'
The technical problem was that the quality system did not insist on developers conveys the vast majority of the functionality of a device used to dry hair . The description
carrying out tasks in such a way that changes such as the requirements modifications of the purpose of some articles might be contained in a user manual which, as well as
demanded by the marketing department could be coped with, for example, by insisting providing details about how to use them, might also include some technical specification .
on the storage of test data . The three projects that bottomed up were all affected by For large items-for example, software or hardware systems-the purpose is enshrined
changes which gave rise to a massive amount of unnecessary work-work which a good in a document usually known as the requirements specification, sometimes known as the
quality system could have minimized . Later in this chapter I will return to this problem system specification. In Chapter 4 I shall describe in some detail what can be found in
of change, as it forms a major input into the process of developing a quality system . a requirements specification for a software system . In reading this chapter all you need
In order to convince the management of the company that they needed to invest in to know about a requirements specification is that it contains a description of what a
a good quality system I carried out an exercise with one of the luckier project managers software system is to do, together with descriptions of constraints such as response time
who remained on the payroll. We calculated the average amount of time required for which are to be associated with the system . The requirements specification is the most
programmers to retest C functions on the three projects, and then calculated the time important document generated in a software project, and it is around this document that
taken if a quality system had insisted on the storage of the test data . Admittedly this was a quality system revolves .
quite a rough calculation, but in the end we had enough confidence in the assumptions The modern view of quality assurance takes a more sophisticated view than that
that we had made to predict that just by addressing one small aspect of the development of fitness for purpose . A high quality product is one which has associated with it a
process the company could have saved a programmer's salary for the five years during number of quality factors. The quality factors, often known as quality attributes, could
which the projects were live . be described in the requirements specification ; they could be cultural, in that they are
4 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 5

normally associated with the object through familiarity of use and through the shared Self Assessment Question 1 .1 What are the three categories of quality
experience of users ; or they could be factors which the developer regards as important, but factor?
which are not considered by the customer and, hence, are not included in the requirements
specification . Solution They are those which might appear in the requirements spec-
It is instructive to examine some examples of each of these three categories of quality ification, those which are cultural in nature and those which are of interest
factors. The first category comprises those which would be contained in the requirements to the developer but only of indirect interest to the customer .
specification . An example of this is portability . A customer for a software system may
require a system to be executable across a diverse range of hardware architectures . Con-
sequently, these architectures would be described in the requirements specification . It is now worth examining some of the important quality factors . The list that is presented
An example of the second category, comprising cultural factors, is usability. Because here can be found in McCall et al . (1978), although I have amended it somewhat . It
a customer may have experience of systems in which it is relatively easy to communicate is worth pointing out that there is some overlap within the quality factors described
with a computer, he or she may not specify how an interface is to be implemented in by McCall : that aspects of one quality factor can be found in another quality factor .
detail in a requirements specification . In the specification there may be a brief directive Nevertheless, his categorization forms a good basis for a quality system .
to use a WIMP interface or a line-by-line command interface, but because of the same The first factor is correctness, that is, that a software system actually conforms to its
shared assumptions with the developer, the details of the interface would normally be requirements specification ; naturally, this factor should be totally present in any system .
omitted from the requirements specification . The next is maintainability or modifiability . This describes the ease with which a
The third category consists of those quality factors which may be of interest to software system can be changed . In the original paper written by McCall, the term
the developer, but not of direct interest to the customer . An example of this might maintainability had a very limited meaning-it referred to the ease with which errors
be reusability : the ability to transfer modules constructed for one software system into could be located and fixed . McCall's original paper was written at a time when the
another software system . For example, a software developer may be currently producing changes which a software system experienced were due to errors being discovered-either
a software system for one customer which is somewhat similar to a system that he or during development or during maintenance, when the software was in operation .
she hopes to construct for another customer . If the second system is being bid for by a However, over the last decade or so it has become quite evident that many other
number of software developers, then the producer of the first system could gain a major categories of change occur . There are, of course, changes due to errors being committed-
commercial advantage during the bidding process by ensuring that the first software these will always be with us, although I would hope that, as technology improves, the
system is highly reusable . This advantage would, of course, be obtained by making a extent of this category will lessen . There are also changes due to requirements volatility .
very low bid since little new programming would be required . This category of quality Paradoxically, one of the indicators for a software developer that a good system has been
factor is, at best, only of indirect interest to the customer, and would not be contained developed is to receive numerous requests from a customer for changes in requirements ;
in a requirements specification . In the case of reusability a customer, while not perhaps these are often framed in terms of new functions . Also, the developer will receive requests
asking for reusability in the requirements specification, may be interested in receiving a for modifications due to changes in a customer's external circumstances . For example,
licence fee for the software that is reused . an accounting package may have been delivered which, although it satisfied the original
It is important to point out that the three categories of quality factor do not rep- customer's requirements, now has to be modified because tax laws have changed . Thus
resent a hard and fast taxonomy . What category a factor is placed in really depends on nowadays, because customers demand more and more from successful systems, and ex-
the customer, the customer's circumstances, the application area, and the developer's ternal circumstances change, a high level of modification can be attributed to changes
circumstances . For example, I described usability as a cultural quality factor in that in requirements . There is another category of change which McCall did not envisage :
everybody expects their systems to be usable ; this is usually the case in a large number changes which somehow improve a system without changing its functionality ; for exam-
of software systems . However, there are some categories of software systems used for ple, tuning a system in order to give it a faster response time, or rewriting a device
safety-critical applications where usability is of such importance that it is included in handler to cope with a new output device .
the requirements specification, for example by specifying a metric which quantifies the The first category of modifications-those due to error fixing -are known as cor-
unacceptable frequency of erroneous commands being initiated by the operators of the rective changes, the second category-due to the developer responding to changes in
system . requirements-are known as adaptive changes and the third category-changes which
Even though the three categories are not hard and fast, what is important is for improve a system-are known as perfective changes.
a quality system to recognize that they exist, and to ensure that right at the start of What is important is not the categories of change, but the fact that as a result, many
the project the manager examines all the quality factors that may be necessary before software systems have a very high level of maintenance, with the majority of the changes
deciding on the quality controls that are required for a project . A project manager should being adaptive . Moreover, many of these changes will not only occur during maintenance
not just assume that `fitness for purpose', as exemplified in a requirements specification, but, for projects which have a long time duration, will occur during the project itself . For
is enough . this reason alone I would say that the maintainability factor should always be present to
6 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 7

a very great extent in a software system . After correctness, I would regard it as the next Efficiency is another important quality factor . It is used to describe the degree to
most important quality factor . which computing resources-file-space, memory and processor time are used in an ap-
plication . This is a difficult factor to categorize : many requirements specifications will
contain detailed descriptions of the amount of hardware available and the desired re-
Self Assessment Question 1 .2 Into what category of change would you sponse time . However, I would say that there are features which make it cultural, in that
place a design and code modification which speeds up a system? a customer will expect, but not explicitly specify, that a software developer will make
efficient use of the computing resources available to him or her .
Solution This is a perfective change because it improves the system . Integrity is used to describe the extent to which the system and its data are immune
to access by unauthorized users . Again, this is a quality factor which cannot be precisely
categorized : some systems, for example those used in financial applications, will have
Maintainability is normally only of indirect interest to the customer . Very few customers, requirements specifications which describe the level of access allowed to such systems .
indeed, include directives about maintainability in the directions they give to a software However, there is a cultural assumption that a delivered system should not be capable
developer, apart, of course, from those customers who intend to carry out the maintenance of being tampered with by unauthorized users or intruders .
of a system themselves . However, it is worth pointing out that there will be an indirect Reusability is also becoming increasingly important, and describes the ease with
interest in that software developers who build maintainability into their systems have a which chunks of software in one system can be moved to another system . This is normally
much greater chance of delivering a correct system within time and within budget . For a quality factor which is only of indirect interest to a customer .
example, if changes to a system are going to take a long time to apply, then there is The final quality factor which I will describe is interoperability. This is the ability of
always the possibility that when a high level of errors are discovered the duration of the a system to operate in conjunction with another software system, for example a spread-
project will overrun the estimated delivery time for the software . sheet . Normally this factor is specified in a requirements specification and is of direct
Another quality factor, which has already been mentioned previously, is portability. interest to a customer .
This is the effort required to transfer a system from one hardware platform to another . This, then, is a brief discussion of what a quality factor is . Many of you might have
Normally, portability will be detailed in a requirements specification . However, for com- been tempted to regard it as somewhat academic ; indeed, it has been something of an
mercial reasons it is sometimes of direct interest to the developer but only of indirect academic game to derive longer and longer lists of quality factors .
interest to the customer, as in the example cited earlier, where portability was built into However, there is a very practical reason for discussing quality factors this early in the
a system in order to give the customer a competitive edge during the process of tendering book . It is connected with the fact that a quality system should be flexible . Normally, a
for another system . quality system is embodied in a document known as a quality manual . At the beginning of
Testability is another factor which is of direct interest to the developer but is very a software project a project manager should decide which elements of the quality system
rarely directly specified by the customer . It describes the ease with which a system, or should be used to ensure the quality of the software that is to be delivered . In order to
part of a system, can be tested . For example, a system which has a very poor requirements do this, the quality manager should examine a list of quality factors ; identify those parts
specification that contains a large amount of ambiguities and platitudes is very difficult of the quality system which should be used to enforce the level of quality factor required ;
to test, because the staff responsible for system testing will have major problems in identify those parts of the quality system which should not be used ; or even develop
specifying the tests which check out the requirements specification . some new quality controls which may not be in the quality manual . A consideration of
Another important factor is usability. This is the effort required to learn, operate the quality factors to be built into a system, and the level to which they are to be built
and interrupt a functioning system . Usability is often a major problem with systems : in, is often embodied in a checklist . Consideration of the checklist by a project manager
many software developers tend to think about the functions of a system at the front- at the beginning of a project drives this process .
end of a project, and then only bolt on an inadequate interface in its dying stages . If Obviously there will always be components of a quality system which will be used
anyone asks me to justify my obsession with quality factors and my contention that in every software project ; for example, a standard for the requirements specification
'fitness for purpose' is not a wholly adequate basis for quality assurance, then I give will always be needed . However, depending on the customer, application or software
them the example of the system which, when implemented, satisfies all the functions in technology to be used, the project manager will decide what components of a quality
its requirements specification but, because of a very poor interface, is highly unusable . system will be either used, strengthened or omitted . In order to understand this link
Reliability describes the ability of a software system to carry on executing with little between quality factors and the quality system it is worth examining some examples .
interruption to its functioning . This factor is normally expected to be ultra-high in certain
classes of safety-critical systems, and will often be specified in terms of metrics involving
factors such as mean time between failure . For other types of system, this factor will be
cultural in that it will not usually be specified in a requirements specification, but will
be assumed by the customer to be present to a high level in a system .
8 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 9

main themes of this book-it is worth looking at who the users of the quality system
Self Assessment Question 1 .3 How is a quality manual used on a soft-
are . There is a popular misconception that the only users tend to be management : a
ware project?
misconception which I would wish to quash as early as possible . Before doing this, it is
worth briefly introducing some terminology which is used in the next section .
Solution The quality manual will contain facilities such as standards and
procedures which are extracted for a particular project . Which of these are A quality system contains procedures, standards and guidelines . A procedure is a
description of the actions which are to be carried out when a particular project task is
extracted depends on the quality factors for a project .
to be executed for example, the production of a system design . I shall define a standard
as a directive which describes how a particular document is to be structured, and what
data is contained in the document . A guideline is a series of instructions which provide
The first concerns a system which is to be developed where the management of the
advice on carrying out a task . For example, a guideline might specify that normally five
company decides that reusability will be high in the system . This decision would normally
be made by the manager of a software project during the process of examining a quality members of staff are to attend a particular project meeting . The difference between a
factor checklist which contains a list of questions relevant to the particular factor . When guideline and the other two types of document is that a guideline represents advice, while
the quality factor heading for reusability is encountered by the project manager there the others represent activities which have to happen on a project . Normally, the advice
enshrined in a guideline is taken ; however, the instructions in procedures and standards
will be a number of questions which will drive the manager's thinking about that factor .
must be followed .
For reusability there will be questions such as :
It is worth making the point that the vocabulary which I have introduced is not used
Are we thinking of developing a similar software system in the future? by some companies ; for example, some software developers use the term 'standard' to
•
cover both procedures and standards . However, for standardization reasons I shall use
• Does our company have a policy of developing a reusable library of components?
the terminology above .
• Are we currently bidding for a software system which has similarities to the one which
is to be developed?
Self Assessment Question 1 .4 What is the difference between a stan-
Let us suppose that the answer to the third question is yes, and the project manager de-
dard and a procedure?
cides to use a programming language such as C++ which has a high degree of reusability .
A consequence of this might be that the quality system does not contain a programming Solution A standard describes the format of some document such as the re-
standard for C++ and one has be developed, or an existing programming standard for C quirements specification. A procedure details the steps that are to be taken
has to be modified since C++ is based on it . in carrying out some software task such as testing a module .
Another example involves the development of a safety-critical system . Such systems
have to have an extremely high correctness quality factor . On examining the quality
manual prior to starting such a project, the manager may decide that because this factor
is so high every technique for ensuring correctness which is contained in the quality 1 .3 SOME USERS OF A QUALITY SYSTEM
manual has to be used . This armoury of validation techniques may be much larger than
those which the manager would normally deploy on a project for which the correctness 1 .3 .1 The project manager
quality factor was lower, for example in developing software for a clerical application .
A third example might be where a transaction-based system is to be produced, where In this book the term `project manager' is used to describe a member of staff who has
the customer employs keyboard operators in an area of the country where such staff day-to-day control of a software project . Such a person carries out a number of func-
are in short supply. The project manager may decide-either after reading documents tions : planning, monitoring, controlling, innovating, and representing . Planning involves
provided by the customer or in conversation with the customer-that a high degree of a number of tasks ; these include estimating the risk of a project, estimating its overall
usability is required . This would mean that some usability studies would need to be cost, deciding which quality controls are to be used, identifying the tasks that make up
carried out and usability testing would need to be scheduled during the later stages of a project, scheduling these tasks, deciding on the particular project organization to be
the software project . The quality manual would be consulted and any sections pertaining used, and establishing the interfaces between the project and both the customer and
to usability would have to be taken on board the project or, if non-existent, would have the senior management of the software development company . It is worth stressing that
to be developed . The topic of the human-computer interface and quality assurance is planning is not just a one-off activity : in the vast majority of cases a manager will, of
discussed in Chapter 10 . course, carry out quite a large amount of planning at the beginning of a project, but will
Thus quality factors, or more correctly a consideration of quality factors, forms the also undertake replanning, for example when a change in requirements to a project is
basis for the quality controls that are going to be applied in a software project . Before notified by the customer .
examining some of the principles behind a quality system-many of which constitute the
10 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 11

Monitoring involves checking the progress of the project against an overall project are needed, or what training is required for staff who are available for a project but
who do not have requisite skills .
plan and evaluating the performance of staff on the project . Monitoring is intimately
connected to controlling : the process of taking actions to ensure that when a project
starts to diverge from the project plan, it is brought back on target . This will involve This is just a subset of some of the facilities that should be provided by a quality system .
However, it does give an idea of the huge influence of these facilities .
some degree of planning, or rather replanning .
Innovating involves the introduction of some new innovation, for example a new
development method, project organization, or testing tool, and evaluating the effect . 1 .3 .2 The programmer
Representing is the final function carried out by a project manager . He or she rep-
resents his or her project in dealings with the customer, the senior management of the I shall assume that a programmer is a member of staff who is given a specification
for a chunk of software-it could be a subroutine or a program-and who then has to
software company and other agencies such as the quality assurance department, the
marketing department and the personnel department . produce program code which he or she tests for correctness . Most programmers are not
only involved in programming, but also carry out a large amount of reprogramming, for
It is worth exploring some of the ways in which a quality system is able to support
example when a requirements change occurs . Some of the help that a quality system
these managerial functions :
should provide a programmer is shown below :

• A quality system should provide facilities whereby a project manager can consider
the vast majority of possible risks which could affect a project . Risk analysis is an
• The quality system should set programming standards which determine the way that
a program is to be constructed and displayed, for example by insisting that certain
important, but much neglected, activity ; for example, consideration of risk is used
layout conventions are adhered to . A good programming standard-and I must say
as one of the factors to be taken into consideration when making a decision about
whether a project should be started or even bid for . As a minimum, the quality there are some pretty terrible ones-should enable a programmer to read source
system should provide a checklist which would outline those factors which might code, and understand the contribution of each part of the source code to the overall
function of the chunk of the software that he or she is examining . In this way, testing
result in a project being late or over-budget . These include factors such as the degree
of familiarity that the customer has with the application which is to be computerized ; and debugging becomes easier, and the effort to understand the software after perhaps
many weeks of absence is minimized .
the degree of knowledge the customer has about software practice ; and the reliance
• It should provide directions to the programmer to store away test data and test
of the project on outside agencies who may not be trusted to deliver on time, such
outcomes in files . In this way, reprogramming becomes much easier . The programmer
as hardware providers and external software contractors . More sophisticated quality
can retrieve the test data, modify it in response to, say, a requirements change, and
systems provide spreadsheet programs which prompt the user about these factors,
then apply it again . Obviously, this is preferable to trying to reconstitute the whole
and give a numerical risk factor which might then be used by the project manager
data set again .
in adjusting upwards a rough estimate for the cost of the project in order to build in
• It should provide a standard which insists that the names of the program chunks
some contingency .
• A quality system should specify standards which will enable staff to report on their which have been produced by a programmer have a correspondence to the files used
to store the source code, object code, test data, and test outcomes for the software .
activities in a uniform way . Such reports should be able to be easily processed by a
For example, if a program has a name Update, then a quality standard should insist
project manager in order to give some idea of how close to schedule a project is .
that if it was programmed in COBOL the file holding the source code should have the
• Software estimating is still quite a hazardous process, even for medium-sized projects .
name, Update .cbl, the object code the name Update .obj, the test data file Update .tda,
A quality system should lay down standards and procedures which ensure that cost
and the test outcome file Update . out. If this trivial standard is used, then programmers
and expenditure records from earlier projects are kept in an easily accessible form, so
that a project manager can check that the estimate for a project is not too different who are asked to reprogram the work of other staff do not need to waste time trying
to find the location of the software in a project library-a search which often involves
from a similar project ; or, more likely, that parts of a system to be produced have
long, time-consuming phone calls .
similar costs to those parts of systems already developed .
• A quality system should provide a way of collecting and analysing defect statistics :
data about errors, their severity, and when they occur on a software project . This
Self Assessment Question 1 .5 What quality factor does a procedure
data can be used in a number of ways : in analysing staff performance ; as one of the
which directs a programmer to store test data away in a file address?
inputs into the process of evaluating whether an innovation has been a success ; or in
predicting the pattern of errors in a product when it is released .
Solution This mainly addresses the maintainability quality factor .
• A quality system should contain a standard for a project plan which gives details
about the capabilities of the staff who are to carry out project tasks such as system
testing . This enables the project manager, when he or she is representing a project
with some resourcing department within the software company, to specify what staff
12 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 13

1 .3 .3 The system designer to be cluttered up with a description of functions for which another member of the
customer's staff is responsible .
I shall assume the existence of a member of staff known as the system designer . He or
she will be responsible for processing a description of the software system that is to
• It should provide a number of checklists which enable the analyst to ensure that

be produced-normally, the requirements specification-and providing an architecture features or issues associated with a software system that is to be developed are not
forgotten . Such checklists are usually general, and might include questions such as :
which implements the functions demanded by the customer, but which at the same time
respects constraints such as response time which have been specified by the customer . `Have you specified what should happen when the system exhausts a hardware re-

The system designer usually produces a process architecture which describes the inter- source such as main memory?' Sometimes software developers who work in a narrow
relationship of the modules which make up a system, and a data architecture which application area will have, as part of their quality system, an application-specific
checklist . For example, a developer who produces software for stock-control applica-
describes any databases that are to be used by the system . A number of the facilities
tions might have a checklist which includes questions such as : `Do the customers who
that a good quality system should offer the system designer are shown below :
are on a back-orders queue have some priority which will determine the order of the
items on the queue?'
• It should provide a standard which describes how the process architecture and data
architecture are to be written down . A good standard should enable the designer to
• It should provide a description of the processes involved when an analyst liaises with
a customer; for example, specifying the way in which functions in the requirements
check that the system which has been designed actually implements the functions
described in the requirements specification . specification are checked against the customer's requirements .

• It should also specify standards which lead to a system that should be easily main-
tainable . For example, a good standard for design is the insistence that each module 1 .3 .5 Senior management
in a system carries out one function-and one function only. Systems designed in this
way tend to be far easier to maintain than systems containing modules which each By the term `senior management' I mean those members of staff who work at the next
level of the management structure above project managers . They are often called business
carry out large sets of functions .
managers, and usually supervise a number of projects . Such staff will require the quality
• It should provide standards which lead to a good requirements specification . For a
system to provide a number of facilities, for example :
system designer, the word `good' implies that the requirements specification should
contain no ambiguities, contradictions or platitudes, and that constraints such as
response time are directly cross-referenced to the functions to which they apply .
• It should provide senior managers with reports of achievement against targets for each
of their projects ; for example, what tasks should have been completed by a certain
date and what tasks have, in fact, been completed ; what expenditure should have
been incurred by a certain date and what expenditure has actually been incurred .
• It should provide direction on the setting up of documentation audit trails . Such
1 .3 .4 The analyst a trail might consist of memoranda which have been generated by checklists ; for
example, when the analysts in a project have queried a particular set of requirements
I shall assume that an analyst is a member of staff who liaises with a customer and
with the customer . Such audit trails are useful for members of senior management ;
produces a requirements specification which is then extensively used by staff, such as the
for example, when resolving some dispute about requirements which has escalated
system designer, who carry out different functions . The analyst's needs tend to be centred
out of a project to his or her level of responsibility .
around the requirements specification . Some examples of how a good quality system is
• It should provide facilities whereby reports on defects discovered during development
able to satisfy those needs are shown below :
are issued regularly . Such reports are not only of value to the individual project
manager, but are also useful for senior management . For example, where a senior
• It should provide a standard for the requirements specification which, among other
manager is actively involved in a management team which is attempting to carry out
concerns, specifies an organization of the requirements specification in which related
some process improvement, diagrams showing where defects were introduced during
functions are textually close to each other . For example, in a stock control system
development, and where they were discovered, would be a major input into the task
the descriptions of the functions associated with querying a database of information
of determining what processes need to be improved .
about products which are out of stock should be physically close to each other .
One of the implications of this is that it makes it easier for the analyst to check
for contradictions . It also makes it easier for the customer to check the requirements 1 .3 .6 The maintainers
specification : related groups of functions are usually the province of one person-in
In many companies the vast majority of resources is spent maintaining existing software :
the case of this example the purchasing manager- who, in accessing the requirements
responding to error reports and changes in system functions by modifying the existing
specification for text which relates to his or her work, does not want the relevant text
code of a system and changing documentation . In order for the staff charged with mainte-
14 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 15

nance to do their job efficiently the quality system should provide a number of facilities, • It should provide guidelines about how facilities provided are to be extracted and
for example : used as quality controls for a project . This list of controls would be specified in the
quality plan . This is often a contractual document, or is at least signed off by the
• It should insist on programming standards which ensure that maintainers can easily customer . The document should be a complete description of the way in which the
determine the function of modules in a system . developer intends to check that the system meets customer requirements, and the role
• It should provide traceability features which ensure that the process of determining of the customer in this process .
which part of the system is executed when a particular function is exercised can be • It should specify the format of reports that are to be sent to the customer concerning
easily carried out . progress and how to determine the frequency with which these reports are sent .
• It should provide facilities which enable staff to track and monitor the various versions
of a system which are created during the maintenance process .
• It should ensure that staff who created a system store away their test data so that Self Assessment Question 1 .6 What part of the quality system provides
the maintainers do not have to create fresh test data . the best means for enabling a customer to check that software will be de-
veloped adequately?

1 .3 .7 The staffing department

Solution The best part is the project plan, or rather the project plan
I shall use this term to describe a central function which is responsible for human re- standard . This contains details such as the technical solution adopted, the
sourcing on projects ; sometimes this department also has a training role . There are a experience of the development staff to be used and how project monitoring
number of ways in which a quality system could serve the interests of this department ; is to be carried out .
for example :

• The standards for a project plan should insist that the skills level of the staff on
1 .3 .9 The testers
the project is properly specified . This obviously helps the staffing department to
allocate the right people to the project, and to detail requisite training if staff with Two critical activities on a software project are system testing and acceptance testing .
the specified skills are not available . Either developmental staff or staff from the quality assurance department derive a series
• The quality system should provide instructions on the activities carried out when the of tests from the requirements specification, which are intended to check that the system
project has been completed . One activity relevant to a staffing department, which which has been developed meets customer requirements . Such staff require a number of
also has a personnel function, is for the project manager to produce a written report facilities from a quality system . Some examples follow :
on the effectiveness of the staff on the project . There should be a standard format for
this report, defined in the quality system, so that comparisons can be made between • It should ensure that the requirements specification is constructed in such a way that
staff, for example when they are being interviewed or reviewed for senior positions . system tests can be easily derived and related back to the functions that they test .
• It should ensure that procedures are in place which enable test data and test outcomes
to be stored somewhere in a project library . The test files in the project library can
1 .3 .8 The customer
be easily retrieved when retesting takes place .
The customer is also a user of the quality system . The customer's concerns are rather • It should ensure that procedures are in place which ensure that those responsible for
different from those of the developer ; for example, he or she is concerned that adequate test planning can check that adequate resources are available during the test phase ; for
means are going to be used to develop the software and to check that it meets require- example, that during the development of a software system intended for a mainframe
ments; that adequate reporting facilities are provided which enable the progress of the computer, the computer is actually available for testing and is not too heavily loaded
project to be monitored ; and that adequate liaison occurs during the period in which by existing applications .
the developer is demonstrating that the system actually meets user requirements . Some
examples of the type of facilities that a quality system should provide for the customer
1 .3 .10 The marketing department
are :
The marketing department sells software . In a company which develops one-off systems
• It should provide directives which specify how progress meetings are to be organized : the effectiveness of this department will determine the profit levels . A quality system
who should be invited from the developer's staff, when meetings should be scheduled, should offer facilities to staff involved in marketing . For example, every project should,
what physical arrangements, such as the booking of a room, should be made, and as part of the debriefing process, produce a list of subsystems and modules which the
how issues arising from a progress meeting are to be resolved . project manager feels could be reusable . Given this list the marketing department, in
conjunction with project staff, may be able to produce a bid for a future project which
16 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 17

is substantially lower than a competitor's bid, because of the potential high reuse of which I have tried to implement ever since . He told me that a programmer is so bound
previously written modules . up with a program that it takes a totally independent view of that program in order to
These, then, are some examples of which staff use the quality system . In summary, debug it effectively . He also told me that individual programmers find it very difficult to
everybody uses it and, hence, everyone in a software company should be responsible for detect errors because this detection is concrete evidence of what they might perceive as
quality assurance . their own incompetence .
Because of the difficulty that staff have in validating their own work a good quality
system should provide facilities whereby the validation of any product is decoupled from
1 .4 SOME PRINCIPLES those who produced it . There are a number of examples of this principle, and it is
worth briefly exploring some of them here . Later sections of this book will look at these
There are a number of important principles which have to be borne in mind when im- techniques in much more detail .
plementing a quality system . These issues permeate the book, and it is worth collecting A good example of this decoupling occurs with the programming of a module . Nor-
them here for reference . mally, on a software project, a programmer codes a module, generates test data and
then checks that the execution of the module matches his or her expectation of the test
output . A good way of ensuring a high degree of independence is to have a procedure
1 .4 .1 Independence which insists that, after producing a module, a programmer sits down with another pro-
The first principle is of independent validation . In order to describe this, it is worth grammer and describes in detail to the second programmer what the module does . My
telling you a short story . But first, some background to give an added dimension to my experience is that even if the second programmer says nothing, the very act of explaining
story. I was brought up in South Wales where religion plays an important part in the the function of the module usually leads to the first programmer detecting errors . Once
life of very many communities . It certainly played a major part in the life of my own this code reading process has finished, the second programmer then tests the module .
village : my brother and I were packed off to chapel three times on Sunday and at least This puts the onus on the second programmer to listen during the read through . A clever
once during the week . We were preached at by a white-haired priest who harangued us project manager will usually ensure that the second programmer is one who the first pro-
about almost every evil in the world . The priest never hesitated to remind us that if we grammer respects, fears, or is in awe of . This tightens the validation and usually results
worked hard in this world, then we would get both the rewards of this world and the in the first programmer delivering high-quality program code which is easy to test .
rewards of the next world . I was impressed by this message : that if I worked hard for a Another example of independence in a quality system concerns the organization of the
comparatively short time-say 40 years-then not only would I get rewards for 40 years, quality assurance function . I have experienced a number of quality assurance regimes, and
but also for the possible infinity that followed . the best tend to be those which are organized as a separate department, with the head of
Because I took the preacher seriously I worked hard at school and university . I the department having a separate reporting line-often up to board level . Projects would
also worked hard at the first job that I took : as a programmer developing real-time normally then have a member of the quality assurance staff attached to them . This form
software for mainframe computers in the mid-sixties . I used to come in early with my of organization usually means that the project manager is unable to browbeat quality
ham sandwiches-for I worked through lunch, sandwich in one hand and core dump in assurance staff into relaxing standards when a project has time or budget difficulties .
another-and worked very late . Two more extreme examples of independence are black teams and bug bounty hunters.
One of the tasks that I had to carry out was to develop a 12k assembler program Both are American phenomena . The former are aggressive testing teams which virtually
which implemented a high degree of parallelism . Undoubtedly, it was the most difficult attack a software system before it is handed over to the customer for acceptance testing .
program that I had to deal with . I came in every morning and spent hours working Such teams are often made up of staff who hate project managers and those who enjoy
through core dumps, sometimes spending days trying to track down errors . One error the act of destruction . Bug bounty hunters are companies who specialize in independent
caused me major problems . I took the best part of a week tracking it down and the way testing. Normally, you pay a bug bounty hunter a fixed fee, together with a sum of money
that it was discovered was, I thought, remarkable . for each bug detected in system testing.
I shared an office with another programmer who did not share my attitude to work .
He would come in late, leave early and have long lunches-liquid lunches . Being friendly, 1 .4 .2 Maintainability
he would often lurch into the office, look over my shoulder and tell me what was wrong
with my software, and collapse into his office chair . I ignored his attempts at being Another principle involves maintainability . Figures gathered over the last decade have
helpful since I assumed a drunk would have little, if any, debugging skills . However, in indicated that software developers can spend as much as 75 per cent of their project
this particular instance I listened to him and he was right . For the next few days I listened budgets on software maintenance : the process of modifying a system once it has been
to him again, and he pinpointed my errors with a remarkable accuracy. made operational . There are a number of reasons for this high figure . The main reason

This made me very depressed : here was a drunk, debugging better than myself who is that a software system is a reflection of the world, and since the world changes the
had believed the preacher of my childhood who told me that hard work leads to rewards . system must change . For example, an existing accounting package has to change when
I went to see a senior programmer and explained my problem . He pointed out something taxation laws are changed . A good quality system should have standards and procedures
18 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 19

which ensure that the process of changing a document or program code is as easy as
possible ; for example, by insisting on naming conventions for test files so that they can Self Assessment Question 1 .7 Explain the difference between reverse
and forward traceability.
be easily found in the project library .
Another implication of the fact that maintenance is a high resource consumer is
that a software quality system should incorporate something known as a configuration Solution Forward traceability describes the ability to trace easily from

management system, the components of which are described in Chapter 8 . This is a set a function in the requirements specification to the modules that imple-
of standards and procedures which govern the process of modifying a system both during ment that function . Reverse traceability describes the ability to trace from
a chunk of code-usually a module to the functions that it helps imple-
development and maintenance : how changes are proposed, agreed, applied and validated,
ment .
together with procedures for notifying relevant staff of changes .

1 .4 .3 Traceability
1 .4 .4 Incrementalism
I am often asked to pass judgement on a software developer, usually on behalf of a
One of the unpleasant facts about software development is that it exhibits what is known
customer who may be thinking of using the developer to construct a system . One strategy
which I use to judge the quality of the software developer's quality system is to ask for the as combinatorial explosion . This means that as a system gets larger and a certain task
is applied to the system, such as testing, the degree of difficulty of that task does not
source listing of a finished project and flip over some pages until I find a module . I then
increase linearly as a function of the size of the system but often rises exponentially .
ask the staff concerned what functions in the requirements specification that module helps
implement . If they can tell me in ten minutes, then I know that their quality system-or A good quality system should provide facilities whereby software development is split
up into small manageable tasks . A good example of this is integration testing . Integration
at least that part of it which is concerned with the requirements specification-is good ;
is the process whereby a project builds up a software system a chunk at a time, with
if they take hours to tell me, then it usually indicates that the software developer will
integration testing being the tests which take place after each chunk has been added .
have problems, especially during maintenance .
This is an example of something known as reverse traceability : the ability to trace Most companies claim that they carry out integration . However, the integration that
they practise is very coarse-grained : usually subsystems are integrated . An integration
from the source code of a module back to a function in a requirements specification .
strategy which adds single modules, or small numbers of modules, invariably leads to a
There is also forward traceability, which describes the ability to trace from a function to
system testing phase which is less error-prone than would be the case if integration was
a module .
not carried out .
Why is traceability important? The reason is connected with maintainability . In
order to see why a good quality system should establish linkages between program code In order to see why integration is such a useful technique it is worth looking at system
testing . When an error occurs in system testing the software developer is faced with a
and a requirements specification, consider the process of acceptance testing . This is the
massive task : that of detecting where an error occurs in a system which may contain
final stage of the software project when the customer, in conjunction with the software
hundreds, if not thousands, of modules . Because of this, debugging during system testing
developer, checks that a system is ready for handover . If an acceptance test for a function
is a highly resource-intensive process . Now, if a software developer practised integration
fails, then there is usually a lot of embarrassment, and the developer has to fix the error .
and integration testing, the process of debugging would be much easier . If an error
If the test that failed was test n, then a poor software developer will correct the
error, repeat acceptance test n, and move on to test n + 1 . This is certainly the wrong occurred after an integration test, then the error is normally found in one of the small
number of modules which have been added to the system at the last integration . By
thing for the software developer to do, for in fixing the error that leads to the failure,
he or she may have inserted another error into the system which could only be detected devoting some extra resources to integration, the developer will normally remove many
of the errors which would have been detected during system and acceptance testing, and
by tests carried out prior to test n . If the documentation for a project does not support
which would have consumed an inordinate amount of resources .
traceability then, when an acceptance test fails, the customer has every right to ask the
developer to start the acceptance test process again .
However, if the quality system does support traceability, then all the software devel- 1 .4 .5 Early validation
oper has to do is to identify the modules which have been changed, trace back to the
A good indication of the quality of a quality system is how much direction it devotes to
functions which have been affected, and then re-execute the acceptance tests which check
the process of early validation : the checking of a system during the early stages of soft-
out those functions .
ware development-during requirements analysis, requirements specification and system
design . The reason for this is that if a developer commits an error early on in the soft-
ware project, and it is not detected until the later stages, then the resources devoted to
rectifying the error can often be far greater than the resources which would have been
needed if the error was detected and rectified at an early stage . Many software projects
20 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 21

have failed because of requirements errors which were only detected towards the end of Secondly, the quality system requires periodic review . There are a number of reasons
the testing phase . There are a whole battery of techniques that can be used for early for this . In the early days of deployment of a quality system there will be a surprisingly
validation : these include technical reviews, meetings where staff examine a document large number of deficiencies reported : producing a good quality system is very much like
for correctness ; prototyping, where an early version of a system can be shown to the developing a software system, and you should not be surprised that some of the standards
customer during requirements specification ; and simulation . that were initially documented do not work as well as they should . In the longer term,
The quality system should offer facilities for all these ; for example, for reviews it the business progress of the software developer, together with the availability of better
should provide direction on how many staff should attend a review, how a review is set technology such as software tools, new programming languages and software development
up, its ideal duration, what documentation is required, and what documentation needs methodologies, should mean that aspects of the quality system will be incomplete or
to be completed after the review . out-of-date . Because of this, a quality system should be periodically reviewed during its
lifetime, with the review periods being shorter during its initial life . This is a requirement
which many external quality standards place great emphasis on .
Self Assessment Question 1 .8 Why do you think late detection of an
error leads to major costs?
1 .5 QUALITY AND THE QUALITY SYSTEM
Solution An error discovered late in a project can lead to a major resource
expenditure in respecifying, redesigning and reprogramming a system . If The last theme-that of the flexibility of the quality system-leads naturally on to a
the error was discovered at the time it was made all that might be needed description of how quality systems work . The aim of this section is to describe the re-
would be a small change in, say, the requirements specification . lationship that a quality system has with software development, and provide the reader
with some terminology which will ensure that the remainder of the book can be properly
accessed . At the heart of the application of quality assurance there is something called a
quality system (sometimes known as a quality management system) . This consists of the
1 .4 .6 The importance of the requirements specification
managerial structure, responsibilities, activities, capabilities and resources which ensure
One of the key documents in a software project is the requirements specification . It is used that software products produced by projects will have the desired quality factors that
by the system designer when deciding on a software architecture, by the staff responsible both the customer and the developer decide will be built into them . This means that a
for writing user documentation, by testing staff to generate tests, and also by project quality system encompasses activities such as :
managers when predicting the resources on a software project . This importance should
be reflected in the amount of effort which the quality system devotes to the requirements • The auditing of projects to ensure that quality controls are being adhered to .
specification : its form, the way it is developed and the way it is validated . All too often • The review of the quality system in order to improve it .
I see quality systems which have excellent standards for program code, but perfunctory • Staff development of personnel employed within the quality assurance area .
ones for the requirements specification . • The negotiation of resources which enable staff who carry out quality assurance ac-
tivities to function properly .

1 .4 .7 The dynamic nature of the quality system

• Providing input into development-oriented improvement activities ; for example, the
adoption of a new notation for requirements specification .
There are two aspects of this . First, when a quality system is used by a manager on • The development of standards, procedures and guidelines .
a project it is often just taken down off the shelf with little consideration of whether • The production of reports to high-level management which describe the effectiveness
some of the facilities offered by the quality system are not needed or whether some of the current quality system .
need to be strengthened . Every software project is different, even those within a narrow • The production of reports to high-level quality management which enable them to
application band . The next section of this chapter will stress that at the start of each put into action activities that aid the improvement of the quality system .
project-usually during project planning-a software developer will need to look at what
a quality system offers and decide whether some facilities need to be dropped or others
These, then, are a selection of activities normally associated with a quality management
need to be augmented before including them in a quality plan for a project . For example,
system .
the quality system of a management information company may be directed towards
The concrete details of a quality system will be held in a quality manual-sometimes
assurance activities for mainstream applications such as retailing and banking where,
erroneously called a quality system . Such a manual will normally contain standards for
perhaps, the customer knows the application area well and is computer-literate . However,
a new project may involve a customer who knows nothing about computers . The project the quality and developmental activities that may be applied to a project . A short extract
from a quality manual is shown below . It shows the directions given to programming staff
manager, in that case, may have to augment the facilities available in the quality system
about the testing of a module in a system :
to keep closer track of the progress his staff are making in eliciting customer requirements .
22 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 23

7 Module testing
One of the most important tasks to be carried out by the programmer is the testing of individual
modules . Normally the module which is to be tested will be one which has been created by the
programmer who is to carry out the test . Each module will have a name and will be found in a file
whose name consists of the name of the module with the extension pas, cob or for depending on
whether the module was programmed in Pascal, COBOL or FORTRAN . The programmer should
carry out two types of test : a functional test and a structural test . The aim of the functional test is
International standards `
to check that the function of the module has been implemented correctly . The function of a module
such as ISO 9001
can be found in the comment which forms the header of the module . The aim of a structural test is
to generate test data which exercises some proportion of a structural metric of the module-usually
100% of statements . Provides guidance on

The relationship between international standards, the quality system and individual
Quality management system
projects is shown in Fig . 1 .1 . International standards such as ISO 9001 provide guid-
ance to companies on how they should organize their quality system. A component of
the quality system is the quality manual which describes the variety of standards and Quality manual
quality controls available to projects . When a project is in its formative stages-normally
during planning-a project manager will identify the quality factors that are important, Standards
and extract from the quality manual those standards and procedures which are necessary Procedures
to ensure that the software product to be developed will contain those factors . These are
placed in a document known as a quality plan, which forms part of the overall project Is used
plan for a software project . to develop
A quality plan will contain a list of quality tasks such as the application of a particular r V + V

system test, when the task is to be carried out and who is to carry it out . The quality plan Project 1 Project 2 Project 3 Project n
will also contain other information such as any quality standards that are to be adopted,
Quality Quality Quality Quality
any quality tools which are to be used and the way in which the quality assurance is to plan plan plan plan
be organised . I 2 3 n
Thus, international standards often-but not always-drive the production of a qual-
ity system which then provides facilities for a project manager to produce a quality
plan-there being separate quality plans for each project undertaken by a software de-
veloper . Figure 1 .1 External standards, a quality management system and quality plans .
As an example of the latter parts of this process, consider a software project whose
manager decides that portability is to be a major quality factor for a particular product . documentary evidence that the quality factor is present . Some examples of quality con-
The manager will consult the quality manual for those parts which address portability trols are :
and incorporate them as quality controls in his or her project . There are a number of ways
in which the quality manual can provide facilities for ensuring portability ; for example, • A system test which checks that a particular function has been properly executed .
by providing standards for : The documentary evidence would be a test record .
• The code review which, as part of its agenda, has been asked to check for portability
• The language chosen for the project which ensures that non-standard features are not concerns . The signed minutes of such a review would be the documentary evidence .
used by staff . • The execution of a tool which checks program code for adherence to programming
• Portability testing across a wide range of computers and operating systems . standards . The printout from the tool indicating the absence of standards violations
• Checking the output from any one-off or proprietary tools which process program would be the documentary evidence .
code and detect non-portable features . • The execution of a tool which gives a numerical readability index for the requirements
• Development techniques such as information hiding which enable a high degree of specification . The printout from the tool indicating a high readability index would be
portability to be achieved . the documentary evidence .
The first control is associated with the correctness quality factor, the second with the
A quality plan will embody a number of quality controls which check that particular portability quality factor, and the third and fourth with the maintainability factor .
quality factors are present in a system . A quality control is normally associated with
24 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION WHAT IS SOFTWARE QUALITY? 25

1 .7 SUMMARY
Self Assessment Question 1 .9 A dynamic analysis tool instruments
modules written in a third-generation language and provides a report on the
The main points embodied in this chapter are :
execution of the module when it is supplied with test data, for example the
percentage coverage achieved by the data . Could this tool act as a quality
• A quality factor is an aspect of a software product which is important to the customer
control? If so, what quality factor does it address and what documentary
or the developer . Sometimes it is important to both .
evidence would be associated with it
• A quality system is intended to ensure that quality factors identified at the beginning
of a project are present in a completed software system .
Solution There a number of quality factors that could be associated with
• A standard is a description of the way in which a project document is structured and
this tool . The main one is correctness . If the tool provides a report on struc-
how information in the document is presented .
tural coverage achieved by test data, then provided the tests were correct
• A procedure is a set of instructions which describe how a particular software task
the tool would address the correctness quality factor . The documentation
should be carried out .
associated with the tool's use as a quality control would be its printout
• A quality control is an activity which ensures that a particular quality factor is present
reporting on the coverage .
in a software system and its associated documentation . It gives rise to documents
which provide assurance that a particular quality factor is present .
• A quality manual is a document which contains standards, procedures and guidelines
which can be adopted by a project manager for a particular software project .
1 .6 STANDARDS AND PROCEDURES
• A quality plan embodies the standards, procedures and quality controls which are to
be used on a project .
In order to conclude this chapter, and to complete the process of providing a vocabulary,
this section briefly describes the major tools provided in a quality manual : standards,
procedures and guidelines . This topic will be treated in much more detail in Chapter 13 ;
1 .8 FURTHER READING
however, since these terms will be used frequently before that it is worth defining them .
First, a warning . Paradoxically, for such a precise subject as quality assurance,
There are few good books on software quality assurance topics . Dunn and Ullman (1982)
nomenclature is still quite loose and the definitions given here are accepted by a large
and Lewis (1992) are excellent American books ; the latter is particularly good on the
number of software developers and books, but by no means the majority . I shall define
need for independence in the quality assurance process . Ould (1990) is an excellent British
a standard as an instruction about how a document should be laid out on paper or on
tutorial which can be read in an evening and Gillies (1992) is a superb British introduction
a computer screen . For example, a requirements specification standard would specify all
to quality assurance . Crosby (1979) is a book which is not specifically about software,
the sections expected in such a document and how each section is to be structured .
but does address the cultural and managerial aspects of implementing a quality system
By a procedure I mean text which describes how a particular software task is to be in a very effective way. Finally, Boehm (1981) is an excellent book which describes the
carried out . For example, a procedure for programming would describe what standards
economics of software production .
to apply, where to store the source code and object code of a program or module, how
to carry out certain categories of test, and what documentation to fill in when the pro-
cess of programming and testing have been completed . The important point about both PROBLEMS
procedures and standards is that once they have been adopted for a particular project
they have to be adhered to . 1 .1 Which of the quality factors mentioned in the chapter does the following activity address : the
A guideline is text which provides advice on an activity . It is not prescriptive like a application of a static analysis tool to program code which detects certain types of errors such as the
standard or a procedure . For example, a company may have a guideline which governs fact that a variable is declared but never used .
the conduct of project progress meetings which, for example, specifies who should nor- 1 .2 Which of the quality factors mentioned in the chapter does the following activity address : the
mally attend such meetings . Although guidelines offer advice you would normally expect application of a tool to the requirements specification which produces an index that quantifies how
them to be followed by projects most of the time . Often, procedures and guidelines are readable that document is .
1 .3 Which of the quality factors mentioned in the chapter does the following activity address : the use
found combined, where a procedure will mainly contain mandatory instructions but also
of a guideline which ensures that all project documents are stored in a central project library .
provide advice, for example about how many staff should attend a particular meeting . 1 .4 What particular quality factors does the process of acceptance testing address?
The important point about such a combined document is that it should be made clear
1 .5 Service staff are normally part of the maintenance function . They are the people who the customer
what exactly is mandatory and what is advice . usually telephones whenever an error is discovered in a system during its operation . What sort of facilities
It is worth pointing out that some companies collectively refer to standards and should a quality system offer such staff?
procedures as standards, and sometimes as guidelines .
26 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

1 .6 If you were to convince your senior management that software quality assurance needs to be taken
seriously in your company what would be the main points that you would make in a report?
1 .7 Why do you think each project will require its own quality plan? Would it not be simpler to apply
the quality manual to each project?
 2
SOFTWARE TASKS AND QUALITY ASSURANCE

AIMS

• To describe the developmental tasks which make up the software production process .
• To describe the validation and verification tasks which are used to check the correct-
ness of a software system .
• To outline some of the problems with developmental, validation and verification tasks
which software quality assurance is intended to address .

2 .1 INTRODUCTION

Before looking at the various elements that make up software quality assurance it is worth
describing a number of technical tasks which are carried out by software developers . The
main aim of this chapter is to provide the reader with a vocabulary which will enable
subsequent chapters to be easily accessed . In order to describe these tasks two models
of software development will be presented . The first is normally used by developers of
system software, real-time software and telecommunications software, where the data
complexity of the product is small . The second is found within companies which produce
information systems where the stored data can be very complex . If you think that you
are familiar with conventional software development then it is worth skipping to Sect . 2 .5
which describes some of the problems that affect software activities and which quality
assurance is intended to address .

2 .2 A SOFTWARE DEVELOPMENT MODEL

The majority of today's software projects are partitioned into a number of phases-
requirements analysis, requirements specification, system design, detailed design, and pro-
gramming. Each of these phases corresponds to a distinct activity that gives rise to some
end-products, which are then fed into the next phase . The starting point in any software

27
28 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SOFTWARE TASKS AND QUALITY ASSURANCE 29

project is a customer statement of requirements . This is a document, produced by the

customer, which expresses his or her aspirations about a future computer system .
The quality of the statement of requirements varies considerably : customers with
little experience of software systems, specifications and information technology will pro-
duce short, very abstract documents which perhaps only give broad hints about their
requirements . The more sophisticated customer, e .g . a major computer manufacturer
who is subcontracting a system, might develop a statement of requirements which is a
multi-volume work, containing descriptions of every facet of the system to be developed .
The statement of requirements is communicated to the software developer, who then
carries out requirements analysis : the task of attempting to discover what the proposed main

system is to do . To carry out this task, analysts from the developer's staff study any
type ~
existing systems-manual, automated, or semi-automated-which the proposed system
is to replace, interview the customer's staff, and carry out some form of technical analysis flagl
new 1
of the statement of requirements .
When the process of requirements analysis is complete, the analysts write down a inl calcl calc2
description of the properties of the system that is to be developed . This process is known J
as requirements specification, and gives rise to a document known as the requirements
newparams /'
specification, often referred to as the system specification . This contains a detailed de- Ioldparams
scription of all the tasks that a system is to carry out, any constraints such as response
time or memory utilization, and any design directives or implementation directives such
as insistence on a particular file organization for a database . The requirements specifica- newparamset oldparamsct
L
tion also contains a number of appendices that address issues which, although important,
are not directly relevant to the description of the system, e .g . they may detail the main- Figure 2 .1 A system design notation .
tenance support for the system and the training of the customer's staff to be be carried
out . The major feature of a requirements specification relevant to the subject matter of language was FORTRAN or COBOL, then a module would correspond to a subroutine .
this book is that it contains a large amount of detail about what a system should do, i .e.
The system architecture can be specified using a number of notations-probably
its functions . the most common class of notation uses graphics . An example is shown in Fig . 2 .1 .
The requirements specification is a key document in the software project for a number
This represents the design of a system which contains six modules . The lines joining
of reasons . First, it is a major input into the next task in the project : system design . the modules are calls, and the items attached to lines represent the parameters that are
Second, it is used, either in its final form or, more realistically, in outline form, by a
passed between the modules . Thus, in Fig . 2 .1, the module main calls the modules inl,
project manager to estimate the resources that are required for a project . Third, it is calcl and calc2, and the interface between main and inl is the collection of parameters
used by either quality assurance or development staff to develop the system tests and
old, new and type. The arrows indicate the direction in which data is passed between the
acceptance tests which will check the correctness of the system . Fourth, it is consulted modules . Thus, type is passed to inl, and inl passes old and new to main .
by staff responsible for developing system documentation when constructing the user
manual .
Because the requirements specification is such an important document, the software Self Assessment Question 2 .1 Can you think of the type of facility that
developer will expend a large amount of resources on its construction, and will ensure
a quality system should provide for staff who check the correctness of a
that everything is defined in minute detail. Errors in the requirements specification that system design?
remain undiscovered until late in a software project, say during system testing, have
been a major cause of project overruns, and have even led to the cancellation of major
Solution Generally the quality system should provide standards, proce-
projects .
dures and guidelines which enable staff to check that a design meets both
The next stage in the development process is system design. This consists of process-
the functional and non-functional properties detailed in the requirements
ing the requirements specification and producing an architecture which implements the
specification . For example, there should be a procedure which governs re-
functions contained in the requirements specification and also respects the constraints
view meetings which check system design correctness .
specified in that document . The architecture of the system is expressed in terms of
self-contained chunks of program code which I shall call modules. If the programming
language used was Pascal, modules would correspond to procedures or functions ; if the
30 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SOFTWARE TASKS AND QUALITY ASSURANCE 31

The next stage of development is detailed design. During system design the staff allocated
to this task will have specified what each module in the system should do . The task of
the detailed designers is to take this description and fill in the processing details inside
the modules . These processing details are usually expressed in terms of some specialized
detailed design notation, e .g. a flow chart or a program design language . An example of Supplier
part of a module expressed in terms of a program design language is shown below . It
counts up the number of items in the array a that are greater than the value maxvalue .
The notation is very similar to that of a programming language, in that all the standard
control constructs are used, the only difference being that it uses natural language for HasA
the specification of the detailed processing that occurs : I1

PROCEDURE Checkup(a, sum)

BEGIN Employee
Set sum to zero
FOR i FROM 1 TO n DO
Figure 2 .2 An example of an entity-relation diagram .
IF a[i] is greater than maxvalue THEN
Add 1 to sum
2 .3 ANOTHER DEVELOPMENT MODEL
ENDIF
EN DFOR
The model presented in the previous section is a fairly common one . It, or a variant, is
END
usually adopted by the developers of software systems which are dominated by software
processes and have little data associated with them ; for example, those systems found in
application areas such as railway signalling, military command and control, and avion-
The task of detailed design is an optional one . Many software developers prefer to miss ics . These systems are not devoid of data but have relatively simple data architectures
it out, and program directly from the system design . Detailed designs tend to be used compared with application areas such as retailing, banking, and strategic information
by software developers who are interested in portability especially when they are im- provision . It is also a model employed by developers of data-rich systems who use third-
plementing a number of versions of a system using a variety of programming languages . generation languages such as C and COBOL . Developers of data-rich systems who use
A detailed design represents a good, language-independent description of what a sys- fourth-generation languages usually adopt a slightly different life-cycle model in which
tem does, and can be easily hand-translated into any programming language-be it an the data architecture is considered well before the development of a program architec-
assembler language or a very high-level language such as Ada . ture . Such developers use a variety of notations to characterize the data in a system . One
After detailed design, programming begins . This involves the programmer taking a of the first notations often produced is known as an entity-relationship diagram, often
module specification, which is either extracted from a detailed design-assuming such a abbreviated to an E-R diagram . This describes the relationships between the entities in
document is constructed--or from a system design if it is not . If a developer constructs a a system . An example of an E-R diagram is shown in Fig . 2 .2, and describes the data for
detailed design, the process of programming is easy : all it involves is a hand-translation of a simple system which administers company cars .
the detailed design into the program code of the chosen implementation language . How- This simple diagram describes the fact that the system is made up of cars, employees,
ever, if the developer has not constructed a detailed design, then programming is a more and the service records of cars . Each arrowed line represents the fact that an entity is
intellectual task, which involves selecting an algorithm that carries out the processing related to another entity . The name of the relationship is written close to a line ; thus,
required by the system design . the vertical line labelled IsDriven between cars and employees describes the fact that a
This marks the end of the development activities in the software project . Fortunately, car is driven by an employee . A relation can be one-to-one or one-to-many ; in the former
this is not the whole story. Overlaid with all the developmental activities that have been case only one entity is associated with an occurrence of another entity, while in the latter
previously described, there is the continual application of a series of activities which check case one occurrence of an entity is associated with a number of occurrences of another
that the developer is carrying out software tasks correctly, and that the system which entity. Thus, in Fig . 2 .2 the fact that only one employee is associated with one car is
is being specified and programmed meets user requirements . These tasks are collectively indicated by the characters 1 :1, and the fact that a car supplier will supply a number of
known as verification and validation and are described in some detail in Sect . 2.4 . cars is indicated by the characters n :1 .
Another notation which is produced by the developers of data-rich systems is the
32 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION
SOFTWARE TASKS AND QUALITY ASSURANCE 33

Table 2 .1 A relational table

Worksno Name Salary Grade
1244 Cleland 45000 A
2894 Lee Yok 23400 D
8871 Roberts 12500 E
Car 3349 Rolands 12800 E
1900 Ince 23450 D
4545 Williams 12500 E
1444 Dickins 23450 D
Obtain Hire body Dispose 2928 Hoare 34000 B
9876 Khan 12500 E
4588 Rimbaud 35000 B
2358 Proust 25000 D
Hire
changes

moment as the functional specification will be more volatile than the specification of the
Customer data ; this means that if the functional specification is delayed, then the developer's staff
Maintenance
hire do not prematurely commit themselves to a functional specification which may change
drastically during the early stages of a project .
For those developers who do wish to commit themselves to a functional specification
Hirings at this stage it would either be expressed in terms of text or in some graphical notation
such as a data flow diagram . Once the requirements specification has been completed,
the next stage is to design the data and, in a fourth-generation language environment,
Hire Return provide a specification of the programs which are to access the data . The architecture of
the data is usually expressed in terms of tables known as relations . An example is shown
in Table 2 .1 .
Figure 2 .3 An example of an entity life-history . This table represents employees in a rather simple personnel management system ;
each line of the table : represents an occurrence of an employee . Each employee occurrence
entity life-history . An example of such a diagram is shown in Fig . 2 .3 . is distinguished by a field known as a key field which uniquely identifies the employee .
This shows the ordering of actions on the entity car, which is an entity in a system In the case of Table 2 .1 the key field is Worksno . A number of these tables will be
for a car hire company. The top level of the diagram gives the name of the entity . The developed, and each will contain linking information ; for example, another table in the
next level states that a car will first be obtained, it will then undergo hirings, and finally personnel management system might contain employee performance figures . This table
will be disposed of . The box marked hire body describes what the hiring part of the would contain a field which held the unique works number of each employee .
system entails : it first involves customer hirings, followed by maintenance on a car . The Once these tables have been completed, the specification of individual programs can
box marked customer hire describes the fact that a repeated number of hirings is to take be refined and their action expressed in terms of operations on the tables . An example
place, the asterisk in the box hirings indicates that all the boxes underneath it are to be of such a specification is shown below :
repeated a number of times . Finally, the boxes underneath the box marked hirings show
that a customer hire is always followed by a return . Program PersonnelQueryl
This program retrieves all the employees in the staff table who have a salary greater than a specific
This diagram, therefore, describes a situation whereby a car is obtained, and then amount and who have reached a specific employee grade .
a series of actions consisting of hires and returns followed by a maintenance action is
executed, the whole process being terminated by the action of disposing of a car .
Such diagrams are a good aid to validation and verification : they enable the analyst Another task which is carried out at this stage is to design the human-computer interface :
to discover whether any events in the system have been missed . Indeed, in Fig. 2 .3, an how individual programs will be linked together and how the user will interact with these
event has been missed : that of a customer crashing a car, and the car being written off . programs . This topic is addressed in Chapter 10 .
The final development task is that of programming. Often a fourth-generation lan-
Diagrams like the ones shown in Fig . 2 .2 and Fig. 2 .3 represent a logical view of
the data that is inherent in an application ; they are produced during the requirements guage is used, which interfaces directly to a relational database management system that
analysis and specification process . The functions of the system may be written down at supports tables such as that shown in Table 2 .1 . Such a language enables program code
this stage, although many software developers will delay this part until the last possible and interfaces to be produced much more easily than languages such as COBOL .
34 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SOFTWARE TASKS AND QUALITY ASSURANCE 35

During the development process, validation and verification occurs . This will largely The requirements specification is examined in detail, together with the statement
be similar to that described in Sect . 2 .4 . However, there will be some differences ; for of requirements . The participants' main concern is that the requirements specification
example, in the life-cycle I described first a practice increasingly insisted on by quality actually represents an adequate description of the system that the customer wants . The
manuals is to achieve a coverage of 100 per cent statements during unit testing . With a meeting also addresses problems such as ambiguities and contradictions in the require-
fourth-generation language there is no direct analogue of statement coverage . Also, with ments specification. A review is an error detection process, and not an error rectification
fourth-generation language development there is much more of an opportunity to carry process, and hence there is little discussion about what needs to be done in order to
out prototyping: the early development of a software system which is used to improve improve the specification-all that is produced is an action list of errors that have been
tasks such as requirements elicitation and the selection of appropriate designs . discovered, and which need subsequent attention .

2 .4 VERIFICATION AND VALIDATION Self Assessment Question 2 .3 Reviews are a highly effective validation
or verification process . Why do you think this is?
Verification is concerned with checking that a particular task has been carried out cor-
rectly, e .g. that a programmer has correctly implemented a detailed design . Validation is Solution The main reason is that it brings a high degree of independence
concerned with checking that a system meets customer requirements ; system testing- to checking . Many of the staff involved in a review will have no vested in-
executing a system in order to check that it meets customer requirements-is an example terest in the correctness or otherwise of the item being reviewed and hence
of validation . There are many verification and validation activities which occur during a they will have an objective viewpoint .
software project . In this section they will be described in the order in which they occur
in the conventional software project .
Prototyping involves the early production of a version of a system that can be used as a
learning medium by both the developer and the customer . There are a number of ways
Self Assessment Question 2 .2 Is unit testing, sometimes known as mod- of developing such a prototype . If the application is for a commercial data processing
ule testing, a verification or validation activity? system, then a fourth-generation programming language would, almost invariably, be
used . Such languages enable large amounts of functionality to be expressed in a small
Solution Unit testing involves the checking of a module to ensure that amount of programming code . Other techniques for prototyping involve the use of very
the task of programming has been carried out correctly . Hence it is an ex- high-level languages such as LISP and PROLOG, the UNIX operating system, relaxing
ample of verification . the quality assurance standards on a project, or only implementing part of a system .
Prototyping is a highly effective process for carrying out requirements analysis . Re-
quirements specifications are becoming so complex and large that both the customer and
During requirements analysis and requirements specification two validation activities can the developer have major difficulties understanding them . A prototype represents the
be carried out and one can be started . The first two activities are the execution of the most concrete manifestation of a software system that is possible, and is able to give the
software requirements review, and prototyping ; the third is preparing for the system customer an exact and complete idea of what will be delivered .
testing and acceptance testing phases . Another validation process that occurs towards the end of the requirements speci-
A review is a meeting of a number of staff who spend some tim o more than two fication is the process of deriving the system tests and acceptance tests . These are the
hours-reading a document or a segment of program code and checking it for correctness . final checks on a system which confirm that it meets user requirements . The system and
Reviews are highly effective : staff who attend such reviews tend to bring a fresh mind to acceptance tests are carried out at the end of the development of a system after all the
a problem, and are able to check a document or program code more thoroughly . Many correctly programmed modules of the system have been brought together . The system
programmers, analysts and designers find it very difficult to detach themselves from tests are a set of preliminary tests carried out within the developer's environment as a
what they produce, and hence find it very difficult to detect errors . The technical review last check: that acceptance will not be too much trouble . The acceptance tests are carried
overcomes this problem in that staff who are completely detached from a document, or out in the environment in which the system is to be installed each test being witnessed
software product, do not have any misconceptions about that product . by the customer or a representative . These tests are crucial since, if the system fails an
Normally a review involves between three and five members of staff . In a require- acceptance test, the customer has every right to refuse to take delivery of a system and,
ments review, which is normally attended by five staff, these would normally be the hence, to pay for it . Also, the failure of an acceptance test often involves the developer
analyst who produces the requirements specification-or more realistically, the part of in a large amount of respecification and redesign .
the requirements specification-that is being reviewed ; a member of the developer's qual- Because the acceptance tests are so critical, the developers carry out a series of
ity assurance department ; a member of staff from another project ; another analyst ; and preliminary tests in their workplace . These so-called system tests have, as a subset, the
a customer representative . acceptance tests, but also contain a large number of other tests which give the developers
36 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SOFTWARE TASKS AND QUALITY ASSURANCE 37

considerable confidence that there will be no embarrassment during the acceptance test An important activity commenced during system design is integration testing. During
phase . The major difference between the system tests and the acceptance tests is that the later stages of the design process the developers decide on an integration testing
items of equipment, such as a nuclear reactor or an aeroplane, will not be available in strategy. Integration testing occurs during the integration process . The developers build
the developers' workplace and, hence, will need to be simulated-either by software or up a system a few modules at a time ; after each addition of modules, a series of tests are
by special-purpose hardware . carried out in order to check that the interfaces between the integrated modules and the
During the latter stages of requirements specification the developers will start to system that is being built up are correctly implemented .
develop the system and acceptance test suite . By the end of the system design phase There are a number of decisions that software developers have to make about inte-
they will have a description of the tests expressed in outline form . An example of such gration and integration testing . First, they have to decide on an overall strategy . There
an outline is shown below . It represents some of the tests for a stock control application : are essentially three options for integration : top-down integration, bottom-up integration,
and inside-out integration . Each has its advantages and disadvantages . For example, top-
TEST 1 .4 This test checks that when an order is processed by the system and the order is for an down integration enables an early, albeit partial, version of a system to be available, but
item that is out of stock then a suitable message is displayed at the originating VDU . is not very good at detecting errors in modules that lie at the bottom of a system .
TEST 1 .5 This test checks that when the query command is processed by the system the correct
price of the item that is queried is displayed on the originating VDU .
Self Assessment Question 2 .4 Can you think of any other advantage of
TEST 1 .6 This test checks that when the back orders command is processed by the system the
list of orders which cannot be currently satisfied is displayed . top-down integration?

Solution It enables early functional testing to be carried out in advance of

There are a number of reasons why the system test process is started so early . First, it
helps the developer's analysts in carrying out requirements analysis . One of the toughest system testing and hence reduces rectification costs.
questions to ask of a statement in a requirements specification is : `How do I test that?'
Many analysts have reported to me that they carry out their job more effectively if there
Another decision that has to be made concerns the granularity of integration : the number
is a tester continually looking over their shoulder at the requirements specification, asking
of modules that are to be added at a time . The developers may decide on an overall figure ;
difficult questions about how functions and constraints are to be checked during system
for example, that three modules are to be integrated at a time . Or they may decide on
testing . It is an excellent way of detecting functions and constraints that are expressed
integrating a variable number of modules at a time, for example, they may integrate
ambiguously.
one module where that module is very complex, or 20 where the modules are relatively
Second, the early development of the system and acceptance tests enables the early
simple .
prediction of the resources required for these activities . The high-level management of
The developers also have to decide whether to execute any early system tests during
a software company would look askance at a project manager who asked for staff for
integration . The developer may find that after carrying out a number of integrations,
acceptance and system testing a few days before those activities started . They require
enough of the system has been constructed for some system tests to be executed . It is
as much notice as possible . A set of outline system tests provides the basis for at least a
often a good idea to carry out these tests : a maxim for all software developers is that if it
rough estimate of project resources early on in a project .
is possible to check something early in the software project, then do so, and save yourself
Finally, many software projects either require special-purpose testing tools, or the
the considerable redesign and respecification costs which would arise if the checking was
development of large test data suites . Only by deriving the system and acceptance tests carried out later .
early will the developers be able to anticipate these needs well in advance of requiring
By the end of the system design phase the developers will have constructed an
them .
integration and test plan. This plan will specify the order of integration, what modules
After requirements analysis and requirements specification is complete, system design
are to be integrated, the degree of testing that is to be carried out, and a list of any
begins . The main validation and verification activity associated with this is the design
system tests that are to be exercised .
review . The conduct of such a review is similar to that of the requirements review, the
As early as the latter stages of the system design process, coded and tested modules
major differences being that the customer is not usually invited to such reviews, and the
will start emerging from the project-usually from subsystems that have been designed
nature of the questions asked will be different .
early. The process of testing which these modules undergo is known as unit testing. This
A design review concentrates on two types of issues . First, it addresses issues con-
involves the programmers who programmed a module developing test data which checks
cerning the degree to which the system design meets the description of the system con-
out the function of the module . For example, if the module sorted an arbitrary collection
tained in the requirements specification . Second, it concentrates on structural issues : is
of integers, the programmers would check the module with one integer, 50 integers,
the specification of a module's function expressed clearly and correctly ; are there any and a large number of integers ; they would also select a sequence of integers that was
features of the design that will give rise to problems during software maintenance ; and
will the design produce a system that will satisfy memory constraints? already sorted, one which was already sorted except for one integer, and a sequence that
was sorted in reverse order . After this functional testing is complete, the programmers
38 SOFTWARE QUALITY ASSURANCE--A STUDENT INTRODUCTION SOFTWARE TASKS AND QUALITY ASSURANCE 39

check that their test data has achieved a high coverage of a structural element of their • The requirements specification which is so badly structured that large amounts of
module-typically, they might check that their test data has exercised all the statements unnecessary resource is expended by both the customer and developer in reading the
in a module and a very high proportion of branches . specification .
The tested modules are then placed in a project library and withdrawn from this • The requirements specification which does not contain the full set of functions de-
library during integration testing, as and when they are needed . Eventually, after all the manded by the customer, the result being the delivery of a particular system which
integrations have taken place, and all the integration tests are complete, the final system is functionally deficient .
will emerge from development . The only validation and verification activities that remain • The liaison with the customer being so badly organised that changes to the require-
will be system testing and acceptance testing . By this stage of the software project the ments specification are not adequately documented . This often results in major ar-
outline system tests, which, you will remember, were generated during the late stages of guments between the developer and the customer during acceptance testing .
requirements specification and the early stages of system design, will have been expanded
until they have become step-by-step descriptions of the tests to be carried out . In this
state they are known as test procedures . An example of a test procedure is shown below . Self Assessment Question 2 .5 What facility might a quality system
Ideally, such procedures can be given to staff who have little knowledge of the system provide to ensure that the problems associated with the fourth bullet point
which they are to test : above are eliminated?

TEST-PROC 3 .4 Attach test file ATEST14 .TST to the system stored in file SYSV3 .OBJ . Execute Solution The main facility would be a procedure which described how
the system and type in a variety of product numbers between 1 and 1000 when prompted by the communication between the customer and the developer is carried out . For
character = . If the product number is not found in the product database then a message, error- example, it might specify the documentation to be filled in when a tele-
unknown product, will be displayed . A list of the product numbers that are in the test database can
be found in the file PRODT .TXT : use this to check on the correctness of the response .
phone call is made which affects the requirements specification .

The test procedures for the system tests are then executed against the final system . If
there are any errors the system is debugged, the errors rectified, and the test repeated, 2 .5 .2 System design
together with any tests that exercise the program code which was modified when the I System design is still quite an important activity, although less so than requirements
error was rectified . Eventually, the system will have passed its system tests, and all that specification . Unfortunately, it still suffers from major problems . A selection is shown
remains is the execution of the acceptance tests . By the beginning of the system test
below :
phase the customer, in conjunction with the developers, will have selected the subset
of the system tests that will be the acceptance tests . These tests will be applied to the • A design being produced which is not adequately validated . This would result in the
software in its operational environment . If the developers have done their job correctly, programmers producing a system which did not meet some of its requirements .
then the execution of the acceptance tests will be a formality!
• A design being so poorly specified that the programmers are unable to understand
the function of individual modules in the system and so produce incorrect code .
• A design so badly expressed that the maintainers are unable to understand the com-
2 .5 SOFTWARE PROBLEMS AND QUALITY ASSURANCE
ponents of the design and either create errors or consume major amounts of resource
in making correct amendments .
The aim of this section is to describe a number of the problems that affect some of
the developmental, validation and verification activities described previously and which
software quality assurance is intended to address . 2 .5 .3 Programming
Programming is an activity for which errors are less serious . However, programming
2 .5 .1 Requirements analysis and specification errors can result in major amounts of resource being consumed during system testing by
staff trying to track down coding errors which manifest themselves as failed system tests .
The requirements analysis and specification process is one of the most important on the
Some of the problems with programming are shown below :
software project . Unfortunately, it suffers from major problems . A selection is shown
below : • The programmer not providing enough commenting to enable maintenance staff to
understand the function of a module .
• The requirements specification which is so ambiguously written that the software • A poorly structured module being produced which is difficult to understand .
developer misunderstands the customer's requirements and delivers a system which • A programmer choosing poor names for variables which do not express the role that
only meets a fraction of the requirements . the variables play in the module .
40 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SOFTWARE TASKS AND QUALITY ASSURANCE 41

• The programmer storing a module away in a file which is difficult to find by staff who rapid change or those which have been specifically developed for new or unusual software
have to maintain the system in which the module is contained . technology.
The concluding section looked at a small selection of the problems that bedevil
software projects . The aim of any quality system is to eliminate or at least severely
Self Assessment Question 2 .6 What facility might a quality system pro-
mitigate these problems by means of tools such as reviews, procedures, standards and
vide to ensure that the first three bullet points above are eliminated? guidelines .
Solution A programming standard would provide direction on comment-
ing, structural issues and variable-naming conventions . 2 .7 FURTHER READING

It is important that anyone learning about the software quality assurance function have
2 .5 .4 System testing some knowledge of the development process . If they do not, they soon lose the respect
of development teams . The two books that I recommend are Pressman (1994) which is
System testing is an activity which is of prime importance . It aims to check out a system far and away the best general book published on software engineering, and A1cDermid
before acceptance testing is started . Problems during system testing can lead to delays (1990) . The publication of the latter was the book event of 1990 ; it is a superb reference
in acceptance testing ; with the almost invariable delay in delivery of a system to the book which covers the vast majority of software topics that the student or the professional
customer . Some of the problems encountered with this activity are shown below : need to know about-and even some that they do not .

• Poorly specified system tests used by the system testers, who then produce an inap-
propriate or even incorrect test . This often results in the real functions of the system PROBLEMS
not being tested until acceptance testing starts .
• The wrong system tests being specified which check out properties that a system 2 .1 Examine the list of problems in Section 2 .5 that arise during system design and describe the specific
should not have . components of a quality system that you would use to eliminate them .
• Poor planning giving rise to the testers being available for testing but the test 2 .2 Examine the list of problems in Section 2 .5 that arise during system testing and describe the specific
databases having not been produced . components of a quality system that you would use to eliminate them .
• A lack of cross-checking between the system test descriptions and the requirements 2 .3 Throw-away prototyping is a term used to describe the process of quickly building a working model
specification . This might lead to the system test process not checking important of a system and then continually showing it to the customer and modifying it until he or she is happy
with the result . Once the customer is happy, the prototype is thrown away and conventional development
components of a system . starts using a description of the prototype . What problems would you expect with this process?
2 .4 A technical review is a meeting of staff who examine a document- usually a requirements specifi-
All these, and many more, problems can be eliminated or at least severely limited by a cation or a design-and check it for correctness . What problems do you think would arise in a technical
good quality system . For example, a good standard for the expression of system tests review where a system design was examined?
would eliminate many of the problems encountered when system testing staff read test 2 .5 A software developer uses the system design notation shown on page 29 . What do you think should
descriptions . The aim of the rest of this book is to describe how the detailed components be in a standard used to describe how such a notation should be displayed?
of a quality system address these problems . 2 .6 What do you think should be in a programming standard for a language such as Pascal?

2 .6 SUMMARY

The main aim of this chapter has been to describe developmental activities and provide
the reader with two views of how software developers organize their software processes . In
doing this, the chapter has also provided a vocabulary of developmental, validation and
verification terms . It is worth pointing out that the two models described in this chapter
are ideals ; for example, the model described in the first section is for large systems,
often split into a number of parallel developments, with a number of separate teams
carrying out the development of subsystems . Also, it is worth pointing outt that there are
a number of other life-cycle models which are available ; for example, those which cater for
 PROJECT PLANNING 43

• The estimation of the overall cost of a project .

• The specification of the risks inherent in a project .

3 •

•
The specification of how a project might cope when events associated with the most
probable risk areas occur .
The development of a suitable project organization which will effectively deliver the
PROJECT PLANNING required software system .
• The development of subcontracting plans which specify how the activities of the
subcontractor are to interface with project tasks .
• The selection of subcontractors .

There are a number of problems that occur during project planning that a quality system
needs to address . However, before describing these problems it is worth making a point
about the development of a quality system-or rather, the quality manual .
Many developers ask me how they should go about developing standards and proce-
dures . Many ask, for example, whether the process should be driven by external standards
such as ISO 9001 : whether the first step to take is to read such external standards and,
based on a reading of those documents, develop the standards and procedures necessary
to implement the standard . The answer I give is that the best starting -off point for the
AIMS development of a quality manual is not external standards, but a consideration of the
errors and problems which have afflicted projects in the past . As you will see in Chapter
• To describe the general process of project planning . 14 the main external standard, ISO 9001, is written at a rather abstract level, and does
• To outline the role of risk and risk planning on the software project . not provide specific guidance on particular topics such as testing . A better strategy to
• To describe how a quality plan is developed . adopt is to look at all the problems that have been encountered in previous projects,
• To outline how the form of a particular configuration management system is decided attempt to predict future problems, and then base your standards and procedures on
upon . eradicating these problems . In my experience, companies who do this develop excellent
• To describe the two main tasks that make up the project planning process : costing quality systems which, almost invariably, meet external quality standards such as ISO
and task identification, and outline the quality implications of these tasks. 9001 .
Many of the problems which arise during planning or which occur later on in a
software project because of poor planning, are detailed below :
3 .1 INTRODUCTION
• An event associated with a risk occurs ; for example, the late delivery of hardware,
Project planning is one of the first steps in a software project and one which, in my with not enough information being available on alternative actions which enable the
experience, is poorly carried out . There are a number of planning tasks which the quality project manager to react efficiently to the event, and put the project back on course .
system must address via standards, procedure's and checklists . They are : • Inadequate liaison with the customer over quality factors, leading to major reworking
later in the project when missing or incompletely specified quality factors need to be
• The identification of the quality factors which are relevant to the software system that implemented .
is being developed . These quality factors will either be extracted from the customer • Project costing being badly carried out, leading to an underestimate in the project
statement of requirements or from discussions with the customer . budget . This often manifests itself in a truncated testing phase due to the project
• The development of a quality plan which details how the developer is going to en- manager trying to save costs at the back-end of a project .
sure that the identified quality factors are going to be met and validated within the • Inadequate information available about project progress . One of the most frequent
software which is to be developed . telephone calls received by a project manager comes from his or her manager ask-
• The estimation of the resources for the project, not only human resources, but also ing about progress according to plan or according to budget . Many projects are so
equipment, software such as tools, and laboratory facilities . badly planned that such information is often not immediately available, or, at best, is
• The identification of the tasks which make up the project and the specification of the available only after a number of hours spent searching for inadequate documentation .
temporal relationships between the tasks ; for example, whether one task has to be • Skill mismatches between staff allocated to a project and the tasks carried out . This
completed before another or whether two tasks can be carried out in parallel . often leads to tasks which are poorly performed .
• The estimation of the resources required for each task that has been identified as
contributing to the development of the software . i
42
44 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROJECT PLANNING 45

• A poor project organization which is not the most efficient for the software being de- 3 .2 RISK ANALYSIS
veloped ; for example, adopting a life-cycle which is intended for conventional software
development for a new development method such as prototyping . Every project should carry out a risk analysis . There are a number of reasons for regarding
• The selection of a subcontractor who is not capable of producing the hardware or this activity as very important . First, by estimating the risk inherent in a project a
software that is contracted for . E
software developer will have a good idea of what financial contingency to build into the
• A project plan being produced which does not interface properly with project plans budget of a project in order to cope with those risks which could manifest themselves
produced by subcontractors, or does not mesh with the project plan produced by the I during the project . Second, by carrying out a risk analysis a project manager is able to
customer for liaising with the software project . devise alternative actions which could be taken in order to minimize the effect of the risk .
• Activities in the project plan being described at too high a level ; for example, an For example, a project manager may have discovered during the process of subcontractor
activity might be described as `Develop subsystem A' . This high level of abstraction evaluation that there is a finite but quite high risk of the chosen subcontractor not
does not provide adequate facilities for the project manager to monitor progress on developing some software on time . In this case, the project manager may decide to issue
a week-by-week basis . a contract to the software developer which includes penalty clauses . If invoked, these
• Documentation on project assumptions being omitted ; for example, statements such penalty clauses would generate revenue at a later stage of the project ; this revenue can
as `the developer will assume for the basis of planning that the customer will read, and then be used to employ more staff in order to speed up activities such as programming
either sign-off or query, major project documents such as the requirements specifica- and system testing . The top ten risks in a project are, according to Boehm (1989)-
tion no later than two weeks after receipt' being omitted . If these assumptions are not
documented somewhere in the project plan, then their absence could lead to major • Staff deficiencies . The project may be staffed with personnel who may not be techni-
time slippages, with the blame for slippage being ascribed solely to the developer . cally equipped for the task assigned to them, or may have not performed optimally
• The costings for a project omitting factors such as travel, computer time or consul- on previous projects .
tancy fees . • Unrealistic schedules and budgets . For example, the project may have been given an
• The assumption that a resource will be available at a particular time, and that full unrealistic budget prior to the process of calculating what the budget will be-an
use can be made of that resource . For example, a common problem with mainframe occurrence which often happens, particularly with in-company software development
computer projects arises from the fact that the project manager has assumed the departments .
availability of the computer at a certain time, without checking that no other teams • The wrong software functions may be developed . This may occur as a result of inade-
are using the computer for resource-intensive activities such as system testing . quate liaison with the customer . There can be many reasons for this : the customer may
not be readily available ; there may be a lot of substitution of customer representatives,
i
These, then, are many of the problems afflicting the project planning process . The re- with the analysts meeting one customer representative one week and another the next ;
mainder of the chapter discusses project planning activities and how they should be or the customer may be totally ignorant about information technology or even the
carried out . The final section summarizes the standards and procedures necessary to application area .
support this best practice . • The wrong kind of interface may he developed. One reason for this may be the fact that
the customer may not allow the analysts to talk to the real users of the system and
confines discussions about human-computer interface issues-such as the capabilities
Self Assessment Question 3 .1 What facility could a quality system offer of users-to staff who, although they are employed within the department in which
that overcame the problem of activities being expressed at too high a level? the software is to be installed, would not necessarily have been in contact with the
(Bullet point 9 above) application for a number of years .
• Over-engineering. Here there may be a temptation to cram too much functionality
Solution First, the standard that covered the specification of tasks could into a system, or to produce an exotic interface .
describe some hierarchical notation . Second, if the quality system provides • Requirements volatility . This is where requirements for the system may change during
standards and procedures for reviewing the project plan, then a checklist the course of a project . There are a number of reasons for this, many of which are
for staff involved in the review might include a reminder to check for too unavoidable : inadequate analysis may have been carried out, the results of which are
abstract a task specification . only discovered after the requirements specification has been frozen ; the underlying
hardware or operating system software on which the software is to be built may
change ; the business circumstances of the customer may change, for example a new
management team may be put into place who require new data from the system being
developed ; and, finally, the outside world may change, for example new tax laws may
require a change in an accounting package .
 46 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROJECT PLANNING 47

• Deficiencies in externally developed items . This is where a subcontractor-either for Project Management
hardware or software may not have the degree of quality assurance that is present For the project manager answer the following questions and enter your answers on the sheet pro-
vided .
on the main project to which they are subcontracted .
• Shortfalls in externally performed tasks . This not only includes software and hardware 1 . Has the project manager been identified yet?

tasks carried out by a subcontractor, but also shortfalls in tasks carried out by the 2 . Is this the first time that the project manager has managed any project?
customer and any providers of external facilities such as documentation services . 3 . If the project manager has had no experience in managing a project before, has he or she had
• Performance problems . Real-time response is becoming an increasing problem on experience in a supervisory capacity such as being a senior programmer?
projects-not just real-time projects-but those which involve the development of 4 . Is this the first time that the project manager has managed a project in this application area?
software that accesses large external databases . 5 . Has the project manager undergone the PM3 modules of the company management programme?
• Strains on current computer technology . This means that the software developer may
use a technology which has not yet become state-of-the-art, but which lie or she as- The Customer
sumes will provide the same profit and lack of problems as existing mature technology. For the project manager answer the following questions and enter your answers on the sheet pro-
vided .
The assessment of risk can be a highly complicated numerical exercise . However, for 1 . Is this the first time that we have contracted a software system for this customer?
the vast majority of software developers a simple risk analysis can be carried out using 2 . If this is not the first time that we have contracted a software system for this customer, is this
a checklist . Such a checklist would ask questions about the ten areas above . Typical the first time that we have contracted for this department?
questions are shown below: 3 . Have customer liaison staff been identified yet?
4 . Has the customer agreed that, except in exceptional circumstances, the same liaison staff will
• Is this the first time that the project manager has managed a project? meet our staff?
• Is this the first time that the project manager has managed a project in this applica- 5 . Would you consider that the customer is fam liar with information technology and the use of
tion area? computers?
• Are any of the items developed by external subcontractors critical items, for example 6 . Is this the first time that the customer has gone outside his or her company for software devel-
in performance terms? opment?
• Is there a wide variety of users interacting with the system? 7 . Is the customer liaison team familiar with the application area?
• Is the hardware technology upon which the application is to be run new?
• Have we achieved a similar response-time on comparable projects in the past?
• A very sophisticated quality system would ask the project manager to submit his or her
Is this the first time that the customer has been exposed to computer technology?
answers to the questionnaire to a software tool or a spreadsheet which would give an
overall risk factor value .
Self Assessment Question 3 .2 If the answer to the first bullet point A good quality system should insist that once such a questionnaire or checklist has
above is yes, then what actions might be taken by the management of the been filled in, the next step is for the project manager to list the most likely risks and
software developer? describe what actions are to be taken to minimize the effect of those risks . For example,
the project manager may have decided that because a particular project delivers a sys-
Solution This depends on the project . If it was very critical, then the tem which has a large database, and the implementation is via a somewhat inefficient
project manager might be assigned to another project and be replaced by fourth-generation language, then the project is risky and a number of risk avoidance
a more experienced manager . If the project was less critical, then the se- tactics need to he put in place . One way to reduce this risk could be to organize the
nior manager who the project manager reports to might monitor his or her project with a simulation or prototyping front-end that would provide valuable infor-
activities a little more closely than normal . mation about potential problems . Another option would be for the project manager to
decide to partially staff the project with some programmers who have third-generation
language expertise so that, assuming the database system allowed access to such lan-
Most of the questions are self-explanatory except, perhaps, the fourth one . The point of guages, critical parts of the system can be recoded in a more efficient way . Another
this question is to gauge the complexity of the human-computer interface : the wider the option would be to plan for the in-house development of an instrumentor which is able to
variety of users who interact with a computer system, the more complex the interface detect those parts of a system that are being executed frequently and are candidates for
becomes . An extract from a risk questionnaire is shown below : optimization . The quality manual should provide some documentation-usually a risk
minimization guideline-which provides the project manager with guidance about the
various options that are available to reduce the impact of risk on a project .
 48 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION
PROJECT PLANNING 49

Some more sophisticated risk questionnaires provide data which can be processed by
a spreadsheet and which enable the project manager to fix on a figure which is then used' Self Assessment Question 3 .3 Can you think of any projects where the
to add contingency to an estimated overall cost of a project . correctness quality factor might not be the prime one?
A final part of the planning of the software project associated with risk analysis is
the determination of how the effects of risk are to be monitored, and the actions which Solution Such projects occur in the finance and banking sectors where a
are subsequently taken reported on . Normally, a quality system will have standards and product may be developed quickly and which requires IT support quickly .
procedures for risk monitoring . The standards will describe documentation which gives Here a customer may tolerate a system with minor functional errors pro-
details of which risks have recently occurred, how important they are, how they are being vided that it is delivered quickly.
resolved, and the effect of the risks on the project .

Once this analysis of quality factors has been completed the project manager then decides,
3 .3 OUTLINE DESIGN in conjunction with staff who carry out the quality function, which quality controls are to
be applied to a project . For example, one quality factor which may be important to the
An important project planning task is outline system design . Many other project planning customer is that of interoperability : the ability to interface effectively with other software
tasks depend on this being carried out correctly . The main reasons for carrying out an systems such as operating systems, word processors and spreadsheets . A customer may
outline design are so that the tasks in a software project can be identified, preliminary have specified that certain files for a financial application should have a format which
sizing studies of the software can be carried out, and the testing philosophy that is to be would enable the user of a spreadsheet package to access them . This means that the
adopted can be specified . project manager may have to specify standards which govern the output of the system,
A later section of this chapter describes the process of task identification and task and employ a series of tests which check that the files can be processed by the spreadsheet,
specification, where the individual tasks that make up a software project are collected for example by exercising many, if not all, of the spreadsheet functions . The quality
together and incorporated in the project plan . Without at least an outline architecture factors, together with their associated controls, are placed in a document called the
of a system it would be virtually impossible to identify these tasks . The development of quality plan. An extract from a quality manual section detailing quality factors, and the
an outline design should be described by standards and procedures . options that should be used in implementing them, is shown below :
Normally, the procedure will describe the analysis that is required to carry out the
design . The standard will describe some design notation for expressing the outline design . 3 . Quality factors

This notation is usually similar, or the same as, that which will eventually be used for full Our company recognizes a number of different quality factors . Part of the planning process involves
design . The outline design standard will usually describe how to express a design in terms the project manager deciding what quality factors are important for a particular system and using
of subsystems or in terms of the first two or three levels of module hierarchy, and will a number of technical facilities to build in these quality factors, together with a number of quality
include a description of the boundary between the user and the system . Some procedures controls which check that the factors have been implemented . The remainder of this document
insist that the project manager also includes a technical implementation strategy in this describes these quality factors and a number of technical options which can be pursued in order to
implement them .
part of the project plan, although many specify that it is a separate chapter .
3.1 Maintainability

3 .4 THE IDENTIFICATION OF QUALITY FACTORS Maintainability is defined as the characteristic of a system which enables changes to be applied
easily-this includes both changes during maintenance and development . Many of the quality stand-
ards and procedures which form part of the company quality manual are aimed at achieving main-
One of the main points made in Chapter 1 was that one of the first jobs that a project tainability, for example the programming standards specified in section 12 .6 of the manual ; however,
manager has to carry out is that of determining which quality factors are important for there are a number of additional options which are available to the project manager . These options
the system that is to be developed . If a feasibility study precedes the project planning are :
part of the software project, then an outline consideration of quality factors would take
place here . However, it is during project planning that this activity comes to the fore . 1 . The use of software tools for the storage of test data . The main software tool which we employ for
this is TransSave which saves menu-based transactions to a file. The quality control which would be
The project manager will, by examining the customer statement of requirements
employed to ensure that this tool has been used is to amend the normal program testing standard
and by meeting the customer, gain an idea of which quality factors are important and to ask programmers to include the T'ransSave file listing within the program testing documentation .
which are not . Correctness will usually be a prime quality factor and, almost invariably,
maintainability will be important . 2 . The use of design metrics . Two metrics are used by the company
: an information flow metric
which measures the amount of data passing through a module or program, and the IF4 metric
due to Ince and Shepperd which measures the interface in a program or module . Details of these
metrics, and the tools which we use, can be found in the Metrics Guidelines Document . The project
manager will stipulate a metric threshold ; no module will be allowed to exceed this threshold
. The
 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

50 PROJECT PLANNING 51

quality control which is used to check that the metrics have not been exceeded is to insist that the tasks . The quality plan would then simply contain the unique identifiers of tasks, together
designer produces a printout of the metric values of the modules in a system as part of the system with their associated quality factors .
documentation .
Whether your project planning standards insist that quality control tasks are detailed
3 . The stipulation that modules carry out only one function . Such modules tend to be easy to in the quality plan, or cross-referenced in the quality plan to the project plan, is really
maintain . The quality control which is associated with this is that design reviews should be asked just a matter of taste . I normally favour the former as it usually means that there is less
to check that all modules in the designed system carry out the one function . turning of pages . It is worth pointing out that during the early stages of a project the
4 . The use of software tools for retesting . The tool that the company uses for this is TestManager . description of quality control tasks will be at a high level of abstraction . Not enough will
This enables tests to be specified in a form such that when a module is changed, a whole test suite be known about the project at this stage for any level of detail to be achieved . However,
corresponding to the module is re-executed . The quality control associated with this is to insist that as the project proceeds, this part of the quality plan should be expanded .
the system testing documentation is amended with a listing of the retest commands used by the An important section of the quality plan will contain the verification requirements .
TestManager tool . These are external manifestations that the system is behaving as the requirements specifi-
cation demands . These verification requirements will be used to produce the final system
and acceptance tests, and work is normally started on these as soon after the completion
In the extract above there is a reference to design metrics . A metric is a numerical value of the requirements specification as possible . Two examples of such verification require-
which reflects a quality factor . These are discussed further in Chapter 11 . ments are shown below :

When the system receives any emergency signal from a reactor monitor a shut-down signal should
3 .5 DEVELOPING THE QUALITY PLAN be sent to the hardware which controls the main inlets and outlets .

The quality plan is a description of how the developer is to ensure that the quality factors When the out-of-stock command is initiated with a correct product identifier, then the system will
identified at the beginning of a project are going to be validated, i .e . what activities are respond with the time periods during which the product was out of stock .
to be carried out to check that a particular quality factor is present in a software system .
It can be seen as a subset of the project plan which deals with verification and validation Both these statements describe events which are of importance to the customer and rep-
activities . resent external manifestations of the behaviour of the system . The first event is detected
The quality plan should : by examining some electronic circuitry ; the second by looking at a VDU . Since these
verification requirements can only be extracted after requirements analysis is complete,
• list the quality factors for a system ; this section of the quality plan will be left blank during the planning process .
• describe the quality controls which are to be applied ; for example, reliability testing,
prototyping and usability reviews ;
• describe the skills of the staff who are to carry out the control activities, - Self Assessment Question 3 .4 Can you remember why verification re-
• describe any special tools which will be necessary ; quirements are extracted at an early stage in a software project?
• specify the standards, procedures and guidelines from the quality manual which will
be adopted by the project ; Solution There are three reasons : first, in order to gain an early idea of
• describe any new standards and procedures, or modified standards and procedures, the resources required for system testing and acceptance testing ; second,
from the quality manual which are to be used on the project ; to provide an early indication of whether special-purpose software testing
• reference any external standards or customer-imposed standards and procedures which tools or hardware testing equipment is needed ; and, third, to provide a
are relevant to the project ; powerful validation of the requirements specification .
• specify, in detail, the individual tasks which implement a quality control : their nature,
duration, when they can be started and the amount of resources needed . Each task
should be cross-referenced to the quality factor which it is intended to validate . An extract from part of a quality manual dealing with the identification of verification
requirements is shown below :
The last item forms part of conventional project planning . Because of this many compa-
nies do not put detailed descriptions of the quality controls in the quality plan, but place 17 Verification requirements
them in a project plan . However, such companies do have some form of cross-referencing A verification requirement is a requirement which is extracted from the requirements specification .
system so that quality control tasks can be easily extracted from the project plan . One It represents some external behaviour of a system which can be detected by the customer, and
way of doing this is to give each project task a unique name in the project plan, but give will eventually be expanded so that it becomes a system test and often an acceptance test . Some
tasks associated with quality controls some prefix which indicates that they are quality examples of verification requirements are shown below :
52 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROJECT PLANNING 53

1 . When an update command is typed with a correct product identifier, the system will display the
number of updates for that product over the last financial year . Project manager
2 . When the out-of-stock command is initiated, the system will produce a hard copy report that
will detail the items which are currently out of stock, together with the number of back orders for
the products .
Senior analyst Senior programmer Senior designer
3 . When the flow command is initiated, the flow meter will be repositioned so that it receives the
maximum flow from the outlets of the nuclear reactor .

Some examples of requirements which are not verification requirements are :

The system should keep an up-to-date queue of messages waiting to be processed .

The system should produce accurate data on the state of the nuclear reactor . Designer Designer Designer

The former is a requirement which has to be satisfied for a system but which can only
be observed by actions which correspond to verification requirements ; for example, by a Quality assurance 1
command which displays the current state of the message queue . The second requirement Testing function function
is at too high a level of detail and requires further decomposition in order to produce
adequate verification requirements .
Figure 3 .1 One popular form of project organization .

Self Assessment Question 3 .5 Is the statement : `The system will re- it is worth the project manager splitting the project up into mini-projects, each of which
spond to a hardware error by producing error messages on the main moni- is managed by, say, a senior analyst, and each of which is responsible for the production
toring console which identify the hardware involved and suggest a possible of a subsystem, with the project manager mediating on design decisions which are made
cause' a verification requirement? by teams but which may affect other teams .
There are two advantages with this form of organization . First, it cuts down on the
Solution Yes it is, albeit at a high level of abstraction . It describes a communicational overhead in a project . As projects get bigger and bigger, the amount
function of the system . of work that staff have to do in terms of attending meetings, reading memos, inducting
new staff, increases, sometimes to the point where it dominates useful work such as
coding a module or designing a subsystem . By organizing the project as parallel teams
with, perhaps, each team using the same office, the communicational overhead can be
3 .6 DETERMINING THE PROJECT ORGANIZATION substantially reduced . Second, it gives technical staff some management experience which
they may not get in a strictly hierarchical project .
Another important task which the project manager has to carry out is to determine how This is just one form of organization ; there are many others . The important point is
a project is to be organized . Most projects tend to be organized as a hierarchy similar to that the quality system, via the quality manual, should provide guidance about the most
that shown in Fig . 3 .1, with staff reporting to senior staff above them who, in turn, report relevant form of organization .
to their seniors, and so on . Figure 3 .1 shows that the hierarchy is confined to development
with quality assurance and testing outside the hierarchy . This is often the way that large
software developers carry out their production, but it is by no means the only way . The 3 .7 DETERMINING THE CONFIGURATION MANAGEMENT
project manager should look at the particular system to be developed and determine the SYSTEM
project organization . Normally, a company will have a preferred organization-usually
some form of hierarchy-but guidance should be provided in the quality manual about Chapter 1 made the point that change is a fact of life for a software developer-not only
other forms of organization . during maintenance but also for the duration of a project . In order to cope with change,
As an example of this, consider a system where the functions can be neatly parti- software developers have a set of standards and procedures in their quality manuals which
tioned into functional subareas ; for example, a system for monitoring and controlling a detail how change is to be handled . This topic is dealt with in more detail in Chapter
chemical plant can normally be partitioned into functional areas concerned with moni- 8 . But since the determination of configuration management practices is a task carried
toring, control, and management information . If this neat form of division occurs, then out during project planning it is worth including some material on it at this point in the
 54 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROJECT PLANNING 55

book . The term configuration management describes the following : 23 Subcontractor evaluation
• The interface between the remainder of the project and the configuration manage- Occasionally we use subcontractors for software development ; normally these subcontractors are
ment system ; for example, how a change which arises from a developmental error is used when we do not have enough experience with a particular programming language or com-
puter. Care has to be taken over the selection of such subcontractors . It is important that the
communicated to the configuration management system . subcontractor(s) chosen for a project develop software which is at least to the quality of the soft-
• The process of deciding whether a change should occur, carried out by a body known ware that your project is to deliver . The following actions and checks need to be carried out :
as the Change Control Board.
• The establishment of baselines . A baseline is a document or program code which 1 . Check that the subcontractor is certified to an external standard such as BS5750 ; certainly, if the
comes under configuration control . Up to the point that a document or a collection subcontractor has achieved BS5750, then there should be little need to carry out further checks on
technical capability. There is still a need, however, to carry out financial checks . If no certification
of program code is baselined, change can occur freely . However, after baselining, a has been achieved, then further questions need to be asked . These are detailed below .
formal process of evaluating whether the change is to be allowed to be carried out,
documenting the change, and checking that only the change has occurred, has to take 2 . Check that any of our projects which have used the subcontractor have been happy with perfor-
place . The items which are baselined are known as configuration items . mance. Details can be found in the project debrief file for each project . This file should contain a
• The updating of the system documentation to reflect the changes that have occurred, completed questionnaire on subcontractor performance .

and the broadcasting of that documentation to parts of the development team that 3 . Ask the software contractor for the names of at least two companies who you can approach to ask
need to know about the change . about performance . A subcontractor's reluctance to give such names should be regarded seriously .
• The proper accounting of change . This would include details of the reason for change, However, you should recognize that there may be circumstances which prevent the subcontractor
when the change occurred, and what other items had to be changed . from giving this information ; for example, it may need clearance from an agency such as the Min-
• Checking that a change which occurred did not include any modifications that were istry of Defence which is not forthcoming . If possible you should ask for the names of more than
two companies and randomly nominate two of them .
riot sanctioned .
These are the main features of configuration management . More details can be found in 4 . Apply the checklist found in Appendix 13 .1 of the quality manual . This will normally require
a day visit to the software developer's premises . This checklist contains 25 questions designed to
Chapter 8 . A project manager has to examine the configuration management system and probe a software developer's technical and managerial practices . Poor answers to more than ten of
ask how it is to be tailored to a project ; for example, who should carry out the functions these questions should be regarded seriously .
of the Change Control Board . Often this is a project manager's responsibility, but for
large projects it may be necessary for the board to contain more than just the project 5 . If the subcontractor _is carrying out a large proportion of work on a project-more than 25
manager. per cent of the monetary worth-then a financial check should be carried out to ensure that the
company is in good health . The body that carries out this check for us is Information Systems and
Another example of tailoring occurs during prototyping . This form of software de- Intelligence, who provide very accurate company financial profiles and are able to pronounce on the
velopment is highly dynamic and relies on rapid feedback from the customer to the financial health of a wide variety of developers .
developer and the rapid implementation of changes . Configuration management systems
which are tailored to conventional phase-oriented projects tend to slow down the proto- There is a reference to BS5750 in the extract . This is the British instantiation of the
typing process considerably, so the project manager has to make the very difficult decision International Standard ISO 9001 . Chapter 14 describes ISO 9001 in detail .
whether to drop the configuration management system for the prototyping phase or to
considerably ease the application of the system .
3 .9 PROJECT COSTING
3 .8 THE EVALUATION OF SUBCONTRACTORS
3 .9 .1 Conventional costing

Large software projects often use subcontractors for the development of software or hard- A vital task during the planning stage is to produce a costing of the project . There are
ware . The quality system should provide procedures whereby the project manager can a number of techniques used to achieve this : some are based on project costing in other
issue calls for bids, and evaluate those bids on the basis of the competence of the sub- industries, while others are based on the modern technique of software metrication . This
contractor to produce hardware or software which is at least of the same standard as subsection will concentrate on the former . Probably the most popular method of costing
that. which is to be developed by a project . A subcontractor selection procedure should a software project is based on an identification of the tasks in the project and a costing
provide guidance on evaluation, and will often incorporate a checklist of questions, such of the individual tasks based on historical data or experience . One of the most common
as whether the subcontractor has an adequate configuration management system . It will means of expressing the breakdown of tasks in a project is the work breakdown structure .
also offer advice about questioning previous customers of the subcontractor . A section of Figure 3 .2 shows a very small fragment from such a structure . It is worth stressing that
part of a quality manual dealing with the evaluation of subcontractors is shown below : the extract shown is very small ; often, a work breakdown structure for a substantial
system will contain thousands of boxes .
 56 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROJECT PLANNING 57

For small projects, work breakdown techniques often give acceptable results . How-
ever, for many projects, this cost is often on the low side . There are two reasons for this :
first, there seems to be a tendency for staff to underestimate the cost of tasks-this ten-
dency seems particularly marked on medium-sized and larger projects ; second, owing to
I a phenomenon known as communicational complexity. When staff work on a project their
Project
work can be split into two components : `useful' work such as programming and designing,
Project Feasibility and communicational work, such as reading memos, attending meetings and inducting
management QA study Analysis
new staff onto a project . As a project increases in size the communicational complexity
I tends to rise exponentially, sometimes to the point where it dominates the real work .
2 3
Work breakdown structures are unable to predict the communicational complexity part
of a project .
In order to cost projects, a quality manual should provide a costing standard for ex-
pressing the costs of various parts of a project and a procedure for calculating the overall
cost and the costs of each task in the project . Normally, these will contain instructions
4 .1 2 .3 for forming a base cost from a work breakdown structure, and then adjusting it based
on a number of approaches . The source of data for the adjustment can be varied . The
Preliminary Full
developer may use a modern, metrics-based costing technique as a back-up and adjust
Prototyping
requirements requirements the estimate based on what the metric-based technique produces . This technique uses
analysis analysis past historical data to predict the future cost of a system . The estimate could also be
adjusted by comparing similar sized projects, or by looking at the risk assessment of the
Figure 3 .2 A fragment of a work breakdown structure. project .
The important point is that the quality manual should offer guidance on the source
of data and what should be done with it . As well as this macro level of comparison, the
There are three important things to notice about a work breakdown structure . First,
quality system should provide directives about micro-level cost comparison, such as simi-
it is hierarchical : tasks at a high level are expressed in terms of tasks at the next level,
larities and differences comparisons . The former involves the comparison of a component
and so on . This reflects the hierarchical structure c .f the project . Second, each task in the
of a system, such as a subsystem, with a similar component of the system or a similar
work breakdown structure is prefixed with the number of the task which is its immediate
superior . This allows a unique numbering to be maintained . This is required because a component of another system . The latter involves looking at dissimilar components and
seeing whether the costs reflect the dissimilarity
project will have a reporting standard which, when a task is completed, will result in a Nowhere is the need for uniform adherence to a quality system clearer than in project
report generated and sent to the project manager ; having unique numbering leaves no
costing . A wise developer who was once very cynical about quality systems-until he
margin for ambiguity or error in the reporting . Third, a project manager, or any other
became a project manager-told me that as long as everybody adhered to the quality
member of staff on a project, given a task number, can trace back to the major tasks
system instructions on costing he was happy, even if the results were not particularly
from which that task is derived . For example, a task 4 .3 .2 .2 can be traced back to tasks
4 .3 .2, 4 .3, and 4 . This is an example of the traceability principle which I described in accurate . At least, he told me, the error in costing would be consistent and the trend in
errors could be discerned and allowed for .
Chapter 1 applied to project planning .
The work breakdown structure is developed during project planning after an initial
system design has been produced, and will be at a relatively abstract level ; it may, for Self Assessment Question 3 .6 If a company used a historical method
example, contain only three levels of task . However, during the initial developmental for costing what would be the managerial implications?
stages of the project it will gradually be refined until tasks are expressed at quite a low
level ; for example, the tasks might be specified at the module level . Solution The main implication is that the actual cost of projects should
The work breakdown structure is produced for a number of reasons . First, it produces be monitored accurately, together with data such as the staff hours devoted
a breakdown of a software project which is in discrete chunks whose completion can be to project tasks such as requirements analysis .
monitored, and hence can be used for progress reporting and cost accounting . Second,
it provides a major input into the costing process . One organized way of carrying out
costing is to develop a work breakdown structure, assign costs to each element, and
accumulate the costs as a total . To this total are added any elements not covered by task
analysis, such as travelling costs, consultant costs, and overheads .
58 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROJECT PLANNING 59

3 .9 .2 Advanced costing and estimation techniques

This chapter would not be complete without briefly describing two advanced methods
of software costing which are based on the idea of measurement and the use of software
metrics : numbers extracted from the products and processes of the software project .
Software metrics are described in more detail in Chapter 11 .
There are two main methods for costing and estimating based on software metrica-
tion : COCOMO (Boehm, 1981) and Function Point Analysis (Symons, 1991) . COCOMO
is a method developed by the American computer scientist Barry Boehm . Boehm pos-
tulated that a software project cost is determined by a number of factors he called cost
drivers . Typical cost drivers are the degree of use of tools on a project and the capability
of the staff on a project . Boehm identified 15 cost drivers and suggested that, for each
project a company undertakes, the project manager estimates each of the cost drivers on
a five point scale together with the size of the system in lines of code . This information,
2 .2 .3 Develop simulator
for all completed projects, is then stored in a historical database . When a project man- 2 .2 .4 Test simulator
ager wishes to calculate the resources required for a new project all that he or she has 2 .2 .5 Install simulator
to do is to estimate the number of lines of code and the values of the cost drivers . This 2 .2 .6 Produce simulator report
3 .1 .2 Review system test suite
information is then processed by a software tool which then examines the historical cost 3 .1 .3 Apply system test suite
database and predicts the resources of the project . 3 .1 .4 Produce acceptance test suite
The second technique . Function Point Analysis, is similar to COCOMO in that a his- 3 .1 .5 Review acceptance test suite

torical database is used . However, the database contains different information . It contains
attributes extracted from a requirements specification . In the first version of Function Figure 3 .3 An extract from an activity network .
Point Analysis, developed by the IBM manager Allen Albrecht (Albrecht and Gaffney,
1993), the attributes extracted from a requirements specification included the number of and often specifies data such as the earliest start date, the latest start date, the earliest
master files, the number of transaction files and the number of enquiries . More modern and latest completion date, and so on .
versions of Function Point Analysis, oriented towards database technology and fourth- The project manager will also specify which tasks have to be carried out serially
generation languages, collect other information such as the number of bubbles in a data and which are to be carried out in parallel . There are a number of graphical devices for
flow diagram . specifying the time sequencing of tasks . Almost certainly the most popular is the activity
The factors that are counted are inserted into an algebraic expression known as a network . In an activity network, tasks are represented by lines which join specific points
function point count . This represents the functional size of a system . For each completed in a software project . A short fragment from an activity network is shown in Fig . 3 .3 . The
project the company stores the eventual cost of the project and its function point count . diagram shows tasks which have to be carried out serially such as tasks 3 .1 .4 and 3 .1 .5,
When a new project is started the project manager extracts the counts of attributes in together with tasks which can be carried out in parallel, such as tasks 3 .1 .3 and 2 .2 .3 .
the requirements specification, calculates the function point count and then uses a tool This is quite a simple version of an activity network ; more complex versions normally
which processes the historical database and, based on previous values of function point include resource information for a task, earliest start, latest start, earliest finish and latest
count and cost, produces a resource estimate . finish information . Many quality systems specify that the tasks on the activity network
use the same numbering as the work breakdown structure, so that there is traceability
between the activity network and the rest of the project plan .
3 .10 TASK IDENTIFICATION Many software developers use automated tools for the processing of such activity
networks . Such tools enable the project manager to examine resource expenditure across
This is the process whereby the tasks that make up a project are identified, specified, the duration of a project and identify, for each task, a time interval known as a float. If
their resources estimated and their timing decided upon . The previous section has al- an activity had a float of n days then the project manager knows that the activity has n
ready described much of this work in connection with estimating . The work breakdown days' slack in which carrying out no progress on that activity will not delay the project .
structure described there provides a good starting point for the remainder of the work . Another important piece of information which such tools provide is the critical path
Costing of each task is normally based on staff experience, for example that a subsystem of a project . This is a set of tasks which form a path through a project such that if
estimated to be 1500 lines of code normally requires 40 person-days . Once each task has any of the tasks were delayed by n days, the whole project would also be delayed by n
been identified and a resource estimate for the task specified, the process of timing the days . Such tasks will require extensive monitoring by the project manager . Because of
tasks is carried out . Here the project manager decides when each task is to be carried out the increasing use of tools to process activity networks many software developers specify
60 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROJECT PLANNING 61

their task identification standards in the format demanded by a particular tool .

Self Assessment Question 3 .7 What is the role of assumptions in this
It is important to point out that both the activity network and the work breakdown
part of the project plan?
structure will be developed in outline at the beginning of a project and will subsequently
be refined as developmental activities occur . For example, on completion of the system
Solution It tells the customer under what conditions the project will be
design, the project manager is able to provide much more detailed information about the
successfully completed . A major reason for describing these assumptions is
various detailed design and programming tasks that occur later in the project .
to inform the customer what his or her acceptable behaviour is in terms of
project success . For example, one assumption might be that the customer
does not substitute any staff in important technical meetings . Customers
3 .11 THE PROJECT PLAN
who do this usually add time to a project when their staff disagree with
each other.
The overall document which describes the process of project planning is the project plan .
A good project plan should have at least five sections which correspond to the following
headings : introduction, project organization, managerial aspects, the technical process,
An important part of this section is a list of dependencies : tasks or activities which cannot
and task aspects .
be started until other tasks are completed ; for example, the task of system design cannot
The introduction should give a brief description of how the project came about ; for
example, it should describe the process whereby the project was contracted for . This part be completed unless the requirements specification has been signed off by the customer .

of the introduction should briefly describe the system that is to be developed-usually This section should also describe how the project is to be staffed : what staff will
be required, what skills are needed and when staff are to be assigned to the project . It
no more than a page of text is required for this . The reason for brevity is that this part of
the project plan will be read by senior management who have very little time to consider should include a description of how work carried out on the project will be monitored and
project documents, and are more concerned with high-level concerns such as whether a how reports on project progress will be presented-not only to the software developer's
project fits in with the business objectives of their company. high-level management but also to the customer .

The introduction should also describe the project deliverables : not only software The fourth section of the project plan should describe the technical solution that will
deliverables which comprise the final system, but also any subsidiary software, documen- be adopted to create the software . This will include any development methods, notations
and software tools which will be used in the project . Often, software developers will
tation, services and training that are required . Subsidiary software usually includes tools
which may have been developed to help produce the software, for example a tool which include descriptions of the documentation that will be used during the various stages of
the project . This section will also normally contain a description of the services provided
generated large files of test data . Normally, subsidiary software will only be specified as
a deliverable if the customer or a third party are to maintain a system . by other parts of the software developer's organization . For example, if the software

In addition, the introduction should describe any important documents which are developer has an independent test department, then its role within the project will be
described here.
relevant to the project . This section will always contain a reference to documentation
initially provided by the customer, such as a statement of requirements, but will also refer The final section-and usually the longest-will be a description of the individual
costed tasks in the project ; when they occur ; the dependencies with other tasks, such as
to documents such as the project contract, and external and internal standards used by
the software developer . The introduction should also include a section which describes whether they are to be carried out serially or in parallel ; and the capabilities of staff who
are to carry out the tasks. The resource requirements for the project should be detailed
any definitions and acronyms used in the plan .
The second section of the project plan should describe how the software project is here, not only on a task-by-task but also on a phase-by-phase basis . The budget for the
project should also be listed in this section of the project plan, together with the budget
to be organized : which life-cycle model is to be adopted, and the developmental tasks
which are to be carried out, and how they are to be validated . This section should expenditure on a week-by-week or month-by-month basis .
Finally, the schedule for the project should be specified . This should be organized
include how the project is organized : which tasks are to be carried out by developmental
around milestones . A milestone is an event which is easy to recognize and provides the
staff, customers and subcontractors ; the managerial relationship between staff within
tick of a clock which indicates that a project is progressing . A popular milestone is one
the project ; the boundaries and relationship between the project and internal agencies
such as the quality assurance department ; the boundaries and relationship between the which is associated with the delivery or signing -off of an important project document such
as a system design . Two types of milestone will be specified : internal and external . The
development team and the customer ; and the boundaries and relationship between the
developmental team and any subcontractors . former are used by the project team to judge progress and are usually based on events
of medium importance, such as the completion of coding of a subsystem ; the latter are
The third section of the project plan should include details of how the project is to
be managed . It should contain the results of the risk analysis, and how likely risks are major events, such as the completion of the requirements specification, which are used
by the customer to judge progress .
to be managed . It is important that this part of the plan should record any assumptions
It is worth stating that the final part of the project plan will evolve and become more
made by the developer ; for example, that the customer will return documents sent for
refined as a project progresses . During project planning only an outline requirements
comment within a specified time period .
62 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROJECT PLANNING 63

analysis and design will have been carried out, and hence tasks will, perhaps, only be 3 .3 You have received a memo from your boss which expresses some degree of cynicism about the need
specified at the subsystem level . However, as a project proceeds and the nature of the for a configuration management system on the project for which you are responsible for quality . Write
system to be developed unwinds, more and more detail should be added to this part a memo explaining why the project needs a good configuration management system .
3 .4 If you adopt a modern costing technique such as COCOMO what are the implications for your
of the project plan . Before leaving this chapter it is worth while stressing an important
quality system?
point : that the specification of the detailed organization of a project plan presented in
3 .5 Why does each project require a quality plan? Surely you could use a single quality plan for each
this section is not prescriptive ; there is considerable leeway in the way such a plan is project?
organized . However, whatever form of project plan organization you adopt it should, 3 .6 The statement : 'The system should keep an up-to-date set of data which contains details of staff
according to Rook (1990), be judged by the main requirements for such a plan : sales' is not a verification requirement . Explain why .

• that it provides the project manager with a basis for allocating resources during a
project and for predicting, monitoring and reporting on progress and cost ;
• that it provides high-level management with a summary of a project and a means by
which they can monitor its progress ;
• that it provides the customer with the same insight and progress monitoring capability
as the developer .

If a project plan does not satisfy these three demands, then it has failed .

3 .12 SUMMARY

Project planning is one of the most important tasks carried out on a software project .
Project planning tasks which fail usually give rise to very serious problems on a software
project . Because of this, the software developer should have standards, procedures and
guidelines which address activities such as risk analysis, the development of the quality
plan, the development of the project plan, cost estimating, and project organization .

3 .13 FURTHER READING

Rook (1990) is an excellent short tutorial on project management which could be read
in two nights. Brookes (1975) is the classic book on software project management . Tech-
nically, it is a little out-of-date, but it still contains some of the wisest and most relevant
advice on how to manage large software projects . Charette (1989) is the best treatment
of risk within a software project environment that I have read . The best treatment of
modern process-oriented project management that I know of is Humphrey (1989) . If you
are interested in costing, the standard work is Boehm (1981) ; a good review can also
be found in Monanty (1981) . A description of Funtion Point Analysis can be found in
Symons (1991) .

PROBLEMS

3 .1 Write down the type of questions that you would include in a risk questionnaire which deals with
the risks inherent in the application area .
3 .2 You have discovered that the customer for a software system is hesitant, does not know the applica-
tion area and is prone to changing his mind . How would you organize a project to take account of these
factors?
 REQUIREMENTS ANALYSIS 65

• A poor expression of requirements, due to factors such as ambiguity or just plain poor

4
writing style .
• An inadequate specification of the data that is to be manipulated by a system . Increas-
ingly, companies seem to be getting better at specifying the functions of a system-
what it does-but still fall down on the process of data specification .
REQUIREMENTS ANALYSIS • Poor organization of the requirements specification . This is often manifested by func-
tions being randomly expressed, with no structure which brings together associated
functions .
• Changes in requirements being generated by the customer . A software system is a re-
flection of the real world. Unfortunately, the real world changes from day to day and
this gives rise to changes in requirements, not only during the process of requirements
analysis and requirements specification, but at later stages of the project . For exam-
ple, a developer producing a financial package will often need to carry out extensive
changes if a government announces new financial laws .

4 .2 THE REQUIREMENTS SPECIFICATION

The key document on any software project is the requirements specification . It is this
document which forms the basis for much of the development that follows the processes
AIMS of requirements analysis and requirements specification : detailed costing; the develop-
ment of the system design ; the construction of the system and acceptance tests ; and the
• To describe the main features of the requirements specification . production of documentation such as user manuals . The first document that the software
• To show what the features of a good requirements specification are and describe the developer receives from a customer is a statement of requirements. Usually this document
standards and procedures required to encourage the development of these documents . will be at a very high level of abstraction and contain many problems, some of which are
• To outline some of the techniques for validating requirements specifications that described below .
should be part of a quality system . Ambiguities . These are statements which are capable of being interpreted in a number
• To describe how verification requirements, used for the derivation of system and of ways .
acceptance tests, are generated at an early stage of the software project . Design directives . These are statements which instruct the developer to carry out
• To list the standards and procedures which are normally associated with the require- design in a particular way, for example by splitting a system into a number of subsystems .
ments specification process . Normally, design directives should be discouraged, and the software developer should urge
the customer to remove them from the system requirements . Sometimes, however, there
are good reasons for a design directive . For example, a customer may request an addition
4 .1 INTRODUCTION to an existing software system which already has a well established process and data
architecture and requires the developer to structure the addition in a way that fits in
One of the sobering facts about the processes of requirements analysis and requirements with this architecture .
specification is that not only are they error-prone, but that an error committed during Contradictions. These are statements which are in direct variance with each other .
these parts of the software project, if not detected until a later stage, can have a serious Statements of requirements usually contain many contradictions . If these were close to
effect on the success of the project, both in terms of budget and time overruns . Some of each other, then their detection would be easy ; however, they are often separated by
the main problems that are encountered are : many pages of text.
Platitudes . A platitude is a section of text which everybody agrees on, but which has
• The requirements specification is expressed at too high a level of abstraction . There no exact meaning . An example of a platitude is the sentence which, time and time again,
is not enough detail in it for the system designer to produce an adequate design . occurs in statements of requirements : `The system must be user-friendly' . Platitudes
• Poor communication channels exist between the developer and the customer . This are usually easy to detect ; however, what is difficult is deciding on the hidden meaning
might occur for a number of reasons ; for example, ignorance of information technology beneath them.
concepts by the customer, multiple substitution of customer representatives during Mixed levels of abstraction . A statement of requirements will contain instructions
the analysis process, and the novelty of the application . which are at different levels of detail . For example, it might contain the sentence :
64
66 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION REQUIREMENTS ANALYSIS 67

The system should provide reports on the functioning of the chemical reactors currently operating . 4.3 SPECIFYING FUNCTIONALITY

together with the sentences : The most important point about the specification of systems functions is that it should
be hierarchic : functions should be split into subfunctions, these should then be further
When the valve report command is initiated a list of valves and their malfunctions will be displayed decomposed into sub-sub functions, and so on . This should be reflected in the standards
on the originating VDU. Only the last ten malfunctions will be displayed . used for requirements specification .
There are a number of ways of documenting this hierarchy . One way is via numbered
Now, both these statements are important : the first represents a high-level view of a paragraphs . An example of this is shown below for part of a system for monitoring the
system which might be read by the customer's senior management, while the second working of a chemical plant which consists of a number of reactors :
represents a more detailed view which might be read by engineers concerned with con-
trolling the chemical plant . The major problem with these statements is that they are 3 The system should produce data on the functioning of valves .

not organized and structured, but spread almost randomly throughout the statement of
3 .1 The system should produce data on the functioning of outlet valves .
requirements .
The two most important components of a statement of requirements are functions 3 .1 .1 The malfunction command will produce data on the degree of malfunctioning associated
and constraints . A function is a statement which describes what a system is to do . For with a specific outlet valve on a particular reactor . Staff who use this command should provide a
example, the statement : valid valve name together with a valid reactor name . The system will respond with the name of the
valve, the name of the reactor and a list of malfunctions over the current operating period . Each
malfunction should be detailed according to its type, when it occurred and the date on which the
The system should produce data on the pressure experienced by a model plane in a wind tunnel .
malfunction was cleared .

is a function, albeit at a high level of abstraction . 3 .1 .2 The current state command will produce data on the current state of a particular outlet
valve. The staff who use this command should provide a valid valve name together with a valid
A constraint is something which constrains the developer in some way . It can be a
reactor name . The system will respond with the current state : this will either be open or closed .
product constraint or a process constraint . A product constraint is a constraint on some
characteristic of the software system. The most frequently occurring constraints are those 3 .2 The system should produce data on the functioning of inlet valves .
which specify a performance threshold or maximum memory occupation .
A process constraint is a statement which constrains the way in which a particular 3 .2 .1 The malfunction command will produce data on the degree of malfunctioning associated
with a specific inlet valve on a particular reactor . Staff who use this command should provide a
software process or collection of processes are carried out ; for example, a customer may
valid valve name together with a valid reactor name . The system will respond with the name of the
ask that all calls to a computer's file subsystem be routed via the operating system, or valve, the name of the reactor and a list of malfunctions over the current operating period . Each
specify that a certain standard be used for a technical activity such as system design . malfunction should be detailed according to its type, when it occurred and the date on which the
malfunction was cleared .

Self Assessment Question 4 .1 Is the statement : `The developer should 3 .2 .2 The current state command will produce data on the current state of a particular inlet
valve . The staff who use this command should provide a valid valve name together with a valid
use the programming language C' a function, a constraint or a process con-
reactor name. The system will respond with the current state: this will either be open or closed .
straint?

Solution It is a process constraint : it constrains the way a project is car- There are a number of reasons why the functional part of the requirements specification
ried out . should be structured in such a way and a standard encourage this . The first reason is
that it provides a description which is neatly partitioned into levels of detail which enable
different staff, in both the customer's and the developer's organization, to read as much
The main aim of a requirements specification is to identify the various categories of infor- as is necessary for them to do . For example, with the chemical plant specification shown
mation in a statement of requirements and then structure the requirements specification above, the manager of the plant will want to read only the top level of the document,
in such a way that it can be easily accessed by staff carrying out developmental activities . while his or her engineers will want to see a much more detailed description of what
Any requirements specification which does not address this is seriously lacking . functional facilities will be provided .
Another reason for having this form of hierarchy is that different staff in both the
development team and the customer's organization may want to access a number of levels,
but may only want to look at part of the document . For example, with the chemical plant
requirements specification, engineers who are only concerned with plant optimization
may wish to read only that part of the specification which deals with the management
68 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION REQUIREMENTS ANALYSIS 69

information necessary for them to perform their tasks ; while engineers concerned with
the day-to-day functioning of a plant will be interested in that part of the requirements Self Assessment Question 4 .2 Which of the sixteen categories above
specification which deals with monitoring and control . The same argument holds for does the sentence `When a catastrophic error is detected the system should
the developer-especially when the software project is a large one . Normally in such shut down and start up in monitoring mode' .
projects an analyst, for example, is only allocated a part of the system to be specified-
usually a functionally related part-and will want to concentrate only on that part of Solution It can be regarded as being in the start-up, system shut down,

the requirements specification concerned with the part of the system with which he or error monitoring and critical condition categories .
she is dealing .
In summary, a hierarchical organization of the functional section of the requirements
specification enables both customer and developer staff to concentrate on those parts of Thus, the reader of the requirements specification who wishes to find out what functions

a system which are of concern to them, without having to read descriptions of parts of are concerned with security can examine this part of the requirements specification, look
the specification which do not concern them . The numbered indented form of functional in the subsection marked security, and extract the unique identifiers of the functions

description described above is a very popular form of organization . However, an increas- which are concerned with this functional area .
ing number of developers are adopting graphical notations . The most popular graphical
notation is the data flow diagram . For a good introduction to this form of specification
see Page-Jones (1988) . An important point which is worth reiterating here is that the 4 .4 DATA SPECIFICATION

functional requirements part of the requirements specification should not contain any
design or implementation details . For example, in the fragment of requirements specifi- Another important component of any requirements specification is a section which de-

cation for the chemical plant system there is no indication whether commands are to be scribes the data thatt forms part of the application . A good standard for requirements
specification should devote a considerable amount of space to this feature .
implemented via a menu, a windows interface, or whether line-based commands are to be
used . This is something which is normally decided during design . Nevertheless, you will This specification should, in common with the functional specification, not contain

sometimes be faced with a customer who will insist on a particular implementation . The any implementation details ; for example, terms such as indexed sequential file, direct file,
and 6 bit field should not be included .
ideal solution is to persuade the customer not to insist on an implementation ; however, if
A good standard to adopt is to insist that the developer specify data in the form
he or she persists then a separate section of the requirements specification should contain
these details . of entities, attributes and relations . This is becoming standard practice in information
system development where a number of development methods insist that such data is
The requirements specification standard should insist that in a hierarchical form
specified graphically. However, other application areas, such as military command and
of specification a unique number is assigned to each function . These functions should
control, are becoming so data rich that I suspect any project which does not specify data
be referenced in another part of the system specification concerned with grouping the
functions into various categories . A list of some of these categories is shown below for in such a way will become the exception during the coming decade .
An entity is an object which is of major interest to the customer . In an air-traffic
both information systems and real-time systems (Redmill, 1988) :
control application typical entities might be a plane, a flight plan, and a radar ; in a
stock-control application typical entities are a warehouse, a product, a product order,
Monitoring Control
Query processing and a queue of product orders which, as yet, cannot be satisfied because the product is
Report generation
System start-up System shutdown out of stock . Eventually entities become implemented, usually in some form of collection
Exception handling of records : in a file, an array, or some form of dynamic data structure such as a linked
Critical condition
Security list .
Database integrity checking
Data validation An attribute is a property of an entity. Typical attributes for a plane entity in an air-
Error monitoring
Recovery after abnormal termination Database update traffic control system are : the plane identifier, its current position, its estimated landing
Optimization time, the type of plane, its destination, and the location from which it started its journey .
Adjustment of system
Again, there is no need to specify any physical details . An important attribute or set of
parameters
attributes is known as a key . This is a collection of attributes which uniquely identify a
particular occurrence of an entity . For example, in the air-traffic control system the key
would be the plane identifier .
The final component of the data specification should be a description of the rela-
tionships between the entities . These relationships will be either one-to-one or one-to- n.
With the former, a single occurrence of an entity is associated with a single occurrence of
another entity. For example, in a staff management system the fact that an employee is
70 REQUIREMENTS ANALYSIS 71
SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

allowed to drive only one company car would be modelled using a one-to-one relation . A which represents the quantity of product delivered . Each order will result in the increase in the
one-to-n relation models the association between one entity occurrence and one or more NurnberStocked of each product .
other entities . For example, in a chemical plant application the fact that a computer
monitors the state of a number of reactors would be modelled by means of a one-to-n
relation. Self Assessment Question 4 .3 Does the fragment above involve the
traceability concept outlined in Chapter 1?
Relations are vital for a designer who might, for example, be designing a part of a
system which produces reports and wishes to know how many subsidiary items to access
Solution Yes, it represents traceability between the functional parts of
after an occurrence of an entity has been processed . Entities, attributes and relations
the requirements specification and the data specification .
represent a logical view of a system which is uncluttered with design and implementation
details and, hence, represents a view which can be easily read, by both the developer and
the customer . Every quality system should have a standard which describes how such
It is important to point out that although this is a useful way of connecting together
information is to be documented . An extract from such a standard is shown below :
system functions and system data it does not absolve the analyst from specifying data
Normally on our projects the first process that is carried out is to determine the entities, attributes
using a notation such as an E-R diagram-this information is still needed . The advantage
and relations inherent in a system . This process is carried out by interrogating the customer, and by of this form of linking description is that it leads to easy validation . For example, when
reading any documentation such as a statement of requirements which is provided by the customer . the system is designed, and the entities and attributes are specified in computer terms,
An entity is an object which will be implemented in data ; an attribute is information which is staff can check that the design specification actually meets the functional part of the
associated with an entity and is normally implemented as a field in a record . requirements specification ; for example, someone checking for the correct implementation
A relation describes how many occurrences of one entity are associated with a single occurrence of the first function shown above can read the design documentation in order to check
of another entity. A one-to-one relation means one occurrence is associated, while a one-to-n relation
means that a number of occurrences are associated . For examples of such relations see section 14 .5 that a data element corresponding to a product identifier is read, a database of product
of the Sample Project Documentation Manual . In documenting relations, entities and attributes details is scanned, and the product names listed .
two forms of layout are specified . It is worth deviating slightly to point out that the example of incorporating data de-
The first is known as an E-R diagram. This is normally used for projects which use the SSADM tails within a functional specification helps support the important feature of traceability
development method . For other projects a strict textual form of layout is used . This layout first lists
each entity followed by a colon, followed by the attributes listed in alphabetical order . Attributes
described in Chapter 1 . By structuring the functional part of the requirements specifica-
which form a key and which uniquely identify an entity occurrence are displayed in bold . The
tion in this way the developer is able to start the process whereby system functions can
entities are listed in alphabetic order . Relations are also listed in alphabetic order. Each relation is be traced to a system design and, eventually, to the program code which implements the
displayed on a line . Its name is displayed first, followed by the first entity together with the second design .
entity and terminated by the relation type : whether it is one-to-one or one-to-n . A sample listing
of this data is shown below .

4 .5 MODES OF OPERATION
ENTITIES
Location : Name, TelNo As well as functional and data components, the requirements specification should also
StaffMember : Worksld, CurrentBonus, Dept, Name, Salary, contain details of the modes of operation which the system can be in . There are a large
number of these modes ; some examples are emergency operation, when a potentially
RELATIONS catastrophic result occurs, or is about to occur ; shutdown operation, when the system is
Attended'IhainingCourse : Employee, Course, 1-to-n to be closed down ; error recovery operation, when the system is trying to return from an
Manages : Manager, Employee, 1-to-n
error back to its normal state ; and self-test, when the system is checking its own operation
for correctness . This section of the requirements specification should detail these modes
of operation and should describe when it enters and leaves these modes . An example of
Some software developers include data information in the functional part of their require- such a description is shown below :
ments specification . An example of this form of combined specification is shown below,
with the data items italicized : Emergency state
The system will enter an emergency state when functions 3 .4, 3 .5, 3 .9, 5 .2, 5 .4, 5 .5 and 5 .6 are
executed . In general this state is entered when a problem is experienced with the inlet and outlet
When the OutOfStock command is initiated the user provides a Productld. The system will then valves which leads to the system's inability to close or open these valves . When in this state functions
scan the system data and will produce a list of ProductName of those products for which there is 13 .2 and 13 .3 are initiated and depending on the results of these functions the system will either
no stock available . When the update command is initiated the system will process a sequence of return to the normal operation state or move to manual state where operator intervention can occur .
ProductOrders . Each ProductOrder will contain a Productld and a DeliveredAmount : an integer
 72 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION REQUIREMENTS ANALYSIS 73

This describes what happens when inlet and outlet valves of a chemical reactor stick . 4 .7 CUSTOMER-ANALYST INTERACTION
When they do, functions 13 .2 and 13 .3, which are concerned with checking the state of
the valves and attempting to open them, are invoked . If these functions are successful the A good quality system should provide standards and procedures which would enable
system moves to its normal operating state ; if not the system enters the manual state the interaction between the customer and the developer to be documented as closely as
where intervention in the operation of the chemical reactors is taken over by process possible . One of the problems that many software developers face during requirements
workers . There are a number of states which systems can find themselves in (Redmill, analysis is that of a statement of requirements which is at far too abstract a level . During
1988) : the early stages of a software project the statement of requirements will become the main
focus of debate . A considerable amount of discussion will ensue at the end of which a
System generation Installation requirements specification will emerge . If the developer does not keep track of additions,
Start-up Self-test deletions and clarifications that a customer has made, then he or she has no defence
Normal operation Emergency from the customer who might claim that, for example, the company has misrepresented
Degraded Automatic a functional set of features in the requirements specification . The production of some
Semi-automatic Manual form of audit trail which points out that the features were confirmed by the customer
On-line Batch would, at worst, lead to some form of moral ascendancy over the customer and at best
Periodic Housekeeping demonstrate to the customer that care is being taken over the project .
Shutdown Error recovery Since the statement of requirements is a poor place to begin such an audit trail,
Error isolation Back-up many companies adopt the strategy that a first draft of the requirements specification is
produced, and that both the developer and the customer regard the draft as a starting
point in the process of developing the final requirements specification, which could then
4 .6 INTERFACES be used by staff such as system designers .
This first draft, which would be expressed using the requirements specification stand-
A section of the requirements specification should describe the interface between the ard, and would hence feature uniquely identified functions, could be used as the final
proposed system and entities which are beyond the scope of the system . This would in- destination of any audit trail . Any negotiations over the requirements specification would
clude external hardware, users, and any external software . The description of the external then refer back to entities such as functions which could be easily identified, rather than
hardware should include information such as : the structure of files which are intended the abstract wish lists that many statements of requirements gem to be . There is no
for processing by other systems ; the rate at which signals from an interface will arrive ; reason why this first draft should not go through a series of cycles ; indeed, a serious
the length of time that data is available at an interface before they disappear ; the trans- software developer will plan for this, and assume that because requirements analysis
mission rate associated with a particular interface ; the nature of errors associated with is such a difficult task, evolution is the only way to deal with the development of the
an interface and the electrical characteristics of an interface . requirements specification.
The description of the human interface should include a description of the staff who A procedure for documenting the interaction between the developer and the customer
interact with the computer : for example, what their technical capabilities are ; whether is shown below:
help facilities are needed ; what form of interface (line, menu or window) is needed ; any
requirements for reaction time ; whether certain functions in the requirements specifi- One of the items that our company takes very seriously is the process of requirements analysis . We
cation need easily recognizable signals such as emergency displays ; and any security have had a number of problems in the past when customers, at a very late stage in the requirements
analysis process, have changed their minds over major requirements and have attempted to convince
requirements .
us that the change of mind was communicated to us at a very early stage of the project . It is hence
The external software interface part of the requirements specification should include important that we keep a diary of interactions between the customer and the analysts working on
details of data formats, file formats, protocols and subroutine calls associated with any the project .
software to be interfaced with the system that is being developed . For example, if a When a change to a draft of the requirements specification is suggested the analyst must enter
spreadsheet is to be used to display results from a file produced by a system, then this the details of the change in the requirements log . Each entry in the log contains a brief description
of the change, how the change came about, the number(s) of the functions that were changed
part of the requirements specification should include the format of the data required by
or, if the change was to a non-functional part of the system, the section and paragraph number
the spreadsheet and also how to invoke the spreadsheet for the particular functions that in the requirements specification . Each entry should also describe how the proposed change was
it is intended to carry out . communicated ; for example, whether it was via telephone or via a written memo . If the change
was in a written form then any identification of the change such as a reference number should be
included. Use the keywords DETAIL, FUNCTIONS AFFECTED, REFERENCE and SEVERITY
to mark the components of an entry .
The final part of the entry should be an indication of its severity ; we have decided on five levels
of severity : from 0 (trivial) to 4 (very serious) . A very serious change is one which would have a
74 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION REQUIREMENTS ANALYSIS 75

substantial effect on the project in resource terms . If an analyst encounters a change at level 3 or It is worth issuing a warning here : that often this will prove a very difficult task-if
4 then either the staff member he reports to, or the project manager, must be informed, as these not an impossible one . Many statements of requirements tend to be too abstract, often
types of changes often lead to budget overruns and late delivery .
resembling a vague wish list . In this case the labour involved in cross-referencing the
It is important that as many of the interactions as possible between the analysts and the
customer should be accompanied by paper documentation . We recognize that there will be rare statement of requirements to the requirements specification would be wasted .
occasions when a telephone call is the medium of communication . If it is, then do attempt to get a However, increasingly, there are customers-usually those with a large amount of
memo from the customer following up the change . If this proves impossible you should at least note experience of dealing with software developers-who produce relatively well structured
the time of the call and the customer representative that you spoke to and write a confirmation statements of requirements, and it is these customers who most welcome cross-referencing
letter.
All changes which lead from one version of the requirements specification to the next should
in order to help them check that a requirements specification correlates with their needs .
be documented and the change sent to the customer for approval and, after approval, to the project For this reason, a quality system should at least offer the project manager the option of
library for archiving. In our project planning standard PP 1/5 Section 3 .4 .2 you will see that all our producing this section of the requirements specification and standards and procedures
projects should write in an assumption that certain documents such as notification of requirements for the process of cross-referencing .
changes should be read by the customer and turned around within a specific time . Analysts should
look at the project plan for their project, see what the time limit is for requirements changes and
add a sentence to the effect that the developer is expecting comments back in n weeks and the
non-receipt of comments will be taken as agreement . 4 .9 VALIDATING THE REQUIREMENTS SPECIFICATION
An example of an entry for a change is shown below :
At the end of the requirements analysis phase both the developer and the customer
Change 17 will need a high degree of confidence that the requirements specification is a correct
DETAIL This change was initiated by the customer who asked for all malfunctions of the system reflection of the needs of the customer . The role of validation is to check that those needs
to be written to a log file .
have been catered for, and that the requirements specification is an adequate basis for
FUNCTIONS AFFECTED 12 .1-12 .22,13 .4-13 .18,14 .1, 14 .3 . further activities such as system design ; procedures should be provided by the quality
system for validation . Currently there are a number of techniques for validation of a
REFERENCE . Confirmed in letter to D Aspinall dated 12/3/90, reference number bpt 56/Awwl2 . requirements specification : prototyping, simulation, and technical reviews are the three
main ones . Simulation is the process of modelling a system in order to determine whether
SEVERITY 2 .
non-functional requirements-usually response times-can be implemented . Prototyping
is the process of producing an early version of a system in order to check with the
customer that the developer has a correct view of what is needed from a system . Technical
reviews are meetings of staff who examine a project document, such as a requirements
4 .8 THE REQUIREMENTS SPECIFICATION AND THE specification, system design or project plan, for correctness .
STATEMENT OF REQUIREMENTS Reviews have been found exceptionally useful for two reasons : first, they constitute
a collective process whereby a number of staff take a dispassionate look at a software
One major problem that . the customer has when reading and checking a requirements document ; and second, they can be held early on in the software project, in order to
specification is that of relating the contents of that document to the statement of re- detect errors that, if undiscovered, are capable of causing a catastrophe . Pressman (1994)
quirements that he or she has produced . It is a good idea to reproduce the statement of has described a number of principles and guidelines for the organization and conduct of
requirements in the requirements specification-usually as an appendix-with the items technical reviews .
in the statement of requirements cross-referenced to items in the requirements speci- Ideally, between three and five staff should be involved in a technical review . In
fication . For example, functional requirements in the statement of requirements would the review of the requirements specification, this would typically be a chairperson, usu-
be annotated with the function numbers from the requirements specification and non- ally a senior member of staff from the project ; the analyst who prepared the part of
functional requirements would be annotated with section and paragraph numbers . Any the requirements specification being reviewed ; an analyst from another project ; a cus-
new requirements which have been discovered and which are not contained in the state- tomer representative ; and whoever is to produce the system and acceptance tests . The
ment of requirements should be referenced in this section . last member is an important participant : he or she is able to probe any weaknesses in
the system design by asking searching questions about how particular combinations of
operations would be tested .
Self Assessment Question 4 .4 What is the cross-referencing described A point which should be made about the chairperson-or, as he or she is sometimes
in the previous paragraph an example of? called, the moderator-is that the project manager should not chair such reviews, or
even attend them . There are two reasons for this . First, project managers have other
Solution It is an example of traceability. pressures on them which they often see as being much more important than software
quality. These will concern issues such as the number of days to the date of release of the
 76 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION REQUIREMENTS ANALYSIS 77

software and the amount of budget remaining for their projects . Consequently, there will should work through a checklist of overall points about the product . This should be
be a subconscious tendency for managers to skimp on the task of chairing the reviews followed by a detailed reading of the item . The amount of time that is devoted to each
in order to sweep problems under the carpet . Second, a review is a bruising experience of these activities will depend on the item that is being reviewed . However, the final
for the member of staff whose work is being reviewed . Often such work is torn to pieces . activity-i.e . reading the item line-by-line-will take up most of the time .
If the manager is in attendance there will be a tendency for staff whose work is being It is important that the moderator or a secretary takes written notes during a review
examined to regard the presence as some sort of personnel evaluation exercise, and to and that a standard should specify the structure of these notes . In particular, a list of
become exceptionally defensive, rather than have the very open attitude to errors that errors-or rather problems-should be produced, for two reasons : first, attendees will
is a prerequisite for the successful review . forget what went on in a review . It may be days before the member of staff who pro-
Standards for review meetings should insist that they last no more than two hours . duced the item reviewed gets around to changing it . Second, quality assurance staff will
When properly run, reviews are intensive and exhausting and most people's span of be required to ensure that reviews are being carried out correctly, and any actions out-
attention lasts for no longer than two hours . After this time they tend to lose interest in standing have been dealt with . They usually do this via formal audits and spot-checks .
the topic of a review very quickly . A set of minutes, together with a list of outstanding problems, is essential for them to
It is also important to point out that guidelines for reviews should specify the amount do their jobs efficiently .
of time that should be devoted to preparing for a review . What the participants who A final point to be made about reviews is that each type of product being reviewed
attend a review are doing is debugging ; this is not something that can be done in a should have a checklist associated with it-provided by the quality system . For example,
few seconds, but requires time for reflective thought . Pressman advises that advance programs or subroutines would have a checklist that contains instructions about detect-
preparation should take no more than two hours per person . ing unsafe programming constructs, non-initialized variables, non-conformance to coding
It should be remembered that it is the document which is the subject of a review, not standards, over-complex interfaces, and convoluted program code .
the person who developed the document . This is an important point, because a review The review process is shown in Fig . 4 .1 which is an extract from an existing proce-
can be quite a harrowing experience for the staff whose document or program code is dure . First, the item to be reviewed is produced and the member of staff who produced
being reviewed . Nobody likes criticism, and at a review the staff involved have to take it considers that it is near to being correct . This is often an iterative process, whereby
two hours of implied criticism . The role of the moderator is key in a review : he or she whoever produced the item checks specifics with colleagues . The staff member then in-
has to organize and run the meeting in such a way that the product is decoupled from forms whoever is to chair the review that the item being reviewed is ready . It is sent to
the person that produced it . the moderator, who then evaluates whether it is in fact ready . If it is, the review is organ-
There are a number of strategies aimed at depersonalizing a review . A simple one ized : the room booked, participants informed, and copies of the review item, together
is never to refer to errors as errors, but as problems . One particularly good moderator with current guidelines for the item type, are circulated .
that I know of announces, at the beginning of the review, that the document or program Finally, shortly before the day of the meeting the agenda is sent to all participants .
code being reviewed is the collective responsibility of the attendees in the review . The The review is then held, and a list of problems produced, together with the minutes of
blame for any errors that remain after the review has been completed is divided equally the review . Copies of these are sent to each of the participants and at least one copy is
among the participants . Another moderator of my acquaintance always tries to encourage filed in the project library.
positive remarks about a program, e .g . that it is well structured or easy to read, as well The procedure for holding a review should specify a number of outcomes . First, the
as encouraging the error-detection process . review may have been totally successful . In this case the item is passed and stored in
Another guideline is to set an agenda and stick to it . Reviews can meander quite a the project library . Eventually, it will be frozen, along with all the other similar items
bit, with a solution to a problem being aired, followed by a discussion of similar problems that have been reviewed . For example, the requirements specification will be reviewed a
encountered on other projects, followed by a discussion of an old project that some of chunk at a time, and when the whole of the requirements specification is deemed to have
the participants worked on, and eventually ending up with a general free-for-all about been satisfactorily reviewed it is frozen, and any further changes are rigorously monitored
office politics, sexual politics, and the quality of the eating houses in the area . and controlled . Second, there may be some small errors associated with the item that
An important point related to keeping to an agenda is that the moderator should not is being reviewed . In this case the item is regarded as having passed the review . The
allow the review to be taken up with discussions about how an error should be rectified . member of staff who produced the item then makes any amendments, and shows them
A review is an extremely useful validation and verification technique for detecting errors ; to the moderator of the review who signs the item off as fully passing the review . Third,
it is less good at remedying them . The only person who is really capable of removing an there may be major problems with the item . In this case it is then modified and has to
error is the person who produced the document under review . It is, of course, impractical be re-reviewed, preferably with the same members of the original review being present .
to ban all discussion about how an error can be eradicated . However, this should be brief A number of review formats have been devised . The most popular are Fagan Inspec-
and along the lines of : `We had that problem with one of our programs : ring Geoff up tions developed by Michael Fagan (1986), which has an emphasis on not only detecting
and he'll tell you how he coped with it .' errors, but also using error data to improve software processes such as requirements
The agenda for a review should be quite simple . Normally, the first item that should analysis . Part of a procedure for running a review is shown below :
be dealt with is any general points about the item for review . Next, the participants
 78 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION REQUIREMENTS ANALYSIS 79

passed subject to minor revision, or has not passed . In the first case the chair signs off the object
on the review report form . In the second case the originator of the item rectifies the minor problem
and the chair of the review checks the changes and signs off the review report form . In the third
case a re-review is needed .
Producer informs moderator that Once an item has been reviewed it should be consigned to the project library, accompanied by
item is ready for review the review report form . The remainder of this section describes information specific to each review
type that our quality manual supports .
1
Moderator checks item • Further work 7.1 The requirements specification review
for readiness on item carried
out There should normally be either four or five participants at this review, one of whom should not
be the project manager or the manager to which the producer of the item to be reviewed reports .
Review organized : room booked . agenda For this review a selection of the following staff should be invited : the producer of the section of
prepared, participants selected . etc .
requirements specification that is to be reviewed ; an analyst from another project ; one of the system

1
Review occurs MAJOR PROBLEMS
design team; a customer representative ; and any staff responsible for the derivation of the system
tests .
The documentation items that are provided with this review are : the section of requirements
OK l specification that is to be reviewed, any verification requirements and the customer statement of
requirements . These items should be circulated to the participants . A document that should also be
Item stored
made available for reference during the review is the journal which records the interaction with the
Producer of item rectifies customer ; the bulk of this document precludes it being copied and circulated to the participants of
minor problems the review . If a scenario analysis has been carried out then any scenarios should be made available
to the participants .

1
Moderator checks that
The checklist which is relevant to this review is checklist RR 1 .2 and all participants should
be provided with it .
rectification is carried
out satisfactorily
The description of the requirements specification review includes a reference to scenario
Figure 4 .1 The review process . analysis . This is a process whereby the customer and the analyst describe a series of
interlinked actions which should be dealt with lay the system which is being specified . For
7 Technical reviews example, in a stock control system a series of interlinked actions would be : a customer
One of the main techniques that we use to implement quality control is the technical review . This ringing in to purchase a product, the salesperson discovering that the item is out of
is just a meeting of staff who examine a critical project document for correctness and who issue a stock, the salesperson placing an order on a back-order queue, and a sales confirmation
report on the problems that have been encountered during the review . The process of organizing a document being issued to the customer . These scenarios are useful for checking that a
review is shown below .
requirements specification is complete and the interrelation between functions has been
The member of staff responsible for an item that is to be reviewed informs the chair of the
review that the item is ready . The chair checks that the item is in a fit state for review . If it is properly documented .
not then it is returned to the originator for reworking . If the item is ready for review the chair
books a room, selects the participants for the review (usually between three and five members of
staff), arranges for the review material and any supporting material to be duplicated and sent to 4 .10 DEVELOPING THE VERIFICATION REQUIREMENTS
the participants, and informs each participant of the date and place of the review . Who the chair
invites to the review depends on the material to be reviewed, as does the nature of the supporting
material . These details can be found in sections 7 .1 to 7 .8 . Another area of activity occurring during requirements analysis which a quality manual
For many of our reviews there will be checklists which describe the particular problems that should address via standards and procedures is the generation of the verification require-
review participants should be looking for . Copies of the checklists should also be sent to each ments . Chapter 4 has already described the nature of verification requirements : that they
participant . The review is held . One of the participants is nominated as a recorder . The chair will are requirements which are of direct interest to the customer, rather than those which
normally ask any of the participants whether there are any general problems with the item that
is being reviewed . After this the document is processed serially, item by item are concerned with the internal functioning of the system .
; for example, if it is
a requirements specification this will be done on a paragraph-by-paragraph basis, if it is program
At the end of the requirements analysis task staff responsible for testing will read
code then it will be done on a module-by-module basis . Each time a problem is discovered it is the requirements specification and derive the verification requirements . It is important
noted down on the review report form . This form contains sections which describe the nature of the that this process is begun at an early stage in the project, for the following reasons :
problem, where it occurs, the staff member responsible, and the severity of the error . For further
details of this form see section 8 .1 .
• Staff who are to carry out the testing of a system may have a requirement for special-
At the end of the review the chair will decide whether the item has passed the review, has
purpose testing tools or hardware . It is important that these requirements are known
80 REQUIREMENTS ANALYSIS 81
SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

as early in a project as possible, so that if there is a need for the tools to be produced
by a subcontractor, or by another part of the developer's organization, then as long a
lead-time as possible is available : subcontracted software is a major problem in many
projects, and a project manager often likes to see as much slack in the subcontracting
process as it is possible to achieve .
• The project manager will need to know, as early as possible, the extent of the system Generation of
verification
testing that needs to be carried out in order to resource this part of the project . requirements
Requirements analysis and
• A good tester who is attempting to extract verification requirements from a require- system specification
ments specification will ask searching questions about any ambiguities and contradic-
tions that occur . This is a major aid in the validation of the requirements specification . Refinement of
verification
System design requirements .
The last of these three points is the most important : a good tester who continually design of
asks the question : `But how do I test this?' soon gets to the heart of some very serious system and
acceptance
problems with a requirements specification . As an example, assume that the analysts in Programming tests and
a software project have let through the sentence : `The system should be user-friendly', production of
test procedures
which occurred in the statement of requirements and the requirements specification . A
good tester would look at the sentence and ask some demanding questions about how
System testing
to specify the tests which check this . What would normally happen would be that the Application of
analysts would be asked to clarify the sentence . They may come back with a number of test procedures
interpretations :
Acceptance testing
• That the phrase means that the customer wants a help facility which provides on-line
guidance that staff using the system can refer to during operation . Now the analyst
and the tester may disagree that this interpretation actually leads to a user-friendly Figure 4 .2 Verification requirements and development .
system . Nevertheless, it provides the tester and the analyst with a much clearer view
of what is required by the user, and enables the tester to devise some tests : a test to per hour that they made, and checked that a numerical threshold specified by the
see whether the help facility can be properly invoked, and a series of tests to check customer was not exceeded .
whether the text associated with a particular help message is in fact a help facility
rather than a hindrance facility. At this point it is worth describing the relationship between the verification requirements
• That, the phrase means that the customer wants a command-based system with long and the other development activities which occur on a project . Figure 4 .2 describes this
and short commands . Again, the analyst and tester may be sceptical about whether relationship . The verification requirements are produced during the process of require-
this is, in fact, a manifestation of user-friendliness . However, it has clarified the re- ments specification . They are then expanded during system design so that it becomes
quirements specification to the point where tests can be derived to check that both more obvious how they are going to be tested . For example, the verification requirement :
long and short commands are recognized by the system .
• That the customer, when he or she uses the phrase `user-friendly', wants a windows
V3 .4 When the back-orders command is initiated with the important order parameter, the system
interface . Now there isn't really a test for having a windows interface : you've either will display the back-order queue in decreasing order of customer importance,
got it or you haven't . Nevertheless, by asking the question : `How do I test this?', the
tester has uncovered a major problem with the requirements specification .
• That the customer, instead of having any facilities in mind, is more concerned with will give rise to the refinement :
the behaviour of the system in that he or she does not want a high error rate when
staff interact with a system . This is mainly a preoccupation of customers who have V3 .4-D The back-orders command will be checked by initiating the command with at least three
test databases consisting of varying lengths of queue . The tester will check that the command has
contracted for safety-critical systems, but also concerns those who have systems which been implemented correctly by running a utility which prints the queue entries on a printer, and
require large amounts of data entry and who may have to spend large amounts of then checking that the screen displays tally with the printed output .
resources in ensuring the accuracy of the data that is keyed in . In this case the
developer would be allowed to devise an adequate interface, and the tester would then The next stage for the testers is to expand this description into a test procedure. This is a
develop some tests to check that a particular error rate was sustainable ; for example,
detailed description of how the test is to be carried out : it will include details on how the
tests could be run with real users of the system, which monitored the number of errors
 82 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION REQUIREMENTS ANALYSIS 83

Table 4.1 Tabular representation of verification requirements There are a number of places where verification requirements can be documented .
Command typed in ok I 1 1 0 Many companies place them in their quality plan . Some add them as an appendix to the
Date ok 1 0 system specification . One or two companies that I have worked for have even put them
Shop ok 1 0 as an appendix to their project plan .

Results displayed 4 .11 THE STRUCTURE OF THE REQUIREMENTS

Date error 1
Shop location error 1
SPECIFICATION
Command error 1
This section details all the components which form part of the standard for a requirements
171 172 173 174
specification . The structure of the requirements specification outlined below is partially
based on the one presented in Redmill (1988) . For a much more detailed description of
the contents of a requirements specification the reader is referred to this work :
test is to be started, terminated, aborted, the location of test databases, how to operate
the utility mentioned in the test design above, and what to look for on the screens that 1. Title
will be generated . 2. Contents
This is just an introduction to the process of testing ; Chapter 7 contains far more 3. Introduction
detail on the topic . The important point worth making here is that a quality manual 4. About this document
should offer specific guidance on how to generate the verification requirements, how 5. Glossary
to document them, and how to interact with the analysts on a project if problems in 6. System requirements
interpretation occur during this process . 7. Target system environment
This chapter has included an extract from a quality manual describing what a veri- 8. Customer-imposed constraints .
fication requirement is . Normally, such requirements are documented using natural lan-
guage . However, for many systems, especially commercial data processing systems, where An important point that should be made before outlining the contents of each section
many of the functions are transaction-based, a tabular form is a useful way of displaying is that staff carrying out activities such as system testing or system design will be con-
collections of verification requirements which correspond to a particular feature . tinually referring to this document, and a numbering system should be adopted which
An example of this tabular display is shown in Table 4 .1 . It describes the verification uniquely identifies every requirement which is to be validated .
requirements for a command which displays the number of customers processed on a The introduction should be brief (no more than a single page), it should describe the
particular day by a particular shop in a large retail chain . The user of this command system and outline any unusual features both in the development of the system and also
specifies the command name, the name of a shop, and a date ; if these have been correctly in the system itself . The section headed About this document should describe any naming
communicated to the system, then data is displayed . However, if there has been an error conventions ; the scope of the document ; the main topics in the document ; the intended
in the date or the name of the shop an error message is displayed . A 1 entry stands audience ; any legal and contractual issues associated with the document ; any relevant
for a condition being true, while a 0 entry stands for falsity . The first column shows standards, both local and external ; how the requirements specification is to be maintained
what happens if the command has been invoked correctly : the command name is correct, and issued ; and, finally, the distribution list for the requirements specification .
the date is of the correct format and the shop name is correct . A correct initiation of The glossary should contain a list of terms and their definitions . It is well worth
the command leads to correct results being displayed . The second column stands for dividing this part of the requirements specification into two sections : terms specific to
what happens when the command has been invoked, but an incorrect shop name has the application and technical terms specific to hardware and software details .
been typed in . Notice that the entry for Date OK has no 1 or 0 entry. This means The major part of the requirements specification will be the system requirements .
that it is irrelevant as to whether the user has typed in a correct or incorrect date . The These are normally divided into a number of sections . One section will detail the func-
numbers on the bottom row represent the numbers of the functions associated with the tional requirements ; this should be structured hierarchically . Another will describe non-
verification requirements . These numbers are the unique identification of each function functional requirements, e .g . those concerned with performance, availability, adaptability,
in the requirements specification . reliability, memory occupancy, file size, and so on . A subsection should be devoted to
This form of documentation is extremely useful . It enables staff responsible for check- each category of non-functional requirement .
ing the verification requirements to check for completeness and consistency . Companies The seventh section-target system environment-should describe the environment
that I know of who have employed this form of documentation always tell me that it into which the system is to be placed . This section should not only contain a description
enables them to have a high degree of confidence that their requirements specification is of the hardware on which the system is to be mounted, but also any other hardware
correct . It also has the advantage that it is automatable via a spreadsheet . interfaces such as communication lines . It should also describe the interface to other
84 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION REQUIREMENTS ANALYSIS 85

software systems and the human interface : the capabilities of the staff who are to use the 4 .14 FURTHER READING
system and what they will use the system for .
The final section is a list of any constraints insisted on by the customer . This section There are few good works that have been published on requirements analysis and speci-
should be divided into two parts : design, validation and implementation constraints, fication . However, Stokes (1990) is an excellent tutorial introduction to these topics,
which constrain the developer in the way that the system is developed and checked ; and Boehm (1984) is an invaluable introduction to the techniques used for validating require-
project constraints, which constrain the way that the project is managed . ments, and Davis-(1990) is a good book-length introduction to requirements analysis .
Easteal and Davies (1989) is a good, short introduction to both design and requirements
analysis .
Self Assessment Question 4 .5 Where in the requirements section should
the fact that a project involves a trans-national collaboration be specified?

Solution It should be in the introduction if this is outside the norm for

the company.

4 .12 QUALITY MANUAL REQUIREMENTS

The following documents or sections should form part of the quality manual for a software
development organization :

1 . Functional requirements specification standard

2 . Non-functional requirements specification standard
3 . Data requirements specification standard
4 . System environment specification standard
5 . Customer-developer interaction standard
6 . Customer-developer interaction procedure
7 . Statement of requirements traceability standard
8 . Requirements specification checklist
9 . Prototyping procedure
10 . Prototyping standard
11 . Requirements review procedure
12 . Requirements review standard
13 . Requirements review checklist
14 . Verification requirements standard
15 . Verification requirements procedure .

Normally items 1 to 4 are included in an overall requirements specification standard and

procedure, which often also includes items 5 to 8 . Items 9 and 10 are often collected
together as one document, as are items 11, 12 and 13, and items 14 and 15 .

4 .13 SUMMARY

Next to the project plan the requirements specification is the most important document
generated during a software project . Because it is used by a large number of staff, ranging
from the project manager to technical writers, it is vitally important that a company
has good standards, procedures and guidelines which describe the process of developing
a requirements specification, its validation and its structure .
 SYSTEM DESIGN 87

sequential files or direct files . For fourth-generation languages the data will be specified
in terms of tables .

5 The main problem that developers face during the design process is checking that the
system design which has been produced actually meets the requirements detailed in the
requirements specification . There are two aspects to this difficulty. The first is technical :
SYSTEM DESIGN for some requirements such as response time there is little theory around which enables
the developer to feel confident that a requirement has been satisfied . The second aspect,
which is less of a problem, is that in projects with poor quality assurance there is either
very little or very poor documentation, which hinders the developer in checking that,
for example, functional requirements are being satisfied . In the previous chapter great
stress was placed on the fact that the requirements specification should detail as clearly
as possible all the user requirements for a system . The aim of this chapter is to show
how, in a number of ways, validation can be made an easier process . This requires both
the accurate specification of a design and the construction of linking documents which
trace from a requirements specification to a system design and vice versa .

5 .2 DESIGN DOCUMENTATION

A major quality concern in system design is the expression of the design-either textually
AIMS or in terms of graphics . Any standard for system design should identify the individual
modules in a system : packages, subroutines or programs, and also identify the relationship
• To describe the main documentation used for system design . between the modules, i .e . how each uses facilities provided by another . Normally, this
• To describe some important techniques and documentation used to validate a system means that the calling structure of the system should be documented . As well as the
design against a requirements specification . calling structure, the system design should also specify the interface between modules,
• To outline the structure of the system design specification . i .e . what the parameters of each module are, and what global data areas are to be read
• To describe how the system design process is quality assured . from or written to . The functionality of each module should be specified in enough detail
so that the programmer responsible for coding the module should be able to take the
specification, the description of the interface and the list of calls to other modules, and
5 .1 INTRODUCTION produce program code without querying the design . An example of a specification for a
module expressed in a third-generation language is shown below :
System design is the process which gives rise to an architecture of a system . The aim
of system design is to produce a modular description of a system which will satisfy MODULE CheckupMain(DataArea: [1 :30] of Real)
the requirements described in the requirements specification-both functional and non- CALLS BringSensor
READS SensorFile : File of Real
functional-at the same time respecting the customer's design and implementation dir- WRITES Malfunction
ectives and the constraints specified in the requirements specification . Indeed, one of the FUNCTION
ways in which to judge the quality of a requirements specification is to see whether it This module brings in the last 30 items of sensor data read by the main sensor and held in SensorFilc
contains enough information for the designer to carry out his or her job efficiently . and places the values in DataArea . It then checks whether the data contained in DataArea is in
The units or modules which a design is expressed in depends on the application ascending order . If it is, then the boolean variable Malfunction is set to true ; it is set to false
otherwise .
area and the implementation media . For third-generation languages such as C, Pascal or
FORTRAN, the units will be subroutines and procedures, for fourth-generation languages
the units will be programs, and for languages which are object-oriented or contain some It is relatively easy to specify a system design notation which could be processed by a
object-oriented facilities, the units will be packages or classes : collections of procedures software tool and which would set up a code skeleton corresponding to the module, with
or subroutines which are associated with a particular collection of data . information such as the function of the module copied in as a comment .
As well as the design being expressed in terms of processing units it should also be This form of documentation could be modified for any target programming language ;
expressed in terms of the underlying data . In a third-generation language this data will a language which implements packages would have a system design notation which, for
either be in-memory data such as arrays or linked lists, or file-based data such as indexed each subroutine in the package, would contain details such as those shown above, together

86
88 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SYSTEM DESIGN 89

with a specification of the data that the package manipulates . For a fourth-generation Data manipulated by programs will consist of simple types such as integers, characters and
language accessing a relational database the format would again be similar, except that reals together with composite types which are records containing fields which have a certain type .
There will also be types which will be collections of objects such as strings, arrays or sequences .
instead of global variables, tables would normally be referenced . An example of a module
The final type are collections of objects where some ordering criterion applies .
specification for a fourth-generation language is shown below . The data specification for a third-generation target language is simple . It consists of the names
of any simple types followed by definitions of array and sequence types . Arrays are introduced by
PROGRAM FindNills (Monthlist : Table of months) means of the keyword array followed by square brackets into which are inserted the bounds of the
READS EmployeeDetails, Sales array, sequences are identified by the keyword sequence followed by the keyword of and the type
This program displays a list of employees who have achieved no sales for Aonthlist . The list of of the objects in the sequence. The types are then followed by the list of global variables with their
employees numbers will be displayed in decreasing order of total sales over the last five years . associated types . Finally a narrative describes the types and the global variables . Any reference to
SCREEN Emplist types and globals will be italicized . The keyword Types introduces the types . Globals introduces
the global variables and Narrative introduces a description of the types and global variables . An
example of this form of specification is shown below :
An extra feature here is that of the name of a screen . In systems which are based on
transactions it is important that the design notation used contains details of any input Types
and output. Salesman : record
A quality manual can take one of two attitudes towards a system design notation : SalesPersonName : String
Salary: Integer
either to have a single notation, or a number of notations, each of which are oriented
Worksld : Integer
towards a particular programming language . The decision as to which to adopt depends CurrentSales: Integer
on local circumstances and trade-offs . SalesForce : array [1 . .CurrentSalestot] of Salesman
SalesOrder = sequence of Salesman .

Self Assessment Question 5 .1 Can you think of an advantage and a Globals

disadvantage of using a single system design notation? HighestSales:Salesman
EmployedSalesmen : SalesForce
SalesQueue : SalesOrder
Solution Adopting one system design notation has the advantage that
it leads to lower training and familiarization costs, but for a software devel- Narrative
oper who uses a wide variety of programming languages the disadvantage Salesman represents data on a salesman currently employed by the company .
is that this could lead to an awkward fit with a number of languages and Salesforce is the collection of salesmen employed by the company .
could result in error-prone validation of a programmed module with respect SalesOrder is a queue of salesmen which is ordered on descending CurrentSales .
to its design .
HighestSales is a variable which contains the details of the salesman who currently has the highest
sales . This salesman should currently be at the head of the queue of salesmen .
EmployedSalesmen is a variable which holds the current employees of the company who are involved
The normal solution adopted by developers who have a wide spread of applications,
in sales .
both third-generation language- and fourth-generation language-specific, is to have two
SalesQueue is a variable which holds the queue of salesmen in decreasing order of sales .
notations : one biased towards the former, the other towards the latter .

The important point about this form of specification is that it should roughly correspond
5 .3 DATA SPECIFICATION to the data definition facilities in a programming language so that programming staff can
easily implement it . This is shown above where the target language is Pascal-like . When
A quality system should also provide standards for the specification of the underlying the language is a fourth-generation language, the data definition standard will be oriented
data to be manipulated by a system . This is vitally important not only for data-rich towards expressing data as tables .
systems such as commercial data processing systems, but also for those systems which
are not traditionally associated with large amounts of data, for example, system software
and telecommunications systems . There are two ways of specifying data depending on 5 .4 DESIGN PROCEDURE
the technology used . The extract shown below is part of a standard for the description
of the data in a third-generation language such as C or Pascal : Another component of the design process which the quality manual must address is
the process of carrying out the design . This should include instructions on how the
9 .1 Data specification design is carried out in a step-by-step fashion, what documents should be examined
It is vitally important that the data manipulated by our systems is adequately documented . when carrying out a design, and the mechanisms which enable the system designer to
90 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION
SYSTEM DESIGN 91

check a design . If the developer uses a standard development methodology such as the Table 5 .1 A verification matrix
American Yourdon Structured Development (Yourdon and Constantine, 1979) or the A B C D E F
British SSADM (Ashworth and Goodland, 1990), then much of this information can 11 x
be found in the manuals for this method . However, if the developer does not use such 1 .2 x x
methodologies, then the quality manual should provide a detailed description of what 13
should occur. An excerpt from a design procedure is shown below : 21 x x
22 x x
23 x
8 .1 System design procedure
24 x x
Our software systems are normally designed in a top-down fashion, unless a customer insists that
25 x
a standard development methodology is employed .
31 x x x
The technique that we use is known as functional decomposition . This involves writing down
32 x x x x x
an outline design using natural language, and then splitting up that description into less abstract
descriptions . For example, the outline design for a system which processes a stream of monitor
readings and produces a summary report is shown below :
be occasioned by factors such as the customer's environment, working practices, and business policy .

1 Open monitor file

3 . If you suspect that some part of the system that you are designing is going to give response time
2 Process data in the monitor file
problems, then that part of the system should be designed using the information hiding principle
3 Produce reports
described in the previous point . Normally, problems with response time will have been identified
during the risk analysis part of project planning, and the project manager will have allowed some
This represents the first level of the design, with each part of the design sequentially numbered .
extra development time which is intertwined with system testing . This time is normally used for
It is important to stress that this specification of processes occurs after the data architecture of tuning: changing the system to speed it up . Hence there is a need to localize design details of those
the system has been defined . The next stage in the design process is to decompose this description
parts of the system which may require modification to meet performance requirements .
further . For example, the second statement above can be expanded to :

4 . In carrying out the design process the designer should always bear in mind that the design is to
2 .1 Produce error data
satisfy a set of quality attributes . These are either functional or non-functional . Our standards insist
2 .2 Produce summary data
that the former are documented in appendix A .1 and that the latter should be placed in appendix
A .2 of the requirements specification .
The third step might be expanded to :

3 .1 Produce error report

3 .2 Produce average readings report 5 .5 DESIGN VALIDATION
3 .3 Produce unusual readings report

An important part of the design stage is the process of providing evidence that a design
Notice that each decomposition is numbered sequentially, and is traced back to the design state-
ment from which it originates . The process of decomposition stops when the designer feels that he
actually meets a requirements specification . In Chapter 1 I explained that the require-
or she has reached a module (function, procedure or subroutine) and no further decomposition is
ments specification gives rise to a number of quality attributes . During a software project
required . These modules are then documented according to the standard contained in section 8 .2 a system and its documentation will be examined or executed in order to check whether
of this document . There are a number of things to be borne in mind when carrying out the design these quality attributes are present .
process .
It is important that during design as many of these checks are carried out as pos-
1 . Modules should be designed so that they only carry out one function . We have found that such
sible . By doing this the developer is saved the embarrassment and increased resource
modules are easy to maintain . Modules which are multi-functional tend to require major rework expenditure of discovering major problems late in a project . The major validation task
when requirements changes occur. during design is that of determining whether the system design represents a system that
meets its functional requirements . There are a number of ways of doing this . The first is
2 . If some aspect of the system to be developed has a high probability of change, then that aspect
by constructing a document known as the verification matrix . This is a two-dimensional
should be hidden away in a module . For example, if you are developing a system for processing a
queue of orders from customers with the queue ordered on the time that items join the queue and
table which lists both the functions of a system and the modules contained in it . An ex-
you know that, in the future, the customer may change the ordering criterion, then the details of
ample of such a matrix is shown in Table 5 .1 . At the top of the matrix is a row containing
how the queue is represented should be hidden away in the small number of modules which add an the names of the modules in the system . The first column contains the numbers of the
item to the queue, remove an item from the queue, check whether the queue is empty, and count the functions specified in the requirements specification . An x in a cell of the matrix indicates
number of items on the queue . It is important that such details should not be placed throughout
that a particular module is executed when the function is exercised . For example, the
the system design, but hidden in a relatively small number of modules . Section 7.1 of a require-
second row shows that modules B, C and D are exercised when function 1 .2 is invoked .
ments specification generated by any of our projects will contain a list of likely changes which may
This matrix would be produced by the designer after the design has been completed . It
92 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SYSTEM DESIGN 93

provides a form of validation that, as a minimum, the design meets all the functional
Self Assessment Question 5 .3 Why is this not the correct action to
quality attributes which have been identified during requirements analysis .
take?

Self Assessment Question 5 .2 Can you think how a verification matrix Solution The change to the system which was applied to make acceptance
is of help in identifying whether the designer has included any extra func- test 5000 work may have inserted other errors into the system-errors which
tions in the system which are not in the requirements specification? would only be detected by acceptance tests prior to test 5000 . If a devel-
oper does not have traceability built into his or her documentation, then
Solution These are often manifested in modules which do not have any all the acceptance tests previous to test 5000 will need to be re-executed . If
x entries in their columns . the developer used the verification matrix the testing effort is substantially
reduced . All that is required is for the tester to examine which modules
have been changed and then rerun all those acceptance tests which involve
It is worth issuing a warning that, although the verification matrix is of great help in those modules .
determining whether redundant functions are present, it does not provide 100 per cent
assurance . For example, the designer may have structured the system in such a way that
extra functions which are not required are supported by modules which also support Many software engineering textbooks stress functional checking . However, many software
the real functions of the system . However, this would usually be manifested in multi- systems give rise to quality attributes which are non-functional in nature (many of these
functional modules that would normally be discouraged by the quality system . were discussed in Chapter 1) . It is important that the software designer provide some
For a large system the verification matrix will be big and take a considerable time evidence that these quality attributes are implemented in the design produced . This evi-
to produce . There are two solutions to this problem : first, the matrix can be automated dence is usually provided by a separate document sometimes known as the non-functional
by using a spreadsheet ; and second, the size of the matrix (the number of cells) can be quality attribute validation document, but sometimes implemented as a subsection in the
substantially reduced by splitting it up into a number of smaller matrices which would system design specification .
correspond to functional subsets of the requirements specification . Some examples of these quality attributes are shown below :
The verification matrix is also of help during testing . For example, the functional
Changes to the staff details in the system should not involve any changes to the program code of
quality attributes described above will give rise to system tests and acceptance tests . If a the system .
module in the matrix has a small number of x entries, then that module is not going to
be thoroughly exercised by system test data . The staff concerned with testing the system The system should produce staff summary data records which can be transferred directly to a row
may decide that they would like that module to be rigorously tested . There are, then, a on a Lotus 1-2-3 spreadsheet .
number of options that could be adopted :
The system should occupy no more than 20k of main memory storage .

• Obsessively test the small number of functions that the module implements during The maximum response time for any Staff Retrieval command is 2 sec .
system testing .
• Ask the programmer who is to produce the module to test it more rigorously than The first quality attribute is associated with the maintainability quality factor . A designer
usual . would demonstrate that the system design met this attribute by explaining that staff
• Ask the integration testers to exercise the module thoroughly during integration test- details are held in a table on a file which is separate from the program code .
ing . The second attribute is associated with the interoperability quality factor . The de-
• Make the module the subject of a code review . signer would demonstrate that this attribute has been satisfied by showing that the
section of the data architecture which describes the records emitted by the system has
The verification matrix implements a form of traceability which is vital in a software the same format as that expected by the spreadsheet .
project, as it relates functions to the code which implements them . In order to see why this The third attribute is associated with the performance quality factor . A designer
is, consider what happens during the process of acceptance testing when an acceptance would demonstrate that the design meets this requirement by estimating the code size
test fails . Assume that acceptance test 5000 fails . The professional software developer will of each module in the system and comparing it with that specified in the description of
then modify the system after the cause of the error has been determined . He or she will the quality attribute .
then run the acceptance test to confirm that the modification has been applied correctly . The fourth attribute is again associated with the performance quality factor . For a
Most developers will then move on to test 5001. This is not the correct action to take. simple system, the designer would calculate the algorithmic complexity of each module
and make a rough estimate of the execution time of each command . For more complex
systems, the project may call for advanced techniques, such as simulation, to be used .
 94 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SYSTEM DESIGN 95

So far we have described the fact that the production of documentation is able every design review . It must be stressed that only a selection from a real checklist is
to provide a framework in which quality controls for design can be implemented . The shown:
controls themselves can be implemented in a number of ways : a senior member of staff
could sign off the documentation produced by the designer, or a formal review of a design 3 .4 System Design Checklist
could take place . Does the system implement its functional requirements as detailed in the functional quality attr-
ibutes section of the quality plan?
This review would not be attended by the customer .
Does the system meet its non-functional requirements as detailed in the non-functional quality at-
tributes section of the quality plan?
Self Assessment Question 5 .4 Why wouldn't the customer be invited Is the interface with entities outside the system adequately defined? For example, are the specifica-
to a design review? tions of modules which interact with hardware correct and do they provide the right level of detail
for the implementation team?
Solution Because they are highly technical and solely involve the devel- Are the interfaces of modules correct?
oper's solution . Can a programmer take the functional description of a module and implement it in the chosen
programming language?
Are objects described in the requirements specification documented in the data design?
Normally a design review would be attended by the designer of the system, or the member
Is the architecture of the system sufficiently modular? Does it, for example, contain modules with
of the design team who produced the design ; staff responsible for testing the system ; a
small interfaces which carry out only one function?
designer from another project ; and the senior programmer responsible for implementing
the system . The documents given to the members of the review team would be the Is there a missing level of abstraction in the design? This is often indicated by a module calling
more than six or seven modules .
requirements specification, the quality plan, and descriptions of external interfaces such
as communication lines . Normally, design reviews only have enough time to deal with a
small section of a design at a time . Self Assessment Question 5 .5 Which of the first two questions above
A design review has a number of aims . The first is to check that the standards and would be the easiest to answer?
procedures adopted by the project have not been violated . Normally this is detected by
staff preparing for a review who notify the chair at the beginning of the review of any Solution Generally it is easier to check functional correctness : a designer
non-compliances that they have found . can talk through what a system does quite easily . On the other side of the
The second aim is to ensure that any non-functional quality attribute specified for coin, some non-functional requirements are awfully difficult to check from a
the system has been implemented . For example, the customer may have specified that design, for example response time and memory occupancy ; however, there
he wants high maintainability in a system . A major task of the review meeting is then tends to be fewer of these to check than functional requirements . The an-
to look at the design and point out any areas of the system which are difficult to change . swer really depends on the system-1 would adjudge a dead-heat normally .
The third aim of the design review is to ensure that the functional characteristics
specified in the requirements specification have been met in the design . This is often done
by the designer of the system talking through these functions and detailing which parts
of the design are executed when a particular function is exercised . 5 .6 THE STRUCTURE OF THE SYSTEM DESIGN
The nature of a design review really depends on the amount of documentation that SPECIFICATION
has been produced by the designer . For example, if the quality plan has insisted that
the designer produces a verification matrix, then a consideration of whether the system The document which is finally produced by the designer is the system design specification .
actually meets its functional requirements can either be skipped, or the designer could Many of the components of this document have already been discussed in this chapter
talk through a small number of functions with the review team . If not, then a large and are now summarized in this section . The headings of the specification should be :
amount of time spent in the review would involve examining each functional quality
attribute with the designer explaining how the design implemented the quality attribute 1. Title
and the members of the review team pointing out problems or pronouncing themselves 2. Contents
satisfied . 3. Introduction
It is important to provide a checklist for staff carrying out the design review process . 4. About this document
This enables them to be clear about what is being examined in a review . 5. Glossary
A fragment from an example review checklist is shown below . It contains major 6. Module architecture
questions about third-generation language designs which should be on the agenda of
 96 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SYSTEM DESIGN 97

7 . Data architecture 5 .7 QUALITY MANUAL REQUIREMENTS

8 . Interface to entities outside the system
9 . Quality attributes and the system architecture . 1. Functional design standard
2. Data design standard
The introduction should be brief, and describe the main inputs into the design process . 3. System design procedure
For example, if security was the overriding concern in the requirements specification this 4. Design validation procedure
should be stated here . Any unusual features of the document should also be described 5. System design review procedure
here ; for example, the first use of a design notation which was insisted upon by the 6. System design review standard
customer should be mentioned . An example of part of an introduction is shown below : 7. System design review checklist .

The design described in this document details a system which controls the injection of drugs into a Items 1 to 3 are usually subsumed under a single system design standards and procedures
hospital patient who is normally in a critical condition . The system is a safety-critical one in which
a design or programming error could cause the death of the patient . Consequently a large amount document . Also, items 5 to 7 are normally grouped together in a system design review
of resource is to be spent on the validation of the system ; in particular the validation of the design standards and procedures document .
against the requirements specification .
Another feature of the system is that it needs to fit into a relatively small amount of memory
and has well defined response rates which are critical : if a drug is not injected at specific times then
5 .8 SUMMARY
the patient will suffer and could even die .
A novel aspect of the system is that the pump used for the injection of drugs is new and this
is the first time that we have produced a system for this type of device . The process of design involves members of a development team deriving a system archi-
tecture which consists of discrete chunks known as modules . There are two aspects to the
The section About this document should describe any naming conventions ; the scope of quality processes concerned with system design . The first is that they should encourage
the document ; the main items in the document ; the intended readership ; any legal and specifications which reflect the requirements contained in the requirements specification :
contractual issues associated with the document ; any relevant standards, both internal there should be good traceability back to functional and non-functional quality factors .
and external ; how the system design is to be maintained and issued ; and, finally, the The second is that the system design should be an adequate basis for tasks which are to
distribution list for the system design . be carried out later ; for example, programmers who read the functional specification of
The Glossary should contain a list of design terms and their definitions . An example a module should, ideally, be able to program that module without reference back to the
is shown below : system designer .

Global data . This is data which is accessible by all the modules in a system .
5 .9 FURTHER READING
Module architecture describes the chunks of software into which the system has been
partitioned, together with the interfaces between the chunks . Data architecture contains Pressman (1994) contains an excellent chapter on software design that describes a number
the specification of the data manipulated by the modules in the system . Interface to of popular design methods . Parnas and Clements (1986) is an excellent article which
entities outside the system describes the way in which the system interfaces with the points out that although system design documentation describes a linear process from
remainder of the system : hardware, other software, and users . the receipt of requirements analysis to the completion of system design, this is just faking
For users, this section will contain details of the screens and outputs produced by it : the design process is quite non-linear and involves a large amount of backtracking and
the system, for hardware it will describe the format of data passed to the hardware discarding of alternatives . This is an antidote to the views of a large number of QA
and a description of the modules which carry out interaction with the hardware-either practitioners of my acquaintance, who imagine that a project progresses linearly, and
modules provided by the hardware manufacturer, or modules which are to be written by attempt to develop quality management systems which have this model of development
the developer. For software, it will describe the format of records passed to any software at their heart . Yourdon and Constantine (1979) is quite an old book but still contains
outside the scope of the system and a specification of the modules which may interact a lot of wisdom about the design process . Easteal and Davies (1989) is a good, short
with this software . introduction to modern design ideas while Page-Jones (1988) is a much longer treatment
The final section, Quality attributes and the system architecture, should include docu- which also includes material on requirements analysis .
mentation which demonstrates that the designer has, in fact, considered and implemented
the quality attributes which have been identified during the early stages of the project .
This documentation may include a verification matrix, but will also contain outline de-
scriptions of why the designer feels the architecture meets the non-functional quality
attributes .
98 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

PROBLEMS
5 .1 How would you use a system design review to check that the interoperability quality factor was
present in a design?
5 .2 An error which is often made by system designers is to develop a design which implements functions
that are not specified in the requirements specification . How would you check a design to ensure that
this does not happen?
5 .3 Write down ten items that you would include in a system design checklist for a system design which
will eventually be implemented in a third-generation programming language such as COBOL or Pascal .
5 .4 Develop a standard for a system design notation for your favourite programming language .
5 .5 Write a briefcase which proposes that your company's design documentation should be augmented
by the use of a verification matrix .
 s
DETAILED DESIGN AND PROGRAMMING

AIMS

• To describe the nature of a detailed design notation .

• To describe the nature of a programming standard .

6 .1 INTRODUCTION

At this stage in the software development process the development team will have pro-
duced a system design and a requirements specification . The remaining developmental
tasks involve converting the system design into program code . This can be carried out
in either one or two stages . If two stages are involved they are normally called detailed
design and programming. The former involves specifying the detailed flow of control in a
system in a form which can be easily translated into program code, while the latter sim-
ply involves the conversion of the individual module specifications . A two-stage process is
usually implemented by developers who have adopted portability as an important quality
attribute for all their projects-including across a range of programming languages .

Self Assessment Question 6 .1 How does a detailed design notation help

portability?

Solution Modules expressed in a detailed design notation should, provided

the notation is good, be translated easily into a wide variety of program-
ming languages .

99
1 00 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION DETAILED DESIGN AND PROGRAMMING 101

Detailed design is also used by developers who employ programming languages which are notation express their designs at a higher level where a single line of detailed design may
not very readable, or which do not provide adequate structuring facilities . Probably the be equivalent to four or five lines of program code and where in-code commenting is not
best example of a family of languages which come into this category are the assembler too intrusive ; or, if they use a detailed design notation of the level shown above, then
languages . they keep the documentation containing the notation separate from the program code .
It is important to point out that this chapter concentrates on third-generation tech- The one example where annotation of the source code is useful is if the code is
nology . The quality assurance aspects of fourth-generation technology are dealt with in expressed in an assembler language . Most assembler languages allow the programmer to
Chapter 9 . write comments side by side with the program code as shown below :

CLS rflg ; Clear RangeFlag

6 .2 DETAILED DESIGN LD cc
ADD me
A good detailed design standard should be able to be easily translated into program ST cc ; Add MonthCount to CurrentCount
code, and also be able to express a wide variety of control structures such as while, for,
repeat, if, and case constructs . An example of a fragment taken from a detailed design The many programming limitations of assembler languages can be greatly mitigated by
language which the author has used is shown below : copying a detailed design into a file, adding the symbols which mark a comment, and
then programming the assembler code into the file .
Clear RangeFlag Normally, the notation used for data design that is employed during system design
Add CurrentCount to NlonthCount is carried through to detailed design . The procedure for detailed design should describe
Repeat the process of receipt of a system design ; the actions that should be carried out when
Call MonitorRead(Reading) the designer detects a problem with the detailed design ; and the process of checking the
Check whether Reading is between 10 and 100 detailed design against the system design . There are three ways in which this checking
If the Reading is between these limits then can be carried out : in a full review similar to a system design review ; as a structured
Add the Reading to CurrentCount walkthrough in which the producer of the detailed design takes a meeting of peers through
Set RangeFlag the detailed design, executing it as if he or she were the computer ; or as a simple meeting
Endlf between the detailed designer and the member of staff designated to sign off the design
Call CheckStatus(StatusFlag) as being correct . The third course is usually adopted . The quality manual should contain
Case StatusFlag of procedures for all three methods .
violation : Call ViolationHandler
normal : Continue
OutOfRange : Call RangeHandler Self Assessment Question 6 .2 Why do you think that most developers
EndCase do not use full technical reviews to validate program code, but usually stick
Until Call MonitorHang(CurrentMonitor) to the signing off of modules by a single member of staff?

Here the keywords which introduce the constructs are in bold . The individual detailed Solution The reason is that if you review every item of code in a technical
design fragments of each module in a system can easily be inserted as comments into review an immense amount of resource would be used up . Programming
the source code of a system . However, it is worth issuing a word of warning about this errors are less serious than requirements or design errors, so a company
practice . For high-level languages it can actually make the program text less readable . that uses technical reviews will concentrate their effort on reviews of the
For example, consider the source code which implements the first few detailed design requirements specification or the system design .
statements above, with the statements inserted as comments :

(* Clear RangeFlag *)
RangeFlag := 0 ; 6 .3 PROGRAMMING
(*Add CurrentCount to MonthCount *)
MonthCount := MonthCount + CurrentCount ; The testing aspects of unit programming are dealt with in Chapter 7 . Also, Chapter
13 describes the process of writing a programming standard within the context of the
This level of commenting is far too high, and actually detracts from readability in that practice of standards and procedure development . However, for the sake of completeness
it can hide the structure of a module . Most developers who employ a detailed design it is worth describing the main components of a programming standard :
 102 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION DETAILED DESIGN AND PROGRAMMING 103

• Bureaucratic comments. There is a need for each module in a system to be annotated insistence that a module had a maximum line length-usually no more than the line
with comments which include the name of the programmer ; his or her telephone length of one sheet of line printer listing paper .
extension ; the date that the module was completed; the version number if the module • Limits on global variables . This is another important feature of a programming stand-
has undergone a series of changes since it was first programmed ; and the function of ard . Programmers should limit the amount of access they make to global variables . A
the module . Normally, the function will be expressed in terms of how the parameters system which contains modules that have a number of both read and write references
of the module or any accessed global variables are used to change other parameters, to global variables is exceptionally difficult to maintain . Often, during maintenance,
global variables or files . It is also a good idea to write the names of the modules which a requirements change gives rise to a change to a global variable ; for example, if the
call the module as comments . A good standard should also instruct programmers to variable was a record, then the change might give rise to some extra fields in the
comment code which is difficult to understand, for example program code which uses record . This means that the change will usually impinge on the modules which refer
an arcane feature of the programming language to achieve a speed-up . to the global variables that have been modified . If there are a lot of references, then
• Unsafe programming constructs . This mainly concerns the goto statement ; however, this can give rise to substantial browsing and modification .
some programming languages also have escape facilities which are, in effect, disguised • Indentation limits. Modules which have a large number of control structures nested
gotos. The standard should direct the programmer to avoid these constructs, or only within each other are terribly difficult to understand and debug . Many programming
to use them within very specific contexts such as when a module of a safety-critical standards insist on an upper limit to the depth . This usually means that the pro-
system discovers an emergency condition . An example of such a condition is when a grammer has either to recode a module to eliminate the extra nesting, or replace the
chemical monitoring system discovers the fact that a reactor has all its inlet valves offending control structures that make a call to a module . Both solutions lead to more
open and its outlet valves shut, with reactants being poured into a potentially ex- readable code .
plosive reactor . In this case, the programmer does not want to carry out a graceful
exit from the module . What is required is a fast goto to some program code which
handles the emergency condition . However, it is worth stressing that such examples Self Assessment Question 6 .3 Why do you think that a well designed

are rare, and the norm should be to avoid constructs which make the program code system should contain modules which carry out one function':
unreadable .
• Layout conventions. A good programming standard should describe how a module Solution If you have multi-functional modules, then you will find that
should be physically displayed on the page . This not only describes the use of in- a change to one of the functions in the module will normally affect the

dentation of control constructs, but also the display of all non-loop initializations at, other functions in the module, thus necessitating further reprogramming .
the head of a module ; the initialization of loop variables just before the loop starts ; Hence, multi-functional modules tend to consume more effort during activ-

any naming conventions for identifiers and the names of modules ; the declaration of ities such as maintenance .
constants ; and the incorporation of fragments of detailed design as comments into
the program code .
• Defensive programming conventions . A programming standard should describe the These, then, are a selection of the major components of a programming standard . They

mandatory elements of a defensive programming strategy ; for example, it should de- are usually intended to make software more readable and also more maintainable . One
other topic remains : that of ensuring that a module or chunk of code is correct-that it
scribe the layout conventions for code which detects error conditions, and allowable
associated actions such as transferring to an error printing routine . It should also meets its design specification .
describe what defensive code is not allowed . For example, for many real-time sys- There are two options available to the software developer . The first is unit testing
(this is described in Chapter 7) . The second is to have a full technical review to which
tems, the inclusion of defensive programming code during development, which is then
the following members of staff are invited ; the programmer ; another programmer on the
removed during operation, can lead to subtle timing changes which may force errors
same project ; a programmer from another project ; and the designer of the module or
during operation .
• Limits on conditions . An important source of error during maintenance occurs when system. The general form of a technical review which examines program code is that the

staff have to read and understand complex boolean conditions which contain a large participants are asked to point out any poor programming practices or any violations of
programming standards . The programmer then describes the design specification that he
number of boolean and and or operators .
• Limits on the functionality of a module . As has been mentioned in Chapter 5 : one of or she was given, and then walks through the program showing how it would be executed
by the computer . The programmer will describe how other modules were called, what
the important features of a well designed system is that the modules in the system
normally carry out only one function . If a programmer is given responsibility for results were expected from these calls, how global variables were updated, how data
was obtained from the environment in which the module was embedded, and give a
producing a chunk of code which contains a number of modules, then part of the
description of the detailed calculations that were performed . As this description proceeds,
programming standard should insist that each module has functional cohesion . This
feature was rather unsubtly enforced in the early programming standards by the the review participants will detect errors in the program meeting its design specification .
As with any technical review, any problems which occurred would be documented, and a
1 04 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION DETAILED DESIGN AND PROGRAMMING 105

decision about the outcome of the review-whether it « as completely satisfactory, largely PROBLEMS
satisfactory (with some minor problems), or unsatisfactory-made by the chairman .
An important point to be made about this form of review is that if all the program 6 .1 Write down the part of a programming standard for your favourite third-generation programming
language which deals with the bureaucratic information required at the front of a module .
code in a system were reviewed in such a way, then the developer would be committing a
6 .2 Write down the headings and subheadings for a programming standard for your favourite third-
large amount of resources to validation at a late stage in the project . Reviews are more
generation programming language .
productive at the front-end of a project where major errors are often discovered, and
6 .3 If your company decided to have detailed design reviews what do you think would happen in these
it is often a good idea to commit most of your validation resources to the validation of meetings?
the requirements specification or system design . There are a number of ways to reduce
expenditure on code reviews . The first is to review only modules which are potentially
error-prone . The criteria used for this will depend on the company. Companies that I have
dealt with have reviewed large modules, modules with complicated control structures,
and modules which are produced either by new-staff or by staff who have a history of
developing error-prone modules .
An alternative to a selective review is to have informal walkthroughs on all the
program code that only involves two or three programmers . In such a walkthrough, a
module is executed and problems notified to the programmer who then rectifies them . It
is assumed that after these reviews the programmer carries out any necessary rework .

6 .4 QUALITY MANUAL REQUIREMENTS

1. Detailed design standard

2. Detailed design procedure
3. Detailed design review procedure
4. Detailed design review standard
5. Programming standard
6. Programming procedure
7. Code review standard
8. Code review procedure .

If the developer uses a number of programming languages, then there will be a pro-
gramming standard for each language . If the developer uses walkthroughs or informal
meetings, then there will be standards and procedures for these . Items 1 and 2, 3 and 4,
5 and 6, and 7 and 8 are normally combined into four separate documents .

6 .5 SUMMARY

The main components of a quality system which deal with detailed design and program-
ming are standards for the layout of the detailed design notation used by the developer
and of any programming languages used . It is also important that procedures are speci-
fied which deal with the process of producing detailed designs and coded modules, and
the process of querying poor module specifications which have been produced by design-
ers .
 TESTING 107

7
Self Assessment Question 7.1 Do you think that unit testing, some-
times known as module testing, should be a black box activity, a white box
activity or both?

Solution It should be both . The tester will check out the functions of
TESTING a module with test data (black box) and then check that these tests have
achieved some structural metric such as 100 per cent of statements have
been executed . If the metric has not been reached, then further data is cre-
ated which exercises the parts of the module which have not been executed
(white box) .

There are five main testing activities : unit testing, package testing, integration testing,
system testing, and acceptance testing . Unit testing is the testing of a module in isolation .
For a fourth-generation language a module will be a program, and for a third-generation
language a module will be a subroutine . A programmer tests a module against a speci-
fication which can be found in the system design . Normally, test data is chosen which
thoroughly explores the function of the module and, as a final check, the programmer
confirms that the tests have achieved an adequate coverage of some structural metric . The
structural metric that is almost invariably chosen is the statement, with many software
AIMS
developers insisting that unit tests cover 100 per cent of all the statements in a module .
Package testing is a technique used in systems which are structured into units known
• To describe the nature of software testing .
as packages . A package is a collection of modules together with a data area . For example, a
• To outline the relationship between software testing and quality assurance . package which implements a queue would contain a data structure that implemented the
• To describe the quality documentation required to support the testing process . queue-perhaps an array-together with modules which access this queue ; for example,
• To describe the main testing activities .
modules which add an item to a queue, remove an item from a queue, count the number
of items in a queue, and initialize a queue .
Once a package has been put together it can then be used by programmers who are
7 .1 INTRODUCTION
implementing the functionality of a system . They use the package by calling the modules
associated with the package ; they do not access the data structure of the package directly,
Without a doubt testing is one of the most widely used quality controls . Indeed, many
because this leads to a poorly maintained system : a system which contains references to
quality assurance practitioners point out that, in the past, an over-emphasis on system the data structure throughout its code is a maintenance nightmare . When a change to
and acceptance testing as a quality control led to serious errors such as requirements that data structure occurred a programmer would then have to examine large segments
errors being detected at too late a stage in the software project, after a large amount of of code and modify the references . However, a package-based organization means that all
resources for the project had been committed .
changes to the underlying data are localized to the package itself, saving a large amount
There are a number of testing activities which are employed on a project . These of effort .
were described in detail in Chapter 2 ; however, it is worth outlining them here before
Packages are a facility of a number of modern programming languages such as Ada
examining the relationship between testing and quality assurance . and Nlodula 2, where, if a programmer did try to directly reference the data in a package,
First, a definition : testing is the application of a quality control which involves the
a compiler error would result . Developers with less modern languages such as C can ensure
execution of a system or part of a system . There are a number of testing activities : these that their programmers follow a package discipline by instructing them not to directly
can be categorized as either white box or black box . A white box activity is one which
access data structures by means of a programming standard such as the one described
relies on a knowledge of the structure of the program code that is being tested . A black in Chapter 6 .
box activity is one which does not require this knowledge . Package testing is a form of testing which ensures that the modules in a package
work correctly in conjunction with each other . For example, testing the queue package
described above would involve a number of tests, one of which would check that when
a queue had an item added and then removed the queue remained the same . Package

106
 1 08 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION TESTING 109

testing is a black box activity, as no knowledge of program code is required . An extract said that, yes, he had attended a lecture on the topic, and that the lecturer had devoted
from a procedure which describes package testing is shown below : the hour to giving them hints and tips on the selection of program test data .
Testing is a large component of quality assurance, but to place it in context it is worth
2 .3 Package testing restating the nature of quality assurance : that it is the definition of quality attributes
which are either customer-related or developer-related, and the definition of quality con-
We normally develop systems as collections of packages . These contain data definitions and the code
for the modules which access these data definitions . We currently use Ada for many projects and
trols which provide explicit evidence that those attributes are present, either in the whole
this language offers a facility whereby packages can be defined and syntactically checked . All our system that has been developed, a part of the system, or in the documentation that has
Ada projects make extensive use of this facility and a form of testing known as package testing is been produced for it .
always to be applied . Testing is a quality control which addresses many of the quality attributes that are
This form of testing involves the programmer compiling the package and writing test cases functional in nature, together with some which are non-functional . Unit testing checks
which check that the package modules work in conjunction with each other; top-level (scaffold)
that a module carries out the function that is specified for it in the system design, for
software will be required to be written in order to invoke the modules . For example, a module
which enters a name into a table would be executed and then a module which checks that that example functional correctness of the module : a developer-related attribute . Package
name has been correctly entered would then be executed after this . Guidelines for the selection of testing verifies that the functions of the modules in a package are correct by checking
package tests and details of possible combinations of modules can be found in section 2 .3 .4 . their execution in conjunction with each other : again, a developer-related attribute . In-
The tests that have to be carried out have to be documented on the form shown in section tegration testing checks that interfaces are correct and that functions implemented in
2 .3 .1 . For each test the programmer has to describe what the test is intended to achieve and the
the partial system that has been constructed are correctly implemented . The former
results expected . The test data, the test outcome and the source code for the test should all be
stored in files described in the naming convention section of this manual (section 1 .4) . is a developer-related quality attribute ; the latter a customer-related attribute . System
and acceptance testing checks that the functions and constraints in a system have been
Integration testing is the process of testing a system as it is being integrated : when correctly implemented : these are also customer-specific quality attributes .
It is worth pointing out that not all testing involves functional attributes . Some
modules are added to it a few at a time . Normally, an integration test occurs when a
examples of testing which involve non-functional attributes are :
number of modules have been added ; the main aim of the integration test is to check
that the interface between the system that has been built up, and the modules that have
• Executing a module to check on its execution speed in order to ensure that the final
been integrated, is correct. However, integration testing is also an opportunity for early
functional testing : a developer may have discovered that after a particular integration, response time of a system is acceptable .
• Executing commands in a system to provide confirmation that the quality attribute
enough modules now reside in the partial system that has been built up to implement
one of the system functions . Therefore, a subsidiary testing task often carried out after of response time has been met .
• Keeping data on failures that occurred during system testing in order to predict the
an integration is to execute a preliminary test of some system functions .
System testing is the process of checking that the system constructed matches user
reliability of a system .
requirements . For this, the functional quality attributes and non-functional attributes • Running a whole series of tests with the real users of a system and, at the same time,
which are of concern to the customer are tested by executing the system . Normally keeping data on the number of errors that they made in invoking system functions .
This would be carried out in order to check that a quality attribute related to usability
this execution is of the whole system, but not necessarily in its target environment . For
example, if the system was to process data from a nuclear reactor, or from a large number was present .
of transaction clerks, the input data from these sources would be simulated, either by
This is just a small selection of non-functional quality attributes which can be validated
hardware or software . System testing is a black box activity .
as a result of testing . However, it is certainly true that the vast majority of testing
Acceptance testing is the demonstration to the customer that the customer-related
activities on a software project are aimed at implementing controls on quality attributes
quality attributes have been properly implemented . To do this, a subset of the system
which are functional-both customer-related and developer-related .
tests is agreed on by the developer and the customer, and is then executed on the system
within its target environment, with the customer witnessing and signing off the tests .
Acceptance testing, like system testing, is a black box activity .
7 .3 UNIT TESTING

7 .2 TESTING AND QUALITY ASSURANCE Unit testing is mainly oriented towards the checking of the function of a module, although
a quality plan will call for response time testing of modules for systems which are time-
critical . The standards and procedures for testing will govern how test data is to be
There is a commonly held belief that quality assurance and testing are synonymous .
selected, how tests are to be documented, and how to resolve any problems should the
I remember asking a student who had recently completed a degree course in software
input to the testing process-the module specification-be poorly expressed .
engineering whether he had been taught anything about software quality assurance . He
 1 10 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION
TESTING 111

The main medium for holding testing information is the unit folder or programmer's Table 7 .1 A tabular representation of data values
notebook. The main testing details that should be included in this document are : Key value Key in table Result

• < 1000 error message

The rationale for the test data that was chosen .
> 2000 error message
• The location of the test data for the module . valid key no not found
• The location of the test outcomes that were generated for the module . valid key yes found
• Documentation of the test coverage .
• Bureaucratic information such as the name of the programmer and the date of the Table 7 .2 Test data for a sort problem
tests .
n Resul

An example of part of a procedure for unit testing is shown below : 1 sorted

maxint/2 sor ed
7 .5 Subroutine testing maxint sor ed

COBOL subroutines are always tested singly . We call this testing subroutine testing ; it is performed
by the programmer who developed the subroutine . It involves the programmer enclosing the sub- will be out of the range of the table . Table 7 .2 gives another example . It represents the
routine with program code which: calls it, transfers test data to the subroutine and displays any tests for a module which has two arguments : arr, an array of integers which contains up
test results . In this quality manual this extra code is known as scaffold software. The tests that are to a maximum of maxint items, and an integer n . The module sorts the first n items in
carried out are known as functional tests . They check that the subroutine is functionally correct . the array . The first column of Table 7 .2 details the values of n and the second column
The only documentation which is used to develop these tests is the subroutine specification pro-
duced by the member of staff who acts as the design authority . This documentation will be given
details the outcome which occurs . Such tables are useful for exploring combinations of
to the programmer before the detailed coding of the subroutine is carried out and is incorporated data values and are normally used to develop an initial data set . There are, however,
as a comment in the header of the subroutine . two further methods for deriving data that tend to be more revealing when it comes to
Detailed guidelines for the generation of test data can be found in section 8 .8 of this quality discovering errors .
manual . Examples of test data can be found in appendix C of the same manual . The first is known as error guessing. Here, the tester derives test data which suppos-
When the programmer has felt that he or she has fully functionally tested the subroutine edly contains an error. In the sorting example above, error guessing data for n would be
an item of documentation known as the unit folder is completed . This contains information such
as descriptions of test carried out . Full details of the information that has to be entered into a negative integer, zero, and an integer greater than maxint .
this document can be found in section 7 .7 of this quality manual. When this documentation has
been completed the unit folder is submitted to a designated member of staff for checking . This
will normally be the senior programmer who has been attached to the project . However, for small Self Assessment Question 7 .2 A module processes an integer represent-
projects this could be the team leader . If this member of staff signs off the unit folder it is submitted ing the number of a record in a file and then prints out all the records
to the project library and comes under configuration control . Full details of this process can be found from that point . If the tester selects an integer greater than the number of
in section 11 .8 of this quality manual .
integers in the file would this be an example of error guessing?

The rationale for test data selection is a description of why certain items of data were Solution Yes it would : the data represents an error .
chosen to test the unit . This should not be too long : usually between 100 and 200 words
describing the strategy adopted . There are a number of strategies which can be used .
The first is very simple : all it involves is writing down a tabular representation of The second is based on a technique known as equivalence partitioning. In order to un-
the individual data items that are processed by a module and deriving tests based on derstand this technique, consider Fig . 7 .1 . This shows the data space for a particular
combinations of these data items . As a simple example, consider the testing of a table
module, which represents all the possible data values that can be processed by a module
module . Assume that the table associates an integer key that lies between 1000 and 2000 or a program . The data space is partitioned into three subspaces . Each subspace repre-
with an employee record . The aim of the module is to check that a particular employee
sents data which is processed in the same way, e .g . one of the subspaces may represent
with a particular key is in the table . If a key that lies outside the range is given, then an the processing of a particular type of error data . Test data generation, based on equiva-
error is displayed . Table 7 .1 shows the different outcomes . Here the various conditions
lence partitioning, involves the tester generating data which either lies on, or is close to,
that affect the module are displayed and all the possible combinations of conditions are the boundary of the subspaces . Test data generated in this way is capable of detecting
combined . For example, the third row states that when a valid key is provided, and the
common programming errors, such as a loop that executes one too many times, or one
key is not in the table, then it will not be found . The first and second rows of the table
too few times, or a condition in which a < operator has been written rather than a <
contain a blank ; this is a `don't care' condition, for which it is irrelevant whether the key operator . As you will recognize, these are all common errors committed by programmers .
is in the table ; indeed we can be certain that it is not, since the key for these two rows
 1 12 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION TESTING 113

Table 7 .3 The initial test set Table 7 .4 The initial test set supplemented by error data
arr n nt m result arr n num result

num in table n = maxint found num in table n = maxint found

num not in table n = maxint not found num not in table n = maxint not found
num in table n =maxint/2 found num in table n = maxint/2 found
num not in table n < maxint not found num not in table - n < maxint not found
nurn in table n = 1 found num in table n = 1 found
num not in table n = 1 not found num not in table n = 1 not found
n = 0 error
n > maxint error
As an example of the three techniques in action, consider the testing of a module that
has three parameters : the first, arr, is an array of integers which contains no more than Table 7 .5 The initial test set supplemented with boundary data
maxint integers ; the second, n, is an integer ; and the third, num, is another integer . The
module searches for num in the first n locations of arr . If it is contained there, then a 1 is err n num result
returned, otherwise a 0 is returned . The initial test set might look like that in Table 7 .3 . nurn in table n = maxint found
This is quite a good test set as it explores the behaviour of the table over its full size . num not in table n = maxint not found
Notice that the value of num has been marked as `don't care' . A number of further tests num in table n = maxint/2 found
num not in table n < maxint not found
can be generated from error guessing . The table can now be supplemented with the data
num in table n = 1 found
which represents an empty array and an over-large array, as shown in Table 7.4 . The table num not in table n = 1 not found
can be further added to by data generated from equivalence partitioning. A number of n = 0 error
tests can be generated : the first would involve the data being positioned at location n+1, n > maxint error
just off the boundary of the array ; the next would involve it being positioned at location num at position n + 1 n < maxint not found
num at position n n < maxint found
n, just on the boundary ; and the final test would involve it being positioned at location num at position n - 1 n < maxint found
n - 1, just within the array.
Table 7 .5 shows the final result of this deletion of data . Such a table, supplemented
with an explanation of the reasons for each row, could be used for the first part of the 7.4 INTEGRATION TESTING
unit test documentation .
There are a number of distinct forms of integration strategy . Top-down integration, as
the name suggests, builds up a system front the top-level modules downwards . Bottom-up
integration starts with the lowest level modules and works upwards . Middle-out integra-
tion starts in the middle of the design of a system, and either integrates upwards or
downwards . Normally, systems developed as packages are integrated in this way : pack-
All the data in this ages are integrated top-down, and the system is then integrated from the interfaces to
subspace is processed
in the same wac the packages upwards .
There are a number of reasons why software projects carry out integration testing :

• When a module is integrated and tested and an error is discovered the process of
detecting the error is efficient as, normally, the error is in the module that has been
integrated . The tester need look no further . This is in contrast to system testing
whereby if an error is discovered, it could be contained in one of many thousands of
modules .
Data processed • Top-down integration in particular allows the customer to get an early view of a
b the function system . Normally the top-level modules in a system are those dealing with the human-
computer interface . Since these are integrated first, the customer or the user can get a
Figure 7 .1 A data space and subspaces. good view of the functionality of the software at a relatively early point in the project
and is able to point out potential problems to the developer .
• During integration some of the system tests which are to be carried out can be
executed . There is a maxim for software development that the earlier a test can be
114 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION TESTING 115

carried out the better because rectification is cheaper the earlier it occurs in the be integrated singly or with a small number of other modules . This strategy is suitable for a system
software project . with a large number of not very complex subroutines, but with a few large ones .
The second point-the order in which the subroutines are integrated-may be constrained by
programmer availability : for example, you may know that some subroutines will be produced by a
programmer who is joining the project late . However, if planning is carried out well there should
Self Assessment Question 7 .3 Can you think how integration testing be quite a degree of leeway for the order of integration .
can be used to give early warning of serious response time problems with a In considering the order of integration the designer or project manager should bear in mind
system? two criteria : the criticality of the functions that are contained in the system and also the importance
of the functions . For example, we normally integrate our systems in a top-down fashion . Also, the
systems we produce tend to have the human-computer interface embedded in the top-level modules .
Solution During integration the staff carrying out integration testing
If there is a feeling that the customer has not quite correctly communicated some of the functions,
would measure the response time as part of the testing process . If it looks then the top-level modules which implement the interface for these functions should be integrated
like the response time is going to be exceeded well before the integration first .
phase is complete, then this is an indication that serious problems have The second criterion is that of importance of functions . As part of the project planning proce-
occurred . dure the project manager will have carried out a risk analysis of the project (see Project Procedures,
section 13 .3) . If the project is a high-risk one, then a good strategy to adopt is to integrate the
system in such a way that the important functions of the system are integrated first, followed by
the next most important functions, and so on . In this way, if the project starts to look as if it is
A quality system should offer a number of facilities to staff who wish to carry out in- heading for trouble in terms of lateness the project manager can ask the customer whether he or
tegration testing : it should provide guidance to staff on the best strategy to adopt, and she wishes to take delivery of the software that has so far been integrated, after some acceptance
also give advice on how the process should be planned on an integration-by-integration testing has, of course, taken place .

basis . An example of this, in the form of an extract from an integration procedure, is As part of the analysis (see Analysis Procedures, section 11 .1), analysts for a high-risk project
will have ordered the system functions in decreasing order of importance ; these details can be found
found below : in section 7 .3 of the requirements specification .

5 .4 Integration procedure 5 .4 .2 Top-down integration

5 .4 .1 Integration There are a large number of reasons for adopting top-down integration .

1 . The system does not require any extra software to hold it together . The system is, in effect, its
unless there is a very good reason for not doing it, all our projects should integrate subroutines
own test-bed . In the past, with bottom-up integration, we have found that we have had to produce
which have been produced by programmers . If a project manager decides not to integrate he or she
scaffold software which has been as large as 40 per cent of the final system size .
should contact his or her technical manager and give good reason before deciding on this course of
2 . It enables us to demonstrate the human-computer interface to the customer at a relatively early
action.
Integration is planned after the overall system design has been frozen and comes under con- stage of the project . The vast majority of our systems have this interface embedded in top-level
figuration control . There are a number of decisions that a project manager or designer has to make modules.
regarding integration : 3 . You have a working version of the system from the moment that the first integration has been
correctly tested .
1 . What form of integration to adopt . There are essentially three strategies which we use . The pros
and cons of each strategy are detailed below in sections 5 .4 .2 to 5 .4 .4 . 4 . It provides an extra fillip to a project . If a project is particularly long you will find staff morale
2 . The order in which integration is to occur . Given that a project manager has chosen the strategy dropping somewhere during the middle of the project because of the perceived lack of progress in
there is still considerable leeway as to the order in which subroutines are inserted . terms of executable code . The news that the first version of the system-albeit the first few modules
which form the initial integration-is working, has a surprising effect on staff morale .
3 . The number of subroutines which are to be added at a time .
5 . It is able to detect top-level control and sequencing errors better than the other forms of inte-
A number of inputs have to be considered before addressing the last two points . First, it is neces- gration that our quality manual describes .
sary to determine the information needed which drives a consideration of the number of modules
to be integrated at a time . Normally, an integration strategy should not involve the integration of
more than 10 modules at a time . Integration over that limit often means that errors in some of the
subroutines which are integrated may interact with each other to produce subtle effects that reduce
the effectiveness of debugging .
In the extract above, the term scaffold software is used . This is software which holds a
There are two ways of carrying out integration, The first is by assuming that each integration system together while it is being integrated .
will always involve n modules . The value of n will be determined by the overall complexity of the As well as providing advice on how to plan integration, a quality assurance system
subroutines in the system . For a system with a large number of modules with complex processing should also specify detailed standards which describe the form of documentation that is
code and big interfaces n will be low . For a system consisting, in the main, of single function
to be produced during integration testing . The main document that should be produced
subroutines n will tend towards a high level .
The second option which can be adopted is to specify that the normal number of modules which
is the integration plan . The structure of this document is simple ; it usually consists of
will be integrated will usually be n, but that very complex modules, identified by the designer, will an introduction which describes the integration strategy that was adopted and includes
116 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION TESTING 117

a rationale for the strategy. The remainder of the document contains a page devoted to demonstrate that certain functions have been implemented ; however, there will also be
each integration . Each integration should be given a unique identifier, which is usually a number of non-functional tests which have to be carried out, for example tests which
the name of a system or subsystem, followed by the number of the integration . It should check on response time . There are a number of documents which are used to describe
describe the modules which make up the system to which further modules are to be and document the system and acceptance test process . They are :
added, and the names of the modules to be added . There should also be a description
in each case of the tests that are to be carried out-both tests on the interfaces between • The test design specification
the integrated modules and the partial system into which they are to be inserted, and • The test case specification
also early functional tests which act as a preliminary check on user functions . • The test procedure
An example of such documentation is shown below : • The test log
• The test problem report form
Integration : TransSys23 • The test summary .
System to be integrated into : MIS, TransUp, TransDown, TransValid : : NewUp, LongID ; Tax-
Upd, RecArchive
Modules to be added : TaxDed, OIdVals, NewVals
The test design specification describes the design of a test which confirms that a particular
Interface tests : quality attribute is present . The test case specification describes each of the individual
1 . This should check that when an update record is written to the receivables database, the database
tests which make up the test design . Test procedures are detailed step-by-step instructions
is correctly updated . on how a particular test is to be carried out . The test log is a description of what happened
during a particular set of tests described by a test design specification . The test problem
2 . This should check that when the tax deduction command has been initiated, the staff member
who is to have his or her tax record amended has his or her financial record correctly retrieved . report form describes what happened when a test failed .
3 . This should check that when information on a new member of staff is inserted into the employment
database the correct record details are entered into the database . 7 .5 .1 Test design specification
4 . This should check that when the new values table and the old values table are merged then the
resulting table contains the correctly merged data . This document is one which describes how a feature or related group of features is to
Functional tests : 12 .3, 12 .4, 12 .5
be tested . It does not specify any test cases, but describes in general terms the test
philosophy that is to be adopted . The following are normally found in a test design
specification .
There are a number of things to notice about this example document . First, the system A unique identifier. This identifies the document and should contain some reference
into which modules are to be integrated is specified in terms of subsystems . If a subsystem
to the quality attribute to be tested . For example, if the attribute was a function, then
is only partially present, the modules are then listed, preceded by a double colon which the number should include a reference to the function in the quality plan .
follows a module name and terminated by a semi-colon . Second, the most important point
Test item . This describes what is being tested . Normally, test design specifications
to notice is that the tests are not specified in very much detail . This assumes that the are only written for system and acceptance tests, although some companies who develop
staff who are going to carry out the tests are familiar with the system and its interface . large systems which require a complex integration phase develop these documents for
If not, then a lot more detail needs to be included : the sort of detail that you will find in integration tests . Therefore, this section would state that the test was either a functional
the documentation for system and acceptance testing, described in the next section. The system test, functional acceptance test, constraint system test, constraint acceptance test
final point to make is that the functional tests are uniquely identified by their number or integration test . If it is an integration test, this section may also specify the version
in the requirements specification . of the system which is being tested.
Test description . This will give an outline description of the test and the test methods
that are to be used . For example, if boundary analysis is to be applied it will specify the
7 .5 SYSTEM AND ACCEPTANCE TESTING range of data values which will be used and, hence, describe the boundaries . A typical
description is shown below :
The process of deriving system and acceptance tests is driven by the quality attributes
which are specified in the quality plan as verification requirements . There will be a wide This test will check that when the out-of-stock command is invoked, then a correct result will be
variety of controls which are used to ensure that these quality attributes are implemented displayed on the VDU which originates the command. A correct result will be a message which
correctly; however, the main control will arise from system testing and acceptance . You says whether a particular product typed in is out of stock . The tester will execute the command by
will remember that acceptance testing is the process of demonstrating to the customer providing correct and incorrect product identifiers . These identifiers will be ten-character strings,
that the software system developed contains quality attributes which are relevant to the and the tester will test correct product identifiers ranging from the alphabetically earliest to the
alphabetically latest, including the end-point identifiers . The tester will also choose incorrect iden-
customer . System testing is a preliminary set of tests which ensure that the acceptance
tifiers ranging from single-letter identifiers to ten-letter identifiers . Each test will use the same test
testing phase will be relatively trouble-free . System and acceptance tests will usually database.
118 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION TESTING 119

Tests applied. These are the test identifiers of the individual tests which are applied .
Self Assessment Question 7 .4 What is the difference between the test
These tests are described by unique identifiers, as described in the next section .
design specification and the test case specification?
Pass/fail criteria . This is a description of how a series of tests would be judged to
have passed or failed . For the test description above, it would tell the tester that the test
Solution The former is a high-level design document which describes how
would have passed if the tester had chosen a product which has no stock associated with
a set of features are to be tested . The latter contains much more detailed
it in the test database .
descriptions of tests ; for example, it will contain the software and hardware
requirements for a test .
7 .5 .2 Test case specification

This document describes the individual tests which make up a test design . Each test case
7 .5 .3 Test procedure
will be uniquely numbered, and will contain a number of different items of data .
Test input. This lists the data which is applied for the test . If the data is stored in a This document describes the step-by-step process whereby a test is executed and also
file, then all that would be required would be to name the file that stores the data . how a series of similar test cases are to be carried out . It will list all the steps needed to
Test output . This should describe the output expected from the test . set up and start a test, make any measurements such as response time, shut down a test,
Software and hardware requirements. This describes the hardware and software re- restart a test after a failure and stop a test . The test procedure will also contain details
quired for the test and includes not only any testing tools and scaffold software but also about where the item to be tested can be found ; what sort of item it is-a module, a
the version of the operating system that is to be used, and version numbers of any other system or a partially integrated version of a system ; and what sort of test is to be carried
supporting software . out .
Special instructions . If the test requires any special actions from the tester, for ex- Normally, test procedures are written in considerable detail when the staff who are
ample, to disconnect the computer from some peripherals, then this section would list to carry out a test are not familiar with the software being tested, for example an inde-
these actions . pendent test group . When the developers are the staff who carry out testing, the quality
Inter-case dependencies . If any other tests are required to precede a test then these system should provide guidance on how the requirements for test procedure documenta-
are described here ; for example, one test may check that the process of writing an item tion can be relaxed .
to a database has been correctly implemented and another might check that the item
added is still in the database . If a test depends on a previous test and that previous test
fails, then this part of the document will describe what the tester should do . 7 .5 .4 Test log
An example of this documentation is shown below for a system which carries out
This is a description of what happens when a particular test is carried out . It should
image processing :
describe what is being tested ; what test procedure is being used ; what happened during
the test ; what hardware or software was used ; a description of any events which were not
Test 3 .36
expected ; and if the test failed there should be references to the problem reports that
Test input . This system test checks that a picture is displayed at the right resolution . A test were generated when a test did not behave as expected .
database containing a large sample of pictures can be found in file TestPic .sys .

Test output . Each test which involves comparing a picture with a defined resolution will be passed
7.5 .5 Test problem report form
if the characters `yes' are displayed in the top left hand corner of the screen . Anything else will be
regarded as a test failure . A test problem report form is generated when a test procedure gives rise to an event
Hardware and software . No special software is required . The system can be found in the project which was not expected by the tester . This document will contain information about
library as Sys .obj . There will be a number of versions of this so, before extracting the system from the event ; what hardware and software environment was used ; whether any attempt to
the project library, check with the library administrator what the current project version is . The repeat the test took place ; when the error was detected ; and who carried out the test .
hardware used will be the standard 486 configuration asked for by the customer with a Tixuy 33
screen .

Special instructions None . 7 .5 .6 Test summary

This is a summary of a series of tests . It is usually issued after a subsystem, or even

Inter-case dependencies . None .
a whole system, has been tested . It should describe what has been tested ; evaluate the
success of the series of tests described ; give details of any outstanding errors that were
 120 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION TESTING 121

discovered by testing ; and provide summary information such as the length of time taken 7.8 FURTHER READING
for testing .
There are few good books published on testing . My two favourites are Myers (1979) and
Kaner (1989) . The former is excellent on topics such as test design but somewhat out-of-
Self Assessment Question 7 .5 During development how do you think date technically, while the latter is excellent on tools and test documentation . Together
test problem report forms are processed by a software project? they form an ideal pair of works which should adorn every software developer's bookshelf.
Heitzel (1985) is a good companion to both the above books .
Solution They are normally given to a member of staff who determines
whether the form describes a real error or whether the tester has misin-
terpreted the test documentation . If the report form describes a real error, PROBLEMS
then the source of the error is determined and the system is modified . For
example, if the error was a design error, then the system design, detailed 7 .1 Describe the relationship between the elements of test documentation described in Section 7 .5 of
design and code are changed . this chapter .
7 .2 One of the guiding principles of quality assurance is that validation activities should be as inde-
pendent as possible of the staff who have produced an item that is to be validated . How would you
implement this principle with regard to testing?
7 .3 Can you think of any circumstances when a unit testing standard and procedure can be relaxed?
7 .6 QUALITY MANUAL REQUIREMENTS
7 .4 Explain how traceability can be maintained between the functions in a requirements specification
and the system tests which check that these functions have been implemented correctly .
1. Unit folder standard 7.5 Can you think of the advantages and disadvantages of top-down and bottom-up integration testing?
2. Unit testing procedure
3. Integration test standard
4. Integration test procedure
5. Test design specification standard
6. Test case specification standard
7. Test log standard
8. Test problem report standard
9. System testing procedure
10 . Acceptance testing procedure .

Normally, all these documents would be subsumed within a testing standards and pro-
cedures document .

7 .7 SUMMARY

Testing is the most popular quality control used on software projects . There are four
main testing activities : system testing, acceptance testing, unit testing, and integration
testing . A quality manual should have standards and procedures which govern all these
activities . They should describe the tests which are to be carried out and the results of
those tests . It is also important that procedures governing testing describe what should
happen when a test fails .
 CONFIGURATION MANAGEMENT 123

The last 15 years have seen many projects get into trouble because they have been
unable to cope with change . A typical instance of this problem is where different staff have

8 received different versions of system documentation, with each version corresponding to

different states of the system during its lifetime . I recently visited one project where
the design did not match the specification, the program code did not match the design,
CONFIGURATION MANAGEMENT and the system test specification documented tests which checked out an out-of-date
system specification, all because changes were applied to project documentation in an
undisciplined way. It is the role of configuration management to control change in such
a way that major problems with modifications do not occur .
It is important to point out that although the previous paragraphs have stressed
change during a project as being a major reason for the existence of a configuration
management system, for some categories of software developer a configuration manage-
ment system is vital for other reasons . These are developers who produce multi-versions
of software which are tailored to a particular customer's needs ; for example, accounting
packages which can be modified according to the financial practices of the customer . For
developers of such systems adequate configuration management is almost an obsession .
Without such a discipline the developer is helpless when, for example, an error report is
transmitted by the customer . If the developer does not know what version of the soft-
ware was sent to the customer, then discovering the cause of the error becomes immensely
time-consuming .
AIMS
A good configuration management system should be able to provide data which
answers questions such as :
• To describe the role of configuration management within a software project .
• To describe the various components that make up configuration management .
• How many proposed changes to the system remain to be applied?
• To outline the steps that take place when a change is applied to a document or
program code . • What changes were applied to version n of the system specification in order to derive
version n + 1 of that document?
• What versions of a system are in existence?
• How many faults have we cleared that have been notified against the current version
8 .1 INTRODUCTION
of the system?
• Have the changes which have been applied to version n of the program code been
There is a myth that change to a software product and its associated documentation
only occurs during maintenance : that after a system is handed over to a customer, then validated? That is, that the changes applied have been checked, and that a check
has been carried out which ensures that other non-sanctioned changes have not been
new requirements which arise during the lifetime of a software product will require that
product to be modified . While change during maintenance is a major resource consumer, applied .
• What versions of modules make up version n of our current system?
change during development is also a widespread phenomenon . During development there
are two major reasons for change . The first has already been alluded to in previous
chapters : the fact that customer requirements will change during a project-particularly Self Assessment Question 8 .1 Would a configuration management sys-
a long project . For example, a project to deliver a financial information system may need tem normally be able to answer a question about who initially produced a
to be extensively redesigned when new financial laws or instruments are unexpectedly document or some code?
brought in by a government which is responding to a crisis ; or a defence system may be
changed in mid-project when the customer for a military air-based command and control Solution No, the name of staff who carried out the initial development
system discovers that a potentially unfriendly country has recently received advanced of a requirements specification, design or program code would normally be
airborne warning technology . placed somewhere in the document that they developed . For example, the
A second reason for change is the fact that developing a software system is a very programmer who developed a module would have his or her name as a com-
complicated business and is error-prone ; often the errors that have been committed in ment in the module header .
one phase are only discovered during a later stage . This not only requires reworking of the
system code, but also the modification of earlier project documents such as the system
design specification .

122
 124 SOFTWARE QUALITY ASSURANCE - A STUDENT INTRODUCTION CONFIGURATION MANAGEMENT 125

Configuration management is a discipline which :

• Provides information about the various configurations or versions of a system at points

in time .
• Keeps information about changes to all system components-including both software
Change
and documentation . request Change request note
• Provides methods and tools for controlling changes . filled in
• Provides facilities for the validation of changes throughout the life of a system .

The remainder of this chapter describes the components of such a system .

Change considered by
the Change Control
Board
8 .2 THE PROCESS OF CONFIGURATION MANAGEMENT Sanctioned

T
Fig . 8 .1 shows the main configuration management activities that occur within a software Rejected Change authorization
project . First, a change request is generated . This may originate from two sources : from note filled in
the customer who wants a new requirement implemented or has discovered an error, or
from the development team in response to the discovery of major problems with the Batched
system during validation . The change is communicated to the configuration management System
system via a change request note ; details of what should appear in this form can be Change documentation
found later in this chapter . The change is then submitted to a part of the software implemented modified
project known as the Change Control Board . This sounds as if it contains a large number
of staff ; however, in practice the duties of a Change Control Board are often discharged
by one person . This will often be the project manager or another senior member of the Current batched Change
project ; sometimes it is a function carried out by quality assurance staff . changes validated
An excerpt from a quality manual detailing the role of a change control board is
shown below :

2 .3 The Change Control Board Validation and Current configuration

test records records updated
Each project that we carry out will have a configuration management plan . Details of what should produced
be in this plan can be found in section 2 .5 . An important part of the configuration management
process is the Change Control Board . The staff who carry out this function will vary from project
to project . On small projects this function will be taken by the project manager. On large projects
we often have this function exercised by the project manager and the member of the quality de-
partment assigned to the project . Figure 8 .1 The process of configuration management .

At periods during the project the Change Control Board will consider requests for changes to reason why severity is specified is so that the Change Control Board can decide what
documentation and code which has been frozen . The frequency of this process will depend on the is to happen to a change . The Board looks at a number of factors in deciding what is
project and also at what point the project is at . At the beginning of an innovative project, where
to happen to a change, the main ones being the impact of the change on the software
a requirements specification has just been frozen, you may find it useful to have relatively frequent
consideration of changes ; during acceptance testing, when the system should be as near to what the project and also whether the change is necessary in order to satisfy the requirements
customer requires, the frequency will normally be much less . specification generated at the beginning of the project . There are three decisions that
can be made:
If one person is carrying out the functions of the Change Control Board there is no need for a formal
meeting . However, no matter what form the Change Control Board takes there will always be a
need to fill in and file the decision documentation described in section 2 .6 . • Allowed : the change is to be applied as soon as possible .
• Disallowed : the change will not take place .
Normally, some form of analysis of the change has been carried out by project staff and • Batched : the change will take place sometime in the future .
information about the severity of the change is provided, as well as a description . The
 126 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION CONFIGURATION MANAGEMENT 127

Changes which are allowed are normally those changes which the project regards as validation could take a number of forms, depending on the severity of the change . A small
important, or which are relatively trivial . For example, a change in response to a serious change-for example, a modification to the program code of a single module-would only
error which would result in the non-implementation of an important function would come require the generation of some module test data and the rerunning of any affected system
under this category, as would a minor change to the display of data requested by the tests ; while a large requirements change would require a number of reviews to take place,
customer . together with the generation of module, integration, system, and acceptance test data .
Changes which are disallowed are those which the Change Control Board agree are An important point to be made about configuration management is that it requires
not necessary, have a serious impact on the project, or fall outside the remit of the project . the definition of what is known as a baseline. A baseline represents a well defined point
For example, a change request by a customer which would result in the addition of a in the development of a system . Normally, the point is associated with the delivery or
function not initially documented in the requirements specification would come into this freezing of an important item of documentation or a version of the program code of a
category . If a customer request has been disallowed, then the configuration management system . For example, there will always be a baseline associated with the delivery of the
system should allow the option of the customer re-requesting the change . In this case requirements specification . Up to the point that a baseline is declared, the documents
the change is costed, the impact on the project in terms of extended delivery time is associated with that baseline can be changed at will . However, after that point, any
specified, and the customer is asked whether he or she wishes to re-present the change, change to these documents have to go through the configuration management process
with of course the eventual outcome being delay of the project and an increase in the detailed above .
cost of the system . If the customer agrees, then the change is normally re-presented and
allowed .
Self Assessment Question 8 .3 Would the system design be a document
Changes which are delayed are those changes which the software developer thinks are
that is baselined?
a good idea, but which, for one reason or another, cannot be immediately implemented .
An example is a change to a financial package requested by a customer who claims that
Solution Yes, all important documents associated with a system are base-
the package does not do what the manual says it should . The developer may discover
lined and the design is a crucial one .
that the customer has misread the manual, but that the proposed change would actually
make sense in marketing the package : that it provides some extra functionality which
may sell more copies of the package . In this case, the proposed change is turned down as
not immediately implementable, but it is placed on a queue of changes to be implemented
during the development of the next version of the package . 8 .3 CONFIGURATION MANAGEMENT ACTIVITIES

Once a positive change decision has been arrived at, two events occur . First, the
There are a number of activities which make up the process of configuration management .
change is added to the configuration management system documentation . Second, the
They will be explored here within the context described in the previous section . The
change is communicated to the staff who have been nominated as responsible for its
implementation . activities of this section make up the configuration management system to be used by a
project . This will normally be documented in the quality plan .

Self Assessment Question 8 .2 Assume that a module has been changed

8 .3 .1 Configuration identification
in response to an error being discovered and this change occurred during
system testing. What validation should a configuration management sys- The process of configuration identification involves the specification of those components
tem insist on? in the software project which will be formally placed under configuration control . These
components are known as configuration items . Normally, configuration items for early
Solution First, the module should be checked for functional correctness . documents in the software project will be associated with subsystems ; for example, sub-
system requirements specifications and designs in the case of large projects . For smaller
Second, all the system tests which the module participated in should be
rerun . Third, staff should check that no extra changes have been made to systems, the full requirements specification and system design will be specified as configu-
the module . ration items . The identification of configuration items from program code will depend
on the nature of the application, but often each module will be designated as a configu-
ration item and will come under configuration control when it is finally tested by the

The change is carried out, and all documentation associated with the change is modified programmer who produced it .
and new versions of the documentation produced . For example, if a requirements change Configuration items are often defined in terms of other configuration items . The
has been authorized, then a wide variety of documents will need to be changed, including code of a subsystem will be a configuration item, as could be the individual modules
the requirements specification, the system design, detailed design, program code, and all which make up the subsystem . Hence, a change proposed to a particular configuration
the associated test documentation . Once the change has taken place, it is validated . This item will often imply that its component configuration items will be changed ; for exam-
 128 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION CONFIGURATION MANAGEMENT 129

ple, a functional change to a subsystem will be recorded as a requirements change to 8 .3 .4 Configuration auditing
that subsystem, and also a change to all those modules in the subsystem which require
modification . This process is often carried out by quality assurance staff . It involves checking that
standards and procedures associated with configuration management have not been vio-
A configuration item will be identified by a version number . All changes to a con-
figuration item will normally be applied to the latest version . In larger projects, as well lated . This means that staff responsible for this function should be able to trace a path
as version numbers some items will have variant numbers . These numbers represent a through from a successful change proposal to the final tests and checks on all the con-
figuration items which have been affected by that change : that there is an audit trail
collection of configuration items which are gathered together by virtue of the fact that
which involves change notes ; minutes of the deliberations of the Change Control Board ;
they implement different functions or they may be tailored to the customer's local cir-
cumstances . For example, a subsystem may exist in a number of variants, each variant a directive informing staff to carry out the change ; evidence that the version numbers
of affected configuration items have been updated ; and evidence that reviews or tests
corresponding to a particular operating system on which the system containing the sub-
which check that the change has been implemented correctly without affecting the other
system is implemented .
parts of the system have been carried out . The last item is vitally important as it is often
Normally, a numbering standard is adopted : each configuration item is given a version
number and a variant number . When a change is carried out to an item, that change is highly likely that a change to a system, for example to modify a function, will affect some
applied to its variants . A configuration is a collection of configuration items . functions of the system which should remain unaffected .

Many of the configuration items will be identified early in the project . These will,
almost invariably, be standard documentation items, such as the requirements specifi-
cation, system design, system test specification, etc . However, when system design has 8 .4 DOCUMENTATION FOR CONFIGURATION

been completed code-related configuration items will be added to the list . MANAGEMENT

The documentation associated with configuration management is as follows :

8 .3 .2 Configuration control

This is the process whereby a change is communicated to a project and then carried
• Change request note
out . A quality manual should provide direction to those staff setting up a configuration • Control board minute

management system for a project, as follows : • Change authorization note

• Change validation report
• Procedures which detail the actions required and the route taken when changes are
• Configuration item change trail

communicated to those members of staff responsible for evaluating and sanctioning

• Configuration bulletin .

changes .
• The remainder of this section looks in a little more detail at the contents of this docu-
Standards for the documentation used to communicate a proposed change, and the
mentation .
result of the deliberations of the Change Control Board .
• Guidelines to be used when evaluating whether or not a change is to be allowed .
• Procedures to be used for informing staff about changes and the effect on the configu- 8 .4 .1 Change request note
ration items they are dealing with .
The change request note should contain the following information : a unique change num-
ber, the date of request, page number, name of project, and a brief identifier for the
8 .3 .3 Status accounting change .
The form should also contain the reason for the change, a description of the change,
Status accounting is the recording and storage of data required for configuration man-
agement. This involves keeping data on which parts of a software system have been a prediction of the amount of effort required for the change, and the configuration items

identified as configuration items ; details of the configuration items which a particular which will be affected by the change . The form should also have some space for comments
part may consist of ; the existing versions of configuration items and their variants ; the which expand on the reason for change .

configurations which are currently being maintained ; and the list of changes which have The final part of the form should be used to record the Change Control Board's

been applied to a version in order to create another version . decision : whether the change is to be allowed, forbidden or batched . It should also contain
information such as the date of the Change Control Board decision, and the name of the
Status accounting also involves the recording and storage of data about changes :
person who was responsible for evaluating the change . This final part of the change
those which have been proposed but not yet considered ; those under consideration, those
which have been allowed or refused ; and those which have been delayed for incorporation request note would also include space for the Change Control Board to explain why they
took a particular decision . This would normally only be filled in when a decision is made
into subsequent versions of a configuration item .
to forbid the change or to batch it in the next version of the system .
 130 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION CONFIGURATION MANAGEMENT 131

tests and reviews which have been carried out to this end, and will usually contain a list
Self Assessment Question 8 .4 Can you think of any reasons why a
of each test and review identifiers . These identifiers can be used to look up the detailed
change request would be turned down?
documentation dealing with these activities in the project library .

Solution There are two main reasons . First, the request is for a require-
ments change which is not in the original requirements specification and 8 .4 .5 Configuration item change trail
the customer will not pay for the change . Second, the change is in response
This is a set of documents, one per configuration item, which detail the evolution of
to an error which is not really an error . For example it occurred because
the item from the moment it was baselined . It consists of a series of change identifiers
the customer misread the user manual or a member of the developer's staff
followed by a brief synopsis of the reason for a change . It will also detail the change in
misread some specification .
numbering which occurred from version to version as a result of the changes .

8 .4 .6 Configuration bulletin

This document contains the current configuration of the system, expressed in terms of
8 .4 .2 Control Board minute configuration identifiers, current version numbers, and variant numbers . It is used by
staff to check that they are currently processing the right version of a configuration item .
This is essentially a linking document which will contain details of the changes that
were considered at one sitting of the Change Control Board . When the Change Control
Board consists of a number of staff, it makes sense for the Board to meet regularly to 8 .5 QUALITY MANUAL REQUIREMENTS
consider non-urgent changes . The Control Board minute details, in summary form, the
change request notes that were processed by the board at one meeting and the outcome 1 . Configuration management system set-up guidelines
of the requests . This document will mainly consist of change request note identifiers and
2 . Change control procedure
brief descriptions of the outcome of the Change Control Board's deliberations . When 3 . Change request note standard
the change control function is delegated to one person, a software developer will omit 4 . Control board minute standard
completing Control Board minutes and will rely on the configuration bulletin to provide 5 . Change authorization standard
information to staff . This item is discussed below .
6 . Change notification note standard
7 . Change validation procedure
8 .4 .3 Change authorization note 8 . Change validation report standard
9 . Configuration item change trail standard .
This document authorizes a particular change or set of changes which have arisen from
a change request note and a decision by the Change Control Board to allow the changes . Normally, all these documents will be subsumed under a configuration management and
It will consist of information such as a reference to the change request note, date of control standard and procedure .
issue, page number, name of project, and the name of the change . It will also detail
the changes that need to be carried out . Often these will be the same as those on the
corresponding change request note ; however, where a company uses change notes which
8 .6 SUMMARY
specify a number of changes, the change authorization note will only contain the subset
of the changes which were allowed .
Change occurs throughout a software project as well as during operation . A configuration
The change authorization note will also contain a list of configuration items which management system is a set of standards, procedures and guidelines which ensure that
are affected by the changes and how their version numbering is to be changed . the process of change is as smooth as it should be . Collectively, a configuration manage-
ment system should govern the processes of notifying change, appraising, specifying, and
8 .4 .4 Change validation report validating changes and the documentation of the state of a system in terms of baselines
and changes . A software developer who does not have detailed standards and procedures
This is a report which is issued after a change or set of changes associated with a change which describe configuration activities is destined to waste a large amount of resources
request note has been implemented . It details the testing and validation that has oc-
and commit a large number of errors .
curred . This testing and validation will be in two forms : checks that the change has been
applied correctly and checks which ensure that remaining functions and other quality
factors in the system have not been affected by the change . The report will detail the
 I

132 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

8 .7 FURTHER READING

The problem which faces the author who wishes to write about configuration management
is that, while it is an important topic, there is not quite enough information on it to fill
a book . Consequently, there is a distinct lack of good texts . The best book that I can
recommend is Babich (1986) and a good, short article on the subject is Bersoff (1984) .

PROBLEMS

8 .1 What sort of events do you think would give rise to a change request being submitted to the Change
Control Board?
8 .2 Write down a standard for a change request note .
8 .3 Write down all the documents which you think would be baselined on a large software project .
8 .4 Give an example of a change to a baselined document which would affect a large number of other
baselined documents .
 y

QA AND THE NEW TECHNOLOGIES

AIMS

• To outline the effect of new technology on a quality system .

• To describe in some detail the effect that four specific new technologies have on quality
in the software project .

9 .1 INTRODUCTION

The aim of this chapter is to examine four of the more popular software advances which
are transforming the way we develop software, and their impact on the QA practices in a
company . In each case the use of the technology described in this chapter requires extra
components to the quality system . One of the most serious problems I find with software
developers is that although many of them are keen to advance in terms of adopting new
technology, little effort is put into adapting the quality system to cope with this new
technology. In some cases some major modifications are needed . For example, the use
of a new programming language might require large changes to programming standards
and the standards and procedures used for module testing.

9 .2 PROTOTYPING

9 .2 .1 What is prototyping?
Prototyping is the process of developing a working model of a system early on in a project .
Such a working model can be shown to the customer, who suggests improvements . These
improvements are incorporated in the prototype and it is shown again to the customer,
and so on . Prototyping is a valuable technique, both for the customer and the developer .
All too often I have met customers, faced with an inadequate implementation, who have
said during acceptance testing : `If only you had shown me the system earlier in the project
I could have told you exactly what I wanted .' Prototyping has a number of advantages :

133
 1 34 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION QA AND THE NEW TECHNOLOGIES 135

• It enables the customer to preview the system well before major development work • Implement only those parts of a system whose requirements are felt to be ill-expressed
starts . or fuzzy . This might mean the developer deciding on an incremental development
• It enables the developers to know exactly what they need to deliver . Previously, strategy, with subsystems that have a high amount of risk attached to them being
developers have attempted to deliver the system that reflects the requirements speci- implemented first .
fication . Unfortunately, since this specification is written in natural language there is • Use a table-driven processor approach to development . This relies on a set of tools
always considerable scope for misunderstanding . By asking the customer to sign off that are table-driven and have a clean interface between them . The ideal candidate
the prototype, the developers are able to have an exact reference point from which for this approach is UNIX, which contains table-driven processors such as YACC that
to judge their implementation . enable software to be produced very quickly .
• It enables early customer training to take place . Customer training is often rushed • Employ a programming language that enables the developer to implement a large
and inadequate, and a prototype can be used to integrate training properly in the amount of functionality in a physically small amount of program code . Probably
development project . It also has the allied advantage of enabling the developer to the best example of this is the fourth-generation programming language . Such lan-
meet the staff who are actually going to use the prototype . A common feature of guages are usually interfaced to relational database systems, and allow very complex
many software projects is that the customer representative has rarely encountered commercial data processing to be implemented in very short sections of program
or been part of the application that is to be developed and is, consequently, not the code . However, fourth-generation languages are not the only medium for this form
ideal person to talk to about system requirements . of prototyping-programming languages such as PROLOG, SETL, LISP, and APL
• It can act as a test oracle during the later stages of system development . A test oracle have all been successfully employed to produce prototypes .
is a version of a system that will always deliver the right results when executed . A • Use an application-oriented programming language . Such languages are aimed at one
prototype can be used as an oracle by feeding its output, together with the output particular application area, and contain data types which are used continually in that
of the final developed system, through a file comparator . This means that the tester area . For example, an application-oriented language for a warehousing application
has very little work to do when examining test output during system testing and would use data types for warehouses, stock bins and picking lists .
acceptance testing . All that needs to be done is to examine the output of the file
comparator in order to see whether the developed system has given a different result These, then, are some of the ways in which prototypes can be generated . Object-oriented
to the oracle . programming languages are also an excellent medium for prototyping . Such languages
contain a facility for defining the entities that were identified in the previous chapter
using a facility known as a class . In terms of prototyping, an important facility of a class
Self Assessment Question 9 .1 Why do you think prototyping is-
is inheritance. In order to explain what inheritance is, it is worth quoting an example :
effec-tive?
that of an invoice in a commercial data processing application .
Such an invoice will always contain the same details, irrespective of the application .
Solution The main reason is that it provides a validation of the require- It will contain the following information : the name of the supplier who has provided
ments specification early in a project . Error rectification during prototyping
the services or goods that the invoice describes ; the supplier's address ; the name of the
costs much less than if it occurred during an activity such as system testing .
customer ; the customer's address ; a list of items supplied ; their price ; and the total
price . In a purchasing application an invoice would, almost certainly, be the first entity
discovered by an analyst .
9 .2 .2 Techniques to achieve prototyping In the first application in which the software developer uses an invoice, it would
be defined using a class description of the components of the invoice, together with the
A number of techniques are available to the developer who wishes to produce an early
functions which manipulate the invoice, e .g . creating an invoice . The application would
version of a system . They range from rather simple techniques, such as showing the
then be implemented, and the class stored in a company-wide library .
customer some sample screens, to using sophisticated programming languages which are The next time an application is developed that requires an invoice, the designer
capable of packing a lot of functionality punch into a small amount of program code . responsible would look in the library for any classes that described an implementation
The main prototyping techniques are as follows :
of an invoice . Let us assume that the system requires a slightly different form of invoice
from the one contained in the library ; for example, it might be for an overseas customer,
• Relax the quality assurance standards of the project . Tolerate errors in the prototype
where the tax deduction part of the invoice would be different .
in an attempt to get out a rough working version of the system quickly . This could
The designer of the new system would define a new type of invoice which contained
mean a whole variety of tactics could be applied : skimping on the requirements speci-
this element but which was mostly made up of the invoice class stored in the project
fication, omitting the system design stage, carrying out perfunctory system tests, and library. This is inheritance-the new invoice class inherits many of the properties and
omitting many of the reviews that would normally be scheduled . operations of the stored invoice in the project library . This reduces the work required
from the designer : all that has to be done is to write down the new components of the
 1 36 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION QA AND THE NEW TECHNOLOGIES 137

invoice, any new operations associated with this component, and then bring in the invoice
class from the project library . Self Assessment Question 9 .2 Isn't prototyping the same as hacking
If a developer has built up an extensive library of classes, then prototyping can be and should be discouraged by a quality system?
effectively carried out . All it involves is for the analyst or designer responsible to call down
pre-written classes from a project library, and join them together with some processing Solution The technical process of prototyping looks very much like hack-
code . The following section will examine object-oriented languages in more detail and ing . However, prototyping is normally controlled quite well by quality sys-
look at their QA implications . tems by the provision of standards and procedures ; for example, a standard
for reporting back on the progress of the evaluation of a prototype with a
customer .
9 .2 .3 Models of prototyping

There are a number of ways of implementing prototyping . The choice is really dependent
on the type of software that is being developed, and project-specific factors such as the 9 .2 .4 QA and prototyping
duration and size of the project . There are three popular prototyping models : throw-away
prototyping, evolutionary prototyping, and incremental prototyping. One of the myths about prototyping is that since it seems to be quite a flexible and free-
Throw-away prototyping corresponds to the popular idea of prototyping ranging process very little quality assurance is needed . The reverse is true, in fact . I have
: the de-
veloper produces a prototype and then initiates the iterative process of showing the come across a number of prototyping projects which have failed ; the vast majority of these
prototype to the customer and modifying it in response to the customer's wishes . When failures have arisen either because of a management failure or because the QA system did
the prototyping phase has finished, the prototype is, in effect, thrown away : it is placed not contain facilities for monitoring, controlling, planning and costing the prototyping
in an archive and conventional software development is started, based on a requirements process . The discussion in this section will centre on throw-away prototyping as that is the
specification that has been written by examining the detailed functionality of the proto- form of prototype development often encountered in software projects . However, many
type . of the points made are equally valid for both evolutionary and incremental prototyping .
Throw-away prototyping is effective for short projects of a few months' duration . The first problem that is often found in quality systems is a lack of guidance about when
Unfortunately, for large projects it tends to be less than ideal . The reason, of course, to prototype and when not to prototype . There is an allied problem in that staff also
is change . When conventional software development begins, the developer is almost in- carry out prototyping with only a hazy view of what the prototyping process is meant
variably bombarded with changes of requirements from the customer . In the end, the to achieve.
conventional software part of a prototyping project can suffer from the same problems Thus, there is a need for a guideline document which describes the various reasons
as a project which did not use prototyping . for prototyping and outlines the reasons why a decision to prototype may be taken . Such
Evolutionary prototyping is the complete antithesis to throw-away prototyping . Here, a document would ask questions about such project-specific factors as the volatility of
the developer aims to keep the prototype alive . The first stage of the evolutionary proto- customer requirements, changes in the customer's circumstances during the project, and
typing project involves the development of a prototype using the same activities and whether there is a perception of an inability of the customer to convey requirements in
techniques that would be used for throw-away prototyping . However, after the customer written documents . Many of the questions in this guideline can be found in procedures
has decided that everything is satisfactory, the developer then bases the remainder of the which govern risk analysis . These are described in Chapter 3 .
development work on the prototype that has been generated . Another problem involves ignorance of what the prototyping process is meant to
In the case of a prototype that has been generated using a fourth-generation pro- achieve . There are a number of useful ways in which prototyping can be used : for require-

gramming language, this involves a process of optimization, whereby the initial prototype ments elicitation, design evaluation, exploring the human-computer interface, and train-
is modified to make it run faster and use less memory and file space . The important point ing ; but, more often than not, we tend to associate prototyping solely with requirements-
about this form of evolutionary prototyping-and evolutionary prototyping in general-is based activities .
that throughout the project a working prototype that matches current customer require- Because of this fuzziness the project plan should have a section which describes why
ments is always available . prototyping is to be used and what the deficiencies of the prototype will be . For example,
The final form of prototyping is known as incremental prototyping . This can be used in aiming to deliver a prototype rapidly, a developer may decide to tolerate quite a high
in a project where the requirements can be neatly partitioned into functionally sepa- number of errors which, although they would be serious if present in the final system, are
rate areas . For example, developing a system for a chemical plant involves the following no more than a slight irritant for staff carrying out prototyping . The section describing
functional areas : monitoring, control, the human-computer interface, and management prototype deficiencies is an excellent remedy for one of the problems which afflict projects
information . If one of these functional areas is fuzzy, or the customer has difficulty ex- that use prototyping : that of the customer being so taken with the prototype that he or

pressing requirements, then a process of incremental development can be adopted . Here she asks for early delivery of a system which is a good prototype but a poor production
the system is developed as a series of small subsystems, each of which implements a system .
subset of the functions . The early subsystems can then be used as prototypes A major problem that has afflicted many prototyping-based projects is that of proto-
.
138 SOFTWARE QUALITY ASSURANCE--A STUDENT INTRODUCTION QA AND THE NEW TECHNOLOGIES 139

type evaluation . Typically, what might happen is that an analyst and customer may be 9 .3 OBJECT-ORIENTED PROGRAMMING LANGUAGES
so taken with prototyping that more and more time is spent on this process, with more
and more functions being continually added, to the point where the prototype becomes 9 .3 .1 What is an object-oriented programming language?
impossible to implement within the hardware and response time constraints originally
Object-oriented programming languages are languages in which objects can be imple-
detailed in a requirements specification . There is a need for a quality system to provide
mented, together with facilities for ensuring that the implementation of objects can be
standards and procedures for prototype evaluation which result in regular reports being
hidden from the programmer, and facilities that enable a high degree of reusability to
sent to the project manager that describe the functional growth of the prototype .
take place . This statement is rather platitudinous, and it is the aim of this section to
explore this matter in a little more detail .
Objects are found in all applications-a plane in an air-traffic control system is an
Self Assessment Question 9 .3 What do you think should be in these
reports? object ; so is an invoice in a purchasing order system ; and a student in an educational en-
rolment system is another example . Object-oriented programming languages implement
objects as data structures and modules which access the data structures . They enable a
Solution If the reports are used to just check the functional growth of
programmer to produce an entity known as a class, which describes the data structures
a project, then they would contain a list of new functions added or deleted
together with some estimate of the size of the functions . and the modules .
Figure 9 .1 shows the structure of a typical class, in this instance a class that describes
a queue . Associated with this class would be a description of the stored data that makes
Another problem found on prototyping projects is that analysts and programmers rush up the object, together with a description of the operations which allow access to the
data-the operations being implemented as modules-and a mechanism for specifying
off and develop a prototype without thinking about whether they have perceived the
conceptual architecture of the intended system correctly . What often happens is that which facilities of the class can be accessed by the programmer who employs the objects

a considerable amount of resources are then devoted to the production of an initial described by the class in an application .
prototype which is highly defective . A good quality system should insist that before This, then, is the base idea on which object-oriented programming languages are

starting the prototyping process a software project carries out a very short paper and based . The facilities associated with objects in a language, the number of these facilities,
and the quality of their implementation are a good guide to the object-orientedness of a
pencil specification exercise-taking no more than a few days-in order to check that
the conceptual architecture of a proposed system is firmly established . Only then should programming language . The remainder of this section describes these facilities .
staff begin prototyping . The first facility of an object-oriented programming language is that of providing a

Lack of proper documentation is another difficnity-particularly in the case of evo- means whereby access to a data structure is restricted . For example, the designer of the
lutionary prototyping . Once a prototype has been agreed with the customer, the proto- queue object described above would, almost invariably, want to prevent a programmer

typing standards and procedures should insist that the design and the requirements from manipulating the base data structure that implements the queue . If such manipu-
specification be documented . Such documentation is vital for staff who have to maintain lation was allowed, it would lead to the development of a system where references to the

the system . underlying data structures are found scattered throughout the system . Such a system
is a software maintenance headache, since a change to the underlying data requires,
A major problem, which has still not been solved satisfactorily, is that of costing
and resourcing a prototyping-based project . For evolutionary prototyping it is still essen- as a minimum, that the programmer implementing the change examines every module

tially a research problem . However, some guidelines can be given for costing throw-away that references the data structure . In practice, it also leads to a vast amount of change
prototyping-based projects . For example, prototyping guidelines should give directions necessitated by modifying each module that refers to the internal details of the data

to the developer to base the resources required for prototyping on a consideration of the structure .
risk factors on a project . Such guidelines should ask the project manager to examine pre- The facility that allows access to a system to be severely restricted is known as

vious resource figures from earlier prototyping-based projects, together with the results information hiding : the information about a data structure is hidden beneath a series of
of their risk questionnaires . From these, a project manager should get a good idea of the operations which only allow access to the data structure via parameters . For example,
the operation of adding an item to a queue would be implemented as a module with two
amount of resources required for the prototyping part of the new project assuming, of
parameters : the first would be the item to be added, and the second would be the queue .
course, that a risk questionnaire has been produced for that project .
Another feature of an object-oriented programming language is polymorphism or, as
it is sometimes called, genericity . This allows the software developer to define the data
and operations that implement an object via the class mechanism, and instantiate the
class for a variety of components in the data part of the class . For example, an object-
oriented programming language would enable a developer to implement a queue class
for, say, integers, and then use that class for a queue of messages in a communications
140 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION QA AND THE NEW TECHNOLOGIES 141

guages which could be described as having object-oriented properties, ranging from those
such as Ada, with a small number of facilities, up to the programming language Smalltalk
which has the object as its base facility, and where every other facility of the language
is integrated with this idea of an object . The language C++ lies in the top quarter of
CLASS the object-oriented language spectrum : it is certainly nowhere near as sophisticated as
Smalltalk, but it still contains a considerable amount of object-oriented facilities, al-
- Specification of the implementation beit that some of them have been implemented in what might seem a slightly eccentric
of a queue
manner .
My suspicion is that of all the programming languages that have object-oriented
4- Specification of an operation to facilities, it is C++ that will be the most popular and long-lived . This is not because it
add an item to a queue
is an elegant language, but because it is built upon an existing popular programming
language-C . The history of software engineering still only spans three and a half decades .
4 Specification of an operation to However, this is long enough to show us that radical change does not occur in software
remove an item from a queue
development, but that change, dampened by the fact that we have a massive software
maintenance mountain, is an evolutionary process which takes place in increments-
Specification of an operation increments which, more often than not, are based on an existing technology, programming
to count the number of items language, or development method .
in a queue

9 .3 .2 Object-oriented technology and software quality assurance

Figure 9 .1 A class that specifies a queue.
There are a number of ways in which object-oriented technology will affect the quality
system of a company . The first is the model of software development adopted . The con-
system, a queue of spool files in an operating system, or a queue of back orders in a ventional life-cycle model involves a requirements specification being produced, followed
purchasing system . Moreover, this instantiation of a new instance of a class is usually a
by a system design and then program code . The requirements specification will mainly
matter of writing a few extra lines of program code . Polymorphism allows a developer
consist of functions ; hence, from an early point in a conventional project, functionality
to build up a library of reusable objects, and contributes greatly towards the ability to
will drive the development process .
develop reusable software .
Since object-oriented technology concentrates on data, a new life-cycle model needs
A further feature of an object-oriented programming language is inheritance (de- to be adopted which identifies objects and operations during the requirements analysis
scribed above) . For example, a commercial data processing system developer may have
phase . Although a functional specification is eventually produced, the standards and pro-
already defined a class for purchase order objects which would describe an implemen- cedures for requirements analysis should direct the developer to produce an object model
tation in terms of a data structure that would hold information about the name of a
at an early stage . This means that a standard for a notation for expressing such models
customer making an order ; the customer's address ; the products being ordered ; and a
should be included in the quality system, together with a procedure which describes some
list of operations which communicate with data described by that class .
means of validating the model .
If another application came along, say, for a slightly different system which handles
orders for dangerous chemicals, where the purchase order contains information about
handling instructions for the items to be delivered, then the developer is able, using an Self Assessment Question 9 .4 Is this a major change over the way that
object-oriented programming language, to define a new class which mainly consists of conventional projects are organised?
items from the old purchase order class, together with new extensions and operations
which handle the new special case . In general, a good object-oriented programming lan- Solution It really depends on what you regard as a conventional project .
guage should allow a class to inherit properties from more than one class, and inherit For a conventional data processing project the answer is no, as these
properties more than once from the same class . projects take the design of the data very seriously and it is almost one
A final facility that should be provided by a class is dynamic memory management, of the first tasks carried out . For a conventional real-time project, say, then
where the underlying compiler software allows an object to be created when it is required, there is a difference because such projects consider the functions of a sys-
and deallocated when it is no longer needed . tem as the prime driver of development .
The degree to which a programming language can be called object-oriented is de-
pendent on how many of the above facilities have been implemented, and the degree of
simplicity with which they have been implemented . There is a broad spectrum of Ian- Another impact of object-oriented technology on the software project arises from the
 142 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION QA AND THE NEW TECHNOLOGIES 143

fact that classes are reusable . Software developers who employ object-oriented technology 9 .4 FORMAL METHODS OF SOFTWARE DEVELOPMENT
will increasingly find that they will have built up bigger and bigger class libraries, the
contents of which can be reused in projects . There are a number of implications here for 9 .4 .1 What is a formal method of software development?
the software developer .
A formal method of software development makes use of mathematics for the specifica-
The first is that a tightening up of the quality controls associated with the validation
tion and design of . a system, together with mathematical proof as a validation method . A
of individual classes is usually needed . If a project manager is to incorporate a class into
developer who uses a formal method first specifies the functionality of his or her system
his or her software, then he or she needs to be reassured that the degree of validation
using discrete mathematics, and then reasons and proves properties of the system, for ex-
that the class received is at least as good as that received by the system into which it is
ample that a specified security level in a secure communications system has been reached .
to be incorporated . This means that detailed records on such things as test cases and test
When the developer is satisfied that the mathematical requirements specification is cor-
coverage must be kept : more so than would normally be kept for conventional software
rect, it is then designed and specified in terms of mathematics . The mathematical design
development .
is then validated with respect to the specification . This mathematical specification of the
The second implication is that directives must be given, via standards and pro-
design is then transformed into program code by staff responsible for implementation .
cedures, to system designers which specify that the first thing they should do is browse
Formal methods of software development offer a great number of advantages to the
through a class library, and if they feel that components from the library are unsuit-
developer . Because mathematics is an exact medium it means that ambiguities and con-
able for the system that they are designing, then they should produce some written
tradictions can be detected more easily than in natural language documents . However,
explanation as part of the design documentation .
there is a cost : staff who use such mathematical methods of software development have
The third implication is that the quality system should normally impose some direc-
to be highly trained, and inducting new staff into a project can take much longer than
tions to the constructors of classes to include some extra bureaucratic information in the
for staff who have been trained in more conventional methods . However, the gains can
classes which enable staff who are going to browse through the class library to identify
be considerable in terms of greatly reduced error rates and projects which do not exceed
potential candidate classes for reusability . For example, a programming standard could
their budgets . Because of the cost in terms of implementing formal methods, they are cur-
insist on an exact description of the functionality of the individual components of a class .
rently used only for projects where there would be very large costs to the developer if an
The costing procedures for conventional project development should also be modified
error occurred during operation . Such projects are mainly confined to the safety-critical
for object-oriented development . We saw in Chapter 3 that the main method for costing a
and security areas .
project is to identify a list of tasks and then cost them on a historical basis . This normally
takes place during the early stages of the requirements analysis and specification part of
a project . However, the cost of an object-oriented system will depend on the degree of Self Assessment Question 9 .5 Would a formal methods project have
reusability that is to be employed . Therefore, the planning procedures of a company which the same life-cycle as a conventional project in the same application area
uses object-oriented technology should be modified so that some preliminary assessment but which used graphical notations?
of the degree of reusability is carried out . Not only would this help to give rise to more
accurate cost estimations, but it would enable project management to check out any Solution Yes, formal methods are purely a technical solution and the se-
claim from system designers that they were unable to reuse much of a class library : quential mode of development where activities follow each other would be
designers are notorious for adopting a not-invented-here syndrome . used .
Another impact of object-oriented technology is on the reviewing process, particu-
larly if a company encourages new classes to be inserted into a class library . This means
that standards and procedures should exist for class reviews rather than module reviews .
Indeed, a developer who uses an object-oriented programming language should not really 9 .4 .2 Formal methods and software quality assurance
use module reviews since the vast majority of the validation that would occur in such Both prototyping and object-oriented technology represent major divergences from con-
reviews would be carried out in a class review . ventional software development . Happily, formal methods do not, and require the same
One of the most difficult parts of object-oriented development is testing, particularly conventional development models that are employed by many software developers . The
system and acceptance testing. When a failure occurs in a system or acceptance test it modifications to a quality system required for formal methods tend not to be too great .
is often not clear where it occurs, particularly if a class is to be instantiated a number The main impact concerns standards for specifying the mathematics used in require-
of times for different objects . Because of this, there should be standards and guidelines ments specifications and system designs, together with the procedures and standards
which direct the designers and constructors of classes to insert debugging information used by staff who conduct reviews . The mathematical notations used in a formal method
into the class which enables staff to identify the cause of a malfunctioning test . are, in a superficial sense, similar to programming languages, and hence require detailed
standards for their layout : what indentation to use ; how to separate the components of
a formal specification ; what conventions to use for the names of variables ; and how to
 144 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION QA AND THE NEW TECHNOLOGIES 145

link together parts of a requirements specification or system design specification which • The generation of code from a design .
are associated by virtue of the fact that they deal with connected sets of functions . • The storage and manipulation of project management information .
One of the major advantages of a formal notation is that reviewing becomes a more
structured process . For example, a component of certain formal methods of software
development is a statement known as a data invariant. These tools, and their associated structured methods, have a potentially great influence
This is a mathematical expression
which remains true throughout the execution of a system, e .g . a data invariant may state on the quality management system of a company . The main influences are shown below :
that there will be no more than MaxUs users of a system .
One part of a review of a formal requirements specification can be structured around
• Many of the tools force the developer to produce requirements specifications and

the process of discovering whether the operations detailed in it violate the data invariant, system designs to a fixed set of standards ; for example, many Computer-Aided Soft-

and staff responsible for developing the requirements specification have to convince the ware Engineering (CASE) tools which manipulate data flow diagrams insist on rigid
members of the review that this does not occur . Normally, the act of convincing uses conventions for the way in which such diagrams are displayed . In effect, these tools
rigorous argument rather than formal mathematical proof ; but, nevertheless, there is enforce their own requirements specification and system design standards, and obviate

still the option of using such proof if rigorous argument does not convince . the need for the developer to specify his or her own standards .

Because of the special nature of mathematics as a specification medium it is im- • They relieve staff who attend requirements and system design reviews of many of the
portant that standards and procedures exist for the execution and documentation of more bureaucratic checking activities . Since they are capable of detecting errors, such
reviews . Such reviews, as in conventional development, are normally confined to require- as the omission of an identifier in a module, they free staff who attend reviews to con-
ments specification, system design, and program code reviews . centrate on important questions such as : does this system design actually implement

It is worth pointing out that even though a developer may use a formal method, there the requirements specification?
is still a need for system and acceptance testing standards and procedures . Formality does • They relieve development staff of the need to construct and generate a large amount
not get rid of the need to demonstrate the correctness of a system to the customer-all of useful but bureaucratic information .
it does is to reduce the levels of error discovered during this process . • They increase the degree of traceability present in system documentation . For exam-
ple, the documents produced by CASE tools enable staff to trace from a system's
functions to the final code which implements the functions without expending very
9 .5 STRUCTURED METHODS AND CASE TECHNOLOGY much effort .

Until the early eighties the vast majority of software developers used natural language
for requirements specification and design . However, the early eighties saw an increase in 9 .6 SUMMARY

use of the graphical notations associated with what are known as structured software
development methods . The main advantage of these graphical notations was that they This chapter has described only a small number of new technologies . However, each

offered a high level of precision and a huge improvement over natural language . The case demonstrates an important point which is true for all new technologies : that to

disadvantage was that in order to be fully effective they required tool support . In 1981 I implement new advances in software development without considering their effect on the
used a graphical notation to specify a large system ; one of the things I noticed was that quality system will lead to the quality system becoming out-of-date, and giving rise to
activities which are less than optimal in terms of the detection of errors .
when I made a mistake I needed to carry out a very large amount of rework to recover
from the error-even if it was a comparatively small one . Much of this rework could have
been carried out in a semi-automatic fashion if tool support had been available . By the
late eighties the tools were becoming available ; they were implemented for high-end work 9 .7 FURTHER READING
stations or the more powerful PC configurations . Typically, these tools offered a number
of facilities : Probably the best treatment of prototyping within a management information systems
environment is Connell and Brice-Shaffer (1989) . While the authors do not explicitly
address quality assurance, they do provide enough technical details to enable quality
• The ability to create diagrams which describe the functions of a software system ; assurance staff to write standards and procedures to govern the process . Ince (1991)
• The ability to create diagrams which describe the system design of both the processes describes the use of object-oriented technology within software engineering and touches
and the data in a system . on the testing and quality assurance aspects of the subject . Beech (1986) is a good
• Error-checking facilities, such as the ability to point out errors in the flow of data in book-length introduction to object-oriented development with an industrial bias, while an
a requirements specification . excellent new book (Sully, 1993) describes object-oriented development from a structured
• The ability to process a graphical requirements specification and simulate its actions methods perspective .
as a rudimentary prototype .
146 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

A good introduction to formal methods of software development can be found as

a chapter in Ince (1988) . For the more mathematically inclined Cohen, Harwood and
Jackson (1986) is probably the best book-level introduction . An excellent case study of
the use of formal methods and statistical testing can be found in Cobb and Mills (1990) .
An easy-to-read introduction to structured development methods and associated tools
technology is Martin and McClure (1987) .

PROBLEMS

9 .1 A dynamic analysis tool monitors the frequency of execution of the code in a system developed
using a third-generation language . What modifications to a quality system do you think are needed to
cope with the introduction of this tool?
9 .2 A static analysis tool is a testing tool which detects errors in program code without executing that
code. Typically the sort of errors that are detected by a modern static analysis tool are : uninitialized
variables, program code that can never be executed and variables given a value which are never used
before another value is given . They are also able to check that certain conditions holding between
variables are true . What quality factors do you think this tool might improve?
9 .3 What quality factors do you think prototyping addresses on a software project?
9.4 What areas of a quality manual do you think would need to be changed if a company adopts a new
notation for the specification of functional requirements?
9.5 What areas of a quality manual do you think would need to be changed if a company adopts
object-oriented technology?
 10
QA AND THE HUMAN-COMPUTER INTERFACE

AIMS

• To outline the relationship between the human-computer interface and software qual-
ity assurance .
• To describe the various quality documents produced for the development of the
human-computer interface .
• To examine the various tasks which are involved in the development of the human-
computer interface .

10 .1 INTRODUCTION

One of the most widespread weaknesses that I find in quality systems is the fact that they
rarely, if ever, address important issues concerned with the human-computer interface .
The result of this is that computer systems are still being developed which, although they
meet the functionality described in their requirements specification, take no account of
the capabilities of the user, the tasks that the user has to carry out, and the environment
in which the system works . Some examples of systems I have come across where major
problems have occurred are :

• The system which used auditory feedback to indicate that an error had been commit-
ted, or that the user was not allowed to carry out a particular operation . Auditory
feedback is an excellent mechanism for this ; however, this system was situated as
part of a control house in a chemical plant which had not been soundproofed . The
result was that the users of the system-staff responsible for operating the plant-
were unable to hear the feedback, and on two occasions a serious accident was only
narrowly averted . One of the operations staff who, quite reasonably, wanted to live
until retirement, had taken the matter into his own hands and built an amplification
system . This took the feedback from the computer and transmitted it to control staff
via a rather old but large loudspeaker which he had kept in his garage .

147
14 8 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION QA AND THE HUMAN-COMPUTER INTERFACE 149

• The data entry system which consisted of a number of transaction types that were These, then, are examples of systems which, although they delivered the required func-
initiated through a windows interface, where each transaction was designed in the tions, had such a poor interface with the user that they were virtually inoperable . I have
same way, but with very small differences in aspects such as the shape of the window deliberately chosen some poor systems ; however, even in the more successful systems
icons and their positions . It was almost as if the developers of the system had assigned there are still annoying quirks which make them much less effective than they deserve to
the design of the interface to separate teams who never communicated with the other be . Normally, such poor interfaces arise from an inadequate perception of the character-
teams . Operators of this system were continually making errors and, even in times istics of the user; a hazy idea of the tasks that the user has to carry out ; and a neglect
of recession, were leaving because of the awkward interactions that they were forced of the social, managerial and physical environments which the user works in . A good
into by the system . quality system should ensure that this will not happen .
• The system intended for schoolchildren, that expected the users to use single line-by-
line commands rather than menu commands . The system was aimed at very timid
users, and the experience of continually making mistakes put many of them off using 10 .2 THE USER/ENVIRONMENT QUESTIONNAIRE
computers .
• The school records system where a careless use of two of the commands available to The first important document necessary for HCI design that forms part of a quality
the teacher user resulted in the complete annihilation of the database . system is a user/environment questionnaire . This is a document which contains questions
• The system which purported to have a help system . In a sense it did, because the about the users : their capabilities, the environment they work in, and the tasks which
user was able to invoke a window which contained instructions on how to use the they carry out . The answers to this questionnaire, elicited during the early stages of the
current command and the various options that were available . However, nobody had requirements analysis process, and often physically bound as part of the requirements
checked the text of the help messages . They were barely grammatical, and contained specification, will be an important input into the process of designing a system's interface .
a large amount of technical jargon which some members of the development team Typical questions which would be asked in such a questionnaire are shown below :
later admitted to me that even they could not understand .
• The system where each command had quite a good interface, but where the user was • Who are the users of the system, i .e . who are the people that are to sit at a terminal
asked to carry out a sort of digital acrobatics when combining a number of commands or VDU and interact with the system?
together to achieve some aim . The system was for foreign exchange dealers and, when • What are the current computer skills of the users? Are they first-time users ; infre-
in use, resulted in a number of mishaps which lost the bank using the system around quent users who will remain infrequent ; infrequent users who will be asked to become
£200000 in two days . When the mishaps were notified to the user's management the frequent users ; or frequent and practised users?
system had to be withdrawn ; the interface redesigned and implemented ; and a large • What is the nature of the tasks that they have to carry out? Do the users, for example,
amount of expensive training material and courses had to be ditched and redeveloped . have to navigate through a complex database or produce a large number of drawings?
• The system which provided facilities that enabled the user to navigate through quite • Is the task repetitive or does it change from day to day?
a complex geographical database . Unfortunately, nobody really had much idea about • Are a number of users involved in the interaction with the computer? For example,
how complex the navigation tasks were, and no facilities were provided which enabled might two or three users be sitting in front of the screen and talking to each other?
the user to keep track of where he or she was in the database . Users continually gave • Do a number of users interact together via the computer system ; for example, as
up in frustration at being unable to return from a path through the database . The users of a system which simulates the conditions on the trading floor of a financial
result was that the system, which was originally targeted for a population of around institution?
200 users, was hardly used at all. By the time navigation facilities were tacked on • If the system does consist of a number of interactive users who can affect the actions
after delivery, the system had got such a bad name that the user population was of other users by means of the computer, are there any other modes of communication
around 20 people . involved, for example shouting?
• Is the time taken to carry out an operation critical?
• Is the environment in which the system is to be used poorly lit, noisy or dusty?
Self Assessment Question 10 .1 How would a QA system have avoided
the problem detailed in the first bullet point above?
Self Assessment Question 10 .2 How might the factor outlined in the
Solution There would normally be a checklist which detailed such environ- fifth bullet point above affect the design of a system?
mentally important issues which the HCI designer would need to examine
before doing very much work on the interface . Solution Usually it would affect basic design decisions which concern the
size of screen to be provided and the possibility of having two keyboards
or mice in tandem .
 1 50 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION QA AND THE HUMAN-COMPUTER INTERFACE 151

Self Assessment Question 10 .3 How would information gathered from

the question detailed in the eighth bullet point above be used?

Solution It would be used to design the interface such that fast response
by the user is possible . It would also form an input into the system and Book return
acceptance testing process, whereby tests would be set up which involved
real users and which monitored the speed of response by these users .
Transactions

The answers to these questions and many others will determine the nature of the interface
that is produced . For example, the interface to a military command and control system,
Errors
where the users might be very experienced, where access is restricted, where noise is a
problem and where lighting is weak, might be designed mainly as a touch screen system ;
where a system has a windows interface, the use of a mouse and auditory feedback would Invalid Book already
be totally inadequate . I Enter details
into card reading returned
Another useful HCI component in a quality system is a short document which de- reader
scribes how the HCI designer is going to respond to the information gathered by the
questionnaire, a document sometimes called the HCI Design Rationale. At this stage in
the project the process of requirements gathering is probably still proceeding, so that Figure 10.1 An example of a task specification notation .
this rationale will still be a fairly succinct document . However, it should explain the
general nature of the interface, based on factors such as the capabilities of the user and The task specification is one of the most important documents produced in the
the physical environment in which the system is to be placed . early part of a software project . It should be completed by the time the requirements
specification has been finished . It is often a good idea to either have a separate technical
review of this document, or to have a review in which the user tasks are validated
10 .3 THE TASK SPECIFICATION against those functions of the system to be carried out directly by the users . This is
an exceptionally good check, not only on the completeness of both documents, but also
Another important part of a quality system is the task specification . There will be a task on whether the HCI staff and those analysts responsible for producing the requirements
specification for each category of user of a system . The purpose of a task specification is to specification have carried out their jobs correctly .
document the tasks which the user of a system carries out . The most useful notations for
this tend to be graphical and hierarchical ; an example is shown in Fig . 10 .1 . It describes
one of the tasks carried out by the user of a library system . At the top level of the 10 .4 HCI DESIGN
hierarchy is the overall task of responding to a returned book . At the next level of the
diagram is a series of blocks which describe the various subtasks involved in this . The Once task specification has been completed and reviewed, the next stage is to carry out
diagram .shows some aspects which are not automated ; for example, the insertion of an a detailed design of the interface in terms of the interaction between the user and the
identification card into a reader . Those parts of the system which are not automatable are system . The task specification should drive this design process . An important component
enclosed in dashed boxes . The subtasks specified at the second level of the diagram include of the process is an adequate notation for expressing the interaction . There are a number
entering book details ; responding to an error in entering these details ; and responding to of notations, and Fig . 10 .2 shows a particularly appealing one which illustrates how a user
the rather unlikely, but possible, error that the book has been previously returned and progresses through the facilities of a system . The diagram, which superficially looks like
not logged out by the user when it has been borrowed . the type of graphical notation used in CASE technology, describes the flow of a system
An interesting feature of the notation is the use of boxes marked with a circle or an in terms of the actions taken by the user and the responses given by the system . Each
asterisk . The former indicates a task which has been split up into a number of alternative bubble in the diagram represents a state that the system is in awaiting some response
tasks (the diagram shows an example of this when normal processing or an error condition from a user . Each arc in the diagram represents the transition from one state to another .
occurs) . A task marked with an asterisk is repetitive ; for example, the task of inputting Every arc is labelled with a number which refers to an entry in a table . The arrows which
book details is potentially a repetitive one since there is the possibility that an error do not point at any bubbles represent connections to other parts of the diagram which
will be committed during the process of inputting the data, thus leading to a potentially are not shown . Table 10 .1 shows a fragment of such a table . The numbers which label
infinite application of the task of communicating book data . arcs represent the event which caused the transition, the response of the system, and a
 1 52 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION
QA AND THE HUMAN-COMPUTER INTERFACE 153

Self Assessment Question 10 .4 Who would you invite to this review?

7 .I 7 .3
Solution The member of staff who designed the interface would be invited ;
also, the designer of the overall system . The remainder of the participants
would be free for the project manager to choose . I would invite an HCI
designer from .. another project and a senior programmer who has responsi-
bility for some implementation aspects of the project .

At this stage in the project, assuming that both the HCI design review and the task
specification review are completed successfully, the architectural features of the design
will be complete and the project will have started implementation of the system .
The role of the HCI staff at this stage of the project is to carry out the detailed
design of the screens or windows that make up the human interface part of the system .
There are two quality documents which are required for this . The first is a document
which describes guidelines for screen and window design . Such a document would offer
advice on screen layout, amount of information, use of tabular displays, the display of
colours, and the use of effects such as flashing, inverse video and underlining .
Figure 10 .2 The documentation of user interaction . The second is some form of standard for specifying the screens or windows that
are to be designed . A good standard is normally very simple and contains a typical
Table 10 .1 A fragment of a transition table screen with the fields highlighted, together with bubbles which provide an explanation
Transition
of the components of the screen . What is important about this notation is that it should
Action Screen
be readily understandable by the programmer : that staff responsible for developing the
1 Update command initiated TOP-UPDATE modules which implement the interface can understand the meaning of each field, the
2 Query command initiated TOP-QUERY
3
data it is associated with, the format of the data, and the use of facilities such as inverse
Help facility for update initiated HELP-UPDATE
4 Help facility for query initiated
video .
HELP-QUERY
5 Return from query help TOP-QUERY
6 Return from update help TOP-UPDATE
10 .5 HCI EVALUATION
reference to any screens or windows which will be displayed when the transition occurs .
These are written into the table displayed (shown as Table 10 .1) . Evaluation is the process of validating the HCI part of a system in order to check that it
is usable by the group of users who will normally employ it . The evaluation process can
Normally, since screen and window design is generally left until a later stage of the occur at two points in the development process : at the beginning of development when
project-usually programming-the final column of the table is left blank . In Table 10 .1,
system requirements are being elicited, or after implementation . If it occurs at the front-
the third entry describes the fact that the user has initiated a HELP facility which
results in a transition from state 7 .3 to 7 .4, with the result that a help facility window end of the software project, then it is normally used in conjunction with some form of
prototyping : the development of an early flawed working version of a system . If it occurs
is displayed, the screen associated with this facility being HELP-UPDATE . Another
at the back-end of a project, then it becomes part of the system testing and acceptance
possible validation exercise that could be scheduled when such an HCI design is produced
is an HCI design review . There would be three major documents that would form an testing process . Prototyping and its QA implications were dealt with in Chapter 9 ; this
input into this process : the task specification, the system design, and the HCI design section examines post-implementation evaluation . There are five ways of evaluating a
specification itself . The aims of the review are twofold : first, review staff examine the system (Preece, Benyon et al., 1993) :
task specification and check that each task has been implemented as part of the interface ;
• The notation used to express the interface is analysed . Here, properties of the interface
then they examine the system design and the HCI design in order to check that there
notation, such as the number of bubbles in a graphical diagram, are used as an input
are modules in the design which invoke the HCI facilities and that they correspond to into the process of producing qualitative judgements about the effectiveness of the
the functions being exercised by the user .
interface .
• The interface is evaluated by experts .
 154 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION QA AND THE HUMAN-COMPUTER INTERFACE 155

• Observational techniques, where users interacting with an interface are directly ob- good way of rapidly pinpointing difficulties in an interface and provides an excellent, large
served, and measurements such as the number of errors made, are used to judge the quantity of qualitative data . However, it suffers from a major problem in that it requires
effectiveness of the interface .
quite a large amount of human resources and can also be very time-consuming . If the risk
• Questionnaires are used to evaluate users' perception of usefulness . analysis pointed out problems such as the fact that the customer was not able to specify
• Experimental methods are used to explore hypotheses about the interface . exactly the skills that users will have, that the system to be developed was critical in terms
of operators not making many errors, that inexperienced staff are involved in developing
This is not a book about HCI design, so I will not describe each of these techniques in the interface, and so on, then the resources required for observational evaluation would
detail . What I will do is to examine one technique, that based on observation, and look
be fully justified .
at the QA implications . In general these are the same as for any other technique . There
Another important factor is the recording of the evaluation . Observational evaluation
are a number of ways of carrying out observational evaluation (Benyon, Preece et al ., gives rise to a large amount of unstructured notes which are generated while observing
1993) :
human operators . This should be summarized in a document known as an HCI Evalua-
tion Report, which forms part of the quality records for a project . This document should
• Direct observation. Here an observer examines users interacting with a system . be divided into sections corresponding to each task, with the results of evaluations con-
• Video recording . Where a video camera is unobtrusively positioned in order to record
cerning each task summarized in some convenient form such as a table . For an evaluation
the user's activity. technique which uses questionnaires, a table would summarize the attitudes of the users
• Software logging . Where the system whose interface is to be evaluated is modified by towards the interface . For observational evaluation, a table would summarize the tasks
inserting probes that record the interaction between the user and the system . that were carried out, for example errors made, successful recoveries from an error carried
• Interactive observation. Here a human operator pretends to be the system . He or she out, any examples of the user taking a considerable time to carry out a task, and the
intercepts user input and attempts to amend system output in order to reduce the number of times a help facility was invoked .
problems with an interface .
• Verbal protocols . Here a user interacts with the system and, at the same time, voices
his or her thoughts into a tape recorder . 10 .6 POSTSCRIPT

The impacts of this form of evaluation on a QA system are various . First, the project plan The important point to stress about the development of the human-computer interface
should describe the evaluation method that is to be used, predict the resources needed, is that it is an engineering task, and the same principles which govern other engineering
and outline when, during system or acceptance testing, it occurs . For example, for most tasks on the software project should be adhered to . For example in Chapter 1 I stressed
projects customers are not too concerned about specifying acceptance criteria for the the fact that software documentation should have a high degree of traceability. This
human-computer interface and simply state that it should be usable . Here, the developer same principle holds for HCI documentation : a reader of the HCI documentation of a
would only employ evaluation during system testing in order to iron out any problems project should be able to trace from the task specification to the HCI design, from the
with an interface ; resources should hence be specified only for the system testing phase . HCI design to the individual specification of screens, windows and menus, and from
In some cases, however, the customer may be very keen to specify acceptance criteria . these to the modules which implement the interface . As well as this, there should also
This occurs often with safety-critical systems, which have considerable human interaction be traceability through HCI from the requirements specification to the task specification
and where the customer may specify that certain classes of error should never happen, and hence into the remainder of the HCI documents .
certain classes occur at a certain rate, and so on . In this case the software developer may
choose one method for system testing, such as direct observation, and use the results
from software logging, applied during acceptance testing, to provide evidence that it is 10 .7 QUALITY MANUAL REQUIREMENTS
highly likely that HCI acceptance criteria will be met during operation .
What is important is that if a technique such as direct observation is used, the The following documents which are concerned with the human-computer interface should
resources, acceptance criteria and the reason for using the techniques should be described be found in a quality manual . Normally, they are bound as one document which is often
in the project plan . It is also important that if a combination of methods is used, the called the Human-Computer Interface Standards and Procedures :
way in which they interact is specified . The project plan should also describe what data
is to be extracted from observational studies : metrics such as the number of times a 1 . User/environment questionnaire checklist
command was invoked, the frequency of errors, and the percentage of correct recovery 2 . HCI design rationale standard
from an error .
3 . HCI design rationale procedure
The project plan should also provide a rationale which describes why the chosen 4 . Task specification standard
evaluation method has been selected . The choice of evaluation method is determined 5 . Task specification procedure
primarily by the results of a risk analysis ;; for example, observational evaluation is a very 6 . Task specification review standard
1 56 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

7. Task specification review procedure

8. HCI design review standard
9. HCI design review procedure
10 . Screen and window design guidelines
11 . Screen and window specification standard
12 . Interface evaluation procedure
13 . Interface evaluation guidelines
14 . HCI evaluation report standard .

10 .8 SUMMARY

Consideration of the human-computer interface is often omitted from quality systems .

This is a major mistake . The interface is the part of a system which initially gives the
user his or her first impression of its quality . A good quality system should offer staff a
number of facilities connected with the development of the human-computer interface :
these include facilities for specifying and evaluating an interface, together with guidelines
which describe the overall design of the interface and the design and implementation of
components such as screens .

10 .9 FURTHER READING

Preece, Benyon et al. (1993) is a book written by a number of my colleagues at the

Open University. It is an excellent introduction to the human-computer interface and
provides a splendid overview of the process of developing customer-friendly interfaces .
Schneiderman (1987) is a slightly academic book which, however, contains plenty of
material that can be used for writing standards and procedures which govern the process
of developing the human-computer interface .

PROBLEMS

10 .1 For the first four bullet points on page 149 describe why these questions are important to the
system developer .
10 .2 What would you expect in a standard for a textual notation for task specification?
10 .3 What sort of problems do you think HCI evaluation would pin-point?
 11
SOFTWARE METRICS

AIMS

• To describe what a software metric is .

• To outline some of the uses of software metrics .
• To describe the relationship between software metrics and quality assurance .
• To briefly describe some of the main product metrics .
• To outline a methodology for the development and use of metrics .

11 .1 INTRODUCTION

The aim of this chapter is to provide an introduction to the subject, describe some of
the more useful metrics and show how even the least ambitious software developer can
derive metrics which are useful .

11 .2 AN EXAMPLE

In order to describe what metrics are, and how they can be useful, it is worth looking at
an isolated software task: that of module programming .
System design is the process of defining a modular architecture of a system and
specifying the processing that occurs in each module of a system design . Programmers
take module specifications, program them and then test them using test data which
checks the functionality of the module and exercises the program code . There are a
number of factors which make the programming task difficult .
First, there is the module specification . The larger or longer this description, the
more effort will be required to produce the module . The second difficulty is the degree

157
 1 58 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SOFTWARE METRICS 159

to which the control flow of the module is unstructured . For example, the more criss-
crossing of control flow the more difficult it will be to understand the logic of the module Self Assessment Question 11 .2 Why would the number of parameters
and test it . in a module be an indicator of the amount of work required to implement
A third problem is the way in which a module is connected to other modules . For the module?
example, if a module calls a number of other modules, then the process of debugging
this module will be very difficult : keeping details of data affected by called modules can Solution The number of parameters in a module is usually a good indica-
clutter up the programmer's memory to the extent that errors in tracing control flow in tion of the amount of functionality bound up in the module . For example,
the module being debugged occur . a module which has a single integer parameter often just updates that pa-
These, then, are three ways in which structural features of a module can affect rameter, while a module which has a number of array parameters usually
the programming, testing and debugging process . These can be possibly quantified and carries out quite a degree of looping amongst the arrays .
expressed as numbers . For example, the length of the narrative describing a module's
functionality could be a good indicator of the difficulty in implementing the module or
the number of control flow criss-crossings might be a good indicator of the degree of
difficulty of testing a module . 11 .3 DEFINITIONS

Before looking at some of the main families of metrics it is worth looking at some defi-
Self Assessment Question 11 .1 Do you think that the derivation of the nitions .
number of criss-crossings in a module is a difficult process? First, a product metric is a measure which can be extracted from a document or file
which forms part of a quality system . Typical documents include the program code or
Solution Done by hand it is tedious and error-prone . However, a software system design . A process metric is some number which can be extracted from a process
tool which extracts the control structure of the code of a module would on a software project . One example of such a metric is the time taken to design a system .
calculate this metric easily. A predictor metric is a metric, normally a product metric, which has the potential to
predict another metric known as a result metric . So, for example, the length of a module
used to predict the amount of resource required to implement the module is a predictor
The aim of metrics research is to derive key features which can be quantified and which metric while the amount of resource is the result metric .
can then be used for quality assurance processes . In software engineering there are a
myriad of possible numeric quantities that can be extracted which may have a relationship
with some activity which a quality system addresses . Some examples are shown below : 11 .4 THE USES OF A METRIC

• The number of modules in a system design being an indicator of the amount of There are a number of uses of a metric including the predictive use outlined in the
resources needed for implementation . previous chapter .
• The number of parameters in a module being an indicator of the amount of resource
required to implement the module . • As a numerical quality threshold . This is probably the most popular use of a metric
• The number of bubbles in a data flow diagram being an indicator of the size of a currently. If a metric has been found to have a good relationship with some result
system to be implemented . metric, then a quality standard might insist that staff carrying out a particular task
• The number of variables in a module being an indicator of the degree of difficulty do not produce artefacts whose metric values exceed some threshold . For example,
that a programmer will find in debugging the module . assume that a company has discovered that the number of control flow criss-crossing
in a module determines the number of errors in that module, then they might insist
The main activity that metrics researches carry out is deriving such metrics and validating that programmers do not create modules which exceed a particular value of criss-
them : showing that they correlate with some key project activity such as system testing crossings .
or programming . • As a predictive mechanism . This is the aim of all metrics research : to be able to take
a value of a particular predictor metric and then predict some result metric . This
is not much used within a project for technical activities but there are now some
metrics-based techniques which are used for project costing .
• To keep track of the degradation of a software system . During the software mainte-
nance process the structure of a system usually degrades . Random patches applied
by members of the maintenance department often result in a system structure which
 SOFTWARE METRICS 161
160 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

looks tackier and tackier . Gradually the system becomes more and more difficult to 11 .5 .3 System design metrics
maintain . System design metrics can be used to keep track on the degradation of a The major problems with both code and detailed design metrics is that they are extracted
system and alert a project manager to the fact that he or she needs to allocate staff too late in a project where little can be done if poor values are discovered . Because of this
to the process of cleaning up the system . the eighties saw an increasing trend towards the development of system design metrics :
numbers which can be extracted from a document that describe the modular architecture
of a system .
11 .5 A TOUR OF SOFTWARE METRICS The driving idea behind the vast majority of system design metrics is that the quality
of a good design can be quantified by examining the degree of coupling and cohesion
The aim of this section is to describe some of the main categories of product metric which
that can be found in the design . Coupling and cohesion measure the degree to which a
are available . module in a system is connected to other modules . A module which is lightly coupled
can be read, developed and understood in isolation and, hence, can be debugged more

11 .5 .1 Code metrics easily and integrated more easily.

Normally, system design metrics try and extract features which are related to cou-
These metrics are those which can be directly extracted from program code . A typical
pling and cohesion . Tools for extracting these measures count entities such as the amount
code metric might be the number of declarations in a program . of access there is to global variables and the degree of calling of other modules there are in
Code metrics have been the most popular metrics with both researchers and indus-
specific modules . The most popular system design metrics are due to Kafura and Henry
trialists . There are two reasons for this : first, they are easily extractable since program (Henry, 1981) and Shepperd (Shepperd and Ince, 1990) .
code is always held in files ; and, second, they are quite easy to extract . The development
of the program code of a tool for extracting code-based metrics is a regular exercise given
to second-year computer science students .
11 .6 THE DERIVATION OF METRICS
Much of the work on code metrics that has been carried out has been inspired by the
late American computer scientist Maurice Halstead (Halstead, 1977) . He postulated that One of the problems with the use of metrics is that industrial users tend to take an
useful properties of programs can be found by extracting the operands and operators
off-the-shelf attitude towards their use on a project : in effect they pick one well-known
from the programs . An operand is typically an identifier, while an operand is either some
metric and hope that it works . The aim of this section is to counteract this tendency, and
arithmetic operator such as * or a sequencing operator such as repeat . outline a more goal-oriented method for devising metrics for both a company and the
Halstead's work was all-pervasive . The vast majority of metrics papers written in software projects it undertakes . The method is due to Victor Basili and his co-workers
the seventies directly cited his work . However, in the eighties it has fallen into disrepute .
at the University of Maryland (Basili and Rombach, 1988) . The method is known as the
There are two reasons for this . First, a number of researchers have pointed out that very Goal Question Metric method (GQM) .
little statistical correlation can be found between the counts that Halstead specified and The main starting point in GQM is a goal . Any metric extracted in a project must
important tasks such as debugging . The second reason is that you cannot do anything reflect some purpose, and the first thing that someone who wishes to use metrics has to
really useful with a metric extracted from program code . By the implementation stage do is to ask some questions about what the reason for gathering metrics is .
of a project much of the project's developmental resources will have been consumed and Once the goals have been decided, the next stage is to ask a number of questions
there will be very little scope for using the metrics information, for example in modifying which clarify the goals (the Q part of GQM) . Most goals tend to be expressed in nebulous
a system design . terms and the questions enable staff to focus in on some metrics . For example, a company
may wish to employ a metric to satisfy the goal that their projects deliver fewer residual

11 .5 .2 Detailed design metrics errors through more thorough testing . A number of questions suggest themselves from
this :
These are metrics which can be extracted from a detailed design : they are often called
graph metrics or control flow metrics . They arose from work carried out by the American What sort of thoroughness is implied in the question?
computer scientist Thomas McCabe, who developed probably the most famous detailed
•
Does the question imply thorough functional testing?
design metric known as the cyclomatic complexity metric. A tool for extracting detailed
•
Does the question imply thorough structural testing?
design metrics examines the control flow of a module or program and extracts the se-
•
Do the errors that are left in a system through poor testing arise from poor perfor-
quence of instructions that make up conditional statements and loops . It then calculates
•
mance on a particular phase?
some metrics based on the degree of unstructuredness of the module or program . How can we identify errors which remain in a system which have arisen from poor
•
The main use of a detailed design metric is as a quality threshold . Often programmers
testing?
are instructed not to exceed a cyclomatic complexity number for the modules which they
program .
 162 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SOFTWARE METRICS 163

Once these questions have been answered, the next stage is to identify the metrics which • The derivation of metrics .
are important . Let us assume that the company who have asked the above questions have • The identification of hypotheses which involve these metrics .
identified the fact that the vast majority of errors that emerge during operation are due to • The planning of the measurement process .
inadequate module testing: that programmers tend to inadequately test certain paths and • The collection of data from the measurement process .
conditions . One metric which might be identified as helping to reduce structural testing • The validation of the data .
errors is the condition coverage achieved by a programmer : that if every condition that • The analysis of the data .
was exercised in a module was executed for its true and false values, then there would • The application of the derived metrics on future projects .
be a considerable reduction in the number of testing-related errors from operation .
The next stage is to identify a hypothesis or set of hypotheses which can be tested It is worth stressing that this is rarely a strictly linear process . For example, the posing
in the developer's projects . Let us assume that a possible hypothesis is this : of questions often modifies the goal and the analysis of the data often results in the
modification of the hypotheses . Nevertheless, it forms a useful way of developing metrics
That by insisting on 100 per cent condition coverage we will reduce the number of residual errors which can lead to considerable process improvement .
due to inadequate module testing by 98 per cent .

This is rather a strong hypothesis, both in terms of the hard numbers used and also in 11 .7 FURTHER READING
terms of what might be possible on a software project, since deriving test data for 100
per cent condition coverage is mightily difficult . The hypothesis can be relaxed to a lower Fenton (1991) is a good introduction to metrics technology . Be warned, however, that
percentage . it occasionally becomes quite mathematical. Kitchenham (1992) is an excellent tutorial
Once the hypothesis has been derived the next stage is to plan the measurement on metrics . An excellent feature of this book is a large annotated bibliography . Two
process on some sample projects which attempt to validate the hypothesis . For example, excellent books on the implementation of metrics on industrial projects are Grady and
this might involve, for the testing example, activities such as writing a small software Caswell (1987) and Goodman (1993) . The former describes the implementation of a
tool which tracked condition coverage, modifying test documentation to include details metrics programme at Hewlett-Packard ; the latter is the distillation of one of Europe's
of condition coverage, introducing an extra procedure for those staff charged with fielding most experienced metrics practitioners . A good project-specific case study can be found
errors from customers and providing resources to track down the source of the errors . in Durst, Stark and Pelnik (1992) .
Once the measurement process has been planned, the next stage would be to collect
the data With the example used here this would mean collecting coverage percentages
and defect data identified as being due to module errors . Once this data is collected the PROBLEMS
next stage is to use statistics to validate it . The statistics would be used to check out
the hypothesis . Sometimes the hypothesis has a high degree of truth in it, since after all 11 .1 What aspects of a module expressed in a third-generation language do you think could be quantified
and used as a metric for characterizing difficulty of debugging?
it is conjectured by experienced staff, but it may need some amendment . For example,
11 .2 Write down part of a procedure which describes the fact that a programmer is not allowed to
it might be discovered that the size of the interface of the module has an effect on the develop a module which has a cyclomatic complexity higher than 10 .
number of residual errors due to inadequate testing .
11 .3 How might you use a detailed design metric to influence the integration activities on a software
Eventually, after trial and error, the developer will either validate the hypothesis or project?
a variant . The final stage, then, is to apply the metric to all future projects . For example,
11 .4 Can you think of some metrics which can be used to determine the degree of readability of a
if the hypothesis above was modified to : requirements document, say a statement of requirements produced by a customer?

That by insisting on 85 per cent condition coverage we will reduce the number of residual errors
due to inadequate module testing by 95 per cent .

Then programmers on future projects would be directed, via a module programming and
test standard, to provide evidence that their functional tests have covered 85 per cent of
the true and false outcomes of the conditions in a module .
The stages in this process of metrics identification and eventual application in the
GQM method are :

• The definition and clarification of goals .

• The posing of questions in order to make the goals clearer .
 PROCESS MODELLING AND PROCESS ASSESSMENT 165

far from the truth ; the sequential model of software development which I presented in

12
Chapter 2 is almost a myth . This will become clear from the following two examples .
The first example is the large project which has a long duration . Such projects will
be affected by requirements changes throughout their life . These changes will often neces-
sitate simultaneous changes to the requirements specifications, system design, program
PROCESS MODELLING AND PROCESS ASSESSMENT code, and test suites . The changes give rise to a large amount of parallelism ; for example,
redesign due to a requirements change may be carried out at the same time as program-
ming . To keep track of all these changes-within a number of parallel processes-requires
a fairly complicated configuration management system, particularly if the developer pro-
duces software packages . Because of the high degree of parallelism, and the information
needs of a large number of staff, there is a requirement for some sort of process modelling
notation to let staff know what is required to happen when changes need to be processed .
The second example is the prototyping project which has a number of subsystems
that are being prototyped in parallel . Each analyst or programmer assigned to the proto-
typing process will be modifying his or her subsystem in response to change requests by
the customer . Normally, in such a project, there will be some central data architecture
which each analyst or programmer needs access to via the prototype they have con-
structed . Now during the process of prototype development analysts will want to make
changes to this data architecture in response to customer change requests . Unfortunately,
AIMS a change arising from the prototyping of one subsystem may slow down another system
or even invalidate some of its functions . Clearly, there is a need for some coordination
• To describe the rationale behind process modelling . and a way of specifying how it is executed . This coordination is carried out in parallel,
• and ways of specifying it using natural language may give rise to major problems in
To describe some notations used in process modelling .
• To outline the nature of process assessment . interpretation.
• To describe some process assessment schemes .
• To describe how process assessment is able to guide process improvement . Self Assessment Question 12 .1 From your reading of the previous sec-
tion could you sum up in a sentence why process modelling is useful?
12.1 INTRODUCTION Solution Process modelling is useful because it is a way of documenting
and controlling the immense complexity that occurs in modern software
When you ask to look at a company software quality manual, more often than not you projects .
will be given a large, but rather unorganised document to look at . Some pages will be
written in natural language, some might be documented as flow charts and others may
only be sketched in the most abstract way . To read and extract information from such a
document can be a very difficult process ; for example, it might be very time-consuming 12 .2 PROCESSES AND PROCESS MODELS
to discover the source of a document which forms part of an important process such as
system testing . One of the aims of the research topic known as process modelling is to
Before looking at process modelling it is worth defining what I mean by a process. This is
provide notations which can be used to describe the processes that make up a software a task in a software project which progresses the project towards one or more of its goals .
project in such a way that information about these processes required by managers, For example, a design review is a process, as it progresses the project towards its goal
technical staff and quality assurance personnel can be obtained quickly and efficiently . It of achieving the correctness quality factor and, if the design review included a checklist
is worth re-emphasizing that process modelling is still a research topic . However, I would which evaluated maintainability, then it progresses the project towards achieving its
consider it sufficiently advanced that the results from research should start impinging on maintainability quality factor .
projects in the near future ; hence, its brief inclusion in this book .
A process will consist of a number of process steps. These are actions which cannot be
Probably your first reaction to reading this chapter is : why should we go to a great decomposed into any further subatomic actions ; for example, a process step of booking
deal of trouble in researching this area? Software projects are all rather similar ; they a room may make up the process of carrying out a design review . While a pedant may
merely consist of a series of tasks which are executed sequentially, with the output from point out that the task of booking a room could be split up into other steps, such as
one task being input into the next . All that is needed is simple descriptions . This is

164
 166 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROCESS MODELLING AND PROCESS ASSESSMENT 167

using a telephone and talking to the clerk in charge of room bookings, within the context ensures that his or her project enacts the process architecture by carrying out the
of a project where using a telephone and speaking to a booking clerk are well understood processes in a designated way .
a process step would be at a higher level . • To provide automated guidance for staff performing a process . Again, the eventual tar-
get which researchers in process modelling aim for is the ability to automate processes
in such a way that a member of staff who is carrying out a process, such as system
Self Assessment Question 12 .2 Would the activity of carrying out a testing, can call up, via software tools, information such as procedures, standards and
requirements elicitation meeting with the customer be an example of a pro- guidelines which enables him or her to carry out his or her jobs effectively .
cess step? • To aid automated execution support . The eventual aim of process modelling research
is to be able to develop notations for expressing the process architecture of software
Solution No, it really consists of a number of steps such as filling in a projects, so that staff who carry out developmental activities are provided with soft-
form after the meeting, updating the requirements specification and, per- ware tools which not only help them to carry out their developmental tasks, but also
haps, booking a room . generate data which can be used by project management or by the quality assurance
department . For example, defect data used in the process of evaluating the effective-
ness of a quality system, and defined in the process model, can be used to improve
A process model is a description of the interlinking processes that occur in a software technical processes such as design . The dream of many process modelling researchers
project with irrelevant detail, such as the use of a telephone in booking a room, omitted . is of a central database which holds the process architecture of a project . When staff
There are a number of uses for process models (Kellner, Curtis and Over, 1992) : carry out a particular process, such as module testing, they would be prompted by a
tool which would consult the database and guide the user through the steps required
• To support human understanding and communication . This is the use alluded to to carry out the process, calling in any tools that were required automatically, based
above . For example, if a company has a well-defined notation for expressing the on information within the process model database .
processes in its projects, and it is used in all these projects, then there will be few
problems in interpreting process descriptions when staff come to carry out a process .
• To support process improvement. Chapter 16 of this book stresses that a company
should always aim to monitor the effectiveness of its quality system . It should also 12 .3 NOTATIONS FOR PROCESS MODELLING
aim to improve its development processes . Process modelling aims to help both these
tasks . For this, a company should have a well-established set of standard processes Much of the research currently being carried out on process modelling concerns notations
which can be measured ; for example, the process of module programming would have used to construct process models . Indeed, perhaps one of the main criticisms that one can
a number of measures associated with it, including time to complete the module, make about the research is the panoply of notations that have been proposed, including
length of the module, and a metric such as the density of Boolean conditions in the those based on programming languages ; mathematics ; object-oriented technology ; gram-
program code . By having standard process templates, a developer is able to easily mars ; graphical notations similar to those used for requirements analysis ; notations used
compare the effect on, say, productivity, of a change in technology-for example, the to express concurrency ; artificial intelligence notations ; and project planning notations .
effect of a configuration management tool from project to project-since each project Indeed, this book contains an example of a fragment of notation on page 186 .
would effectively employ the same standard process template . A fragment of a graphical notation based on data flow is shown in Fig . 12 .1 . It shows
• To support project management and quality assurance . In Chapter 1 I described the the initial stages of the process of programming and testing a module . The circles repre-
fact that in developing a quality plan and project organization, a manager will consult sent standards which are to be used for a process, while the three-edged objects represent
a quality manual and select those aspects of the manual suitable for the project . In products of the process which could be reused by that process or another process . It is
carrying out this selection the project manager would consider quality factors, the important to point out that this is only a fragment of the module programming and
current business policy of the company, the risk inherent in the project, and the nature testing process . The arrows which lead to mid-air point at the remainder of the process .
of the customer . This tailoring can be immensely difficult if the processes which are
available to the manager-both developmental and quality assurance processes-are
expressed in a non-standard way in the quality manual and its associated documenta- Self Assessment Question 12 .3 Can you give two reasons why the no-
tion. The eventual aim of process modelling is to enable a project manager to consult tation shown in Fig 12 .1 is a good one for process modelling?
a database of process descriptions, tailor all or some of the processes to the particular
project that is started, and, eventually, join up the process descriptions in the same Solution It can be understood by all staff on a project, both technical
way as a jigsaw . The 'jigsaw' would form an overall description of the process architec- and managerial . It is also a notation to which CASE technology can be
ture of the system in terms of entities such as process details, data flows, documents, applied .
and decision points . This description can then be given to a project manager who
 1 68
SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION
PROCESS MODELLING AND PROCESS ASSESSMENT 169

One of the main arguments currently being fought out by researchers is how formal

0
the language for process modelling should be . On the one hand, there are proponents
SpecStan 1 .2 who, motivated by the need to develop project support environments and software tools,
say that the notation used to model processes should be as exact as a programming
language-indeed, should look like a programming language . On the other hand, there
Module specification
are researchers who point out that human beings require other qualities apart from
Check for
correctness formality in a notation which describes the tasks that they are expected to carry out on
a software project . They state that, for example, human beings are quite good at carrying
not correct out instructions which are ambiguous, but require expressiveness and comprehensibility
correct from a notation (Kellner, Curtis and Over, 1992) .
It is difficult to predict the way that process modelling notations will move . If this
Query with was the eighties I would say that they would progress towards becoming more formal.
designer T
Program the However, one of the lessons that we learnt in the eighties was that different users of
module a technology required different facilities from that technology . For process modelling,
the range of users of process notations is so large-ranging from business managers to
ProgStan 1 .3
programmers-that I suspect the process modelling notation which is totally formal will
Coded module
be a rarity. I suspect a more likely scenario is of a core process modelling notation which
might be formal but which will have a number of instantiations depending on the user
Generate who is employing the notation . For example, project managers may have an instantiation
test data of the notation which is graphical, programmers may see an instantiation which looks like
a programming language, and customers may see a sort of constrained natural language
version of the model .
TestStan 1 .1
Module test data

12 .4 INDIVIDUAL PROCESS IMPROVEMENT

Figure 12 .1 An example of a graphical process modelling notation As an example of how process modelling can be useful for the company which at least
.
has ISO 9000 certification comes from technical reviews . A technical review is a meeting
of staff who examine a particular document, usually a requirements specification, system
There are three aspects to a notation used for process modelling . First, it should describe
the processing which occurs design or program code, and then attempt to discover defects . Normally technical reviews
; for example, when a test procedure is applied it should are used as a validation technique ; in a company which is at least at a high level 3 within
detail the tasks that make up that process
particular process . Second, it should describe who carried out a the process maturity framework, the results of these reviews can be used to improve a
. This may be expressed as a single person, group of persons or, more number of development processes as well as technical reviews themselves .
likely, in terms of job function
. Third, it should give details of the data and information A company which does not have a well-defined process may have technical reviews
that is required and generated when a process is carried out
. For example, a process available in their quality system ; however, since they are not well-defined the amount of
description for programming a module should describe the process steps involved in variation between reviews will give rise to so many parameters that a statistical analysis
the programmer receiving a module specification, producing code and testing the code ;
standards for programming and module testing which attempted to answer questions such as : `Is our design notation at fault?' or `Are
; and the production of a coded module, particular parts of a design prone to errors?' would be difficult to answer without a
a test suite and a quality record which confirms that adequate testing has been carried
out. prohibitively large amount of generated data. However, if technical reviews are strictly
governed by process descriptions which describe factors such as who attends, maximum
A good process modelling notation should not only include these features, but should duration, the accompanying documentation that is to be used and the documentation
also provide facilities where each of the features can be isolated from the others
. For that is to be issued after a review, then there is a much greater chance that conventional
example, a manager who wants to know the flow of a document through a project-who
statistical analyses can be carried out-although it must be said that it is only companies
initiates the document, who reads it, who signs it off and where it is eventually stored-
that have generated a large amount of data, and regularly update a statistical database,
does not want the tool that he or she uses to browse through a process model database to
that are able to take full advantage of the process improvement opportunities offered by
display information such as step-by-step procedures which would clutter up the process
of finding out about data flow. process modelling .
A typical process improvement sequence for a very advanced company is :
 1 70 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION PROCESS MODELLING AND PROCESS ASSESSMENT 171

• The company has a strict design notation standard which all software engineers have 12 .6 PROCESS ASSESSMENT
to adhere to .
• Technical reviews are governed by a process template which can be instantiated for Software companies vary in their competence . They range from those companies which are
different reviews .
short-lived and who have virtually no forms of quality assurance to successful companies
• A review is held according to the process template . who have a quality system and have quality systems which are mature and are being
• Errors are detected in the design . regularly examined and improved . The last decade has seen a number of attempts to
• These errors are rectified by the designer and the chair of the review issues a defect try and develop schemes which attempt to give an idea of where a company is in both
report which details what errors occurred and where they occurred .
developmental and quality terms .
• The error data is entered into a database of past errors and correlations are made There are a number of reasons for having such schemes . First, customers are naturally
between the location of the errors and some metric which measures design complexity . interested in the capability of a software developer and an independently developed as-
It is discovered that the errors are highly correlated with the design metric . sessment of a software company's competence would be the main determinant of whether
• At the next revision of the quality system the design procedure is modified to insist to use a company for some software task . Second, many companies well understand that
that designs should not normally be developed which have a metric value higher than their software processes are not perfect and wish to know where they are in terms of some
some threshold . If a designer wishes to exceed this threshold he or she must gain the technical baseline and where improvement is needed . There have been a number of soft-
permission of the technical member of staff he or she reports to . ware assessment schemes. Virtually all are implemented by means of site visits, checklists
and questionnaires by some independent body . The first of the significant schemes was
developed by Watts Humphrey at the Software Engineering Institute in Carnegie-Mellon
12 .5 AN EXAMPLE OF PROCESS MODELLING IN ACTION
University.
Humphrey was an ex-senior staffer from IBM . He was tasked by the American De-
The special issue of the journal Information and Software Technology (Vol . 35 No partment of Defense (DOD) to produce some form of questionnaire which could be used
. 6/7
July 93) describes a case study from IBM of process modelling in action
. What is inter- to appraise their subcontractors . The DOD had had some very poor experiences with
esting about this case study is the fact that fairly low-tech ideas were used to carry out subcontractors during the seventies and early eighties ; this had led to large numbers of
process improvement based on the process model that was discovered and documented cancellations and of software being delivered which has not satisfied many of its critical
by the IBM staff. For example, for the most part, pencil and paper drawings were used
. requirements .
The Federal Systems Company in IBM develops software for government and state The DOD wanted some scheme whereby they could appraise software contractors
bodies in the USA . A team based in the company have used process modelling to try and eliminate poor developers before development started .
to integrate the processes that they use for software development . This activity arose The resulting questionnaire developed by Humphrey enabled the DOD to categorize
because of problems the company encountered with confusion about roles and responsi- software developers from level 1 to level 5 :
bilities, poor communication about the products of software development, the omission
and duplication of work, the limited spread of new technology and sluggish response to • Companies at level 1 are in anarchy . There may be a quality system or, more likely,
customer and business needs . a quality manual ; however, whether a project takes on standards, procedures and
The team led by a senior manager at IBM, Mike Dyer, used relatively low-level individual quality controls depends on the whim of project managers .
technology to document the various processes . The main notations they used were system • Companies at level 2 have more control over their projects . This is established by
flow diagrams and tables . Even with such low-level technology the modelling produced means of proper project planning, task monitoring, the proper management of sub-
major gains . One of the main areas they identified for improvement was reuse
. The contractors and decent configuration management . At this stage process modelling
modelling that was carried out identified a large scope for reuse over and above that starts to become a useful tool .
already practised at IBM . The team identified major opportunities for the reuse of parts • At level 3 a company has an organisation-wide process which can be tailored to
of requirements documents and also major chunks of testing documentation ; they even individual projects . The process model becomes an asset which can be refined through
managed to identify reuse potential in areas such as product training and field support . experience.
The second benefit, over and above those being sought, was that the ability to collect • The big change comes at level 4 when processes are instrumented and measurements
measurements on a software project had been greatly increased . IBM are very advanced are made . Typical measurements would involve product metrics and defect statistics .
in their use of defect data generated from technical reviews to improve software tasks At this level industrial statistical quality control can be employed in order to ascertain
such as programming . However, a number of activities generated data on defects which the effect of changes to the process model . For example, a developer may use facilities
were ignored . The process modelling that occurred led the team to discover that defect
provided in a level 4 quality system to examine whether technical reviews of code are
data on activities such as system engineering and integration testing could be treated more productive than module testing.
in the same way as existing defect data and provide a fuller view of the efficiency of
• Level 5 companies use the data generated by metrics in order to continually drive
individual projects. improvement programmes .
 PROCESS MODELLING AND PROCESS ASSESSMENT 173
1 72 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

Hinsley and Bennet (1993) . Humphrey has also described a case study which applied
Self Assessment Question 12 .4 At what level would a company be who . Almost certainly the best tutorial
his ideas in Snyder, Humphrey and Willis (1991)
had a quality system which applied a quality plan to each project, but . Details of a large
introduction to process modelling is Kellner, Curtis and Over (1992)
where there was no auditing of projects to ensure that they adhered to the number of process modelling notations can be found in this article .
quality plan?

Solution The company would be at level 1 . PROBLEMS

.1 Develop an outline questionnaire that can be used to judge whether the configuration management
12
It is at levels 4 and 5 where process modelling comes into its own, and is most useful system used by a company is adequate.
because of its link with process improvement . .2 Develop an outline questionnaire that can be used to judge whether the module testing standards
12
This initiative was followed by a number of others . There are in existence about used by a company are adequate .

ten process assessment programmes . The best known British one was developed by the 12 .3 How would you organize an assessment visit to a software company?
Institute of Software Engineering (now MARI Northern Ireland) . While such programmes
are very useful, many companies can carry out relatively quick health checks by asking
a selection of questions .
Some typical questions are shown below ; a full selection, cross-referenced to the
chapters of this book, is reproduced as an Appendix:

• Does your system documentation have built-in traceability?

• What efforts do you take to validate a design?
• Do you always use the same quality manual, or do you orient your quality plan to
each project by extracting or removing elements from the quality manual?
• How do you evaluate the effectiveness of your quality system?
• What records exist which demonstrate that a programmer has carried out a test?
• Does your quality manual cover the risk analysis that a project manager often has to
carry out at the beginning of a project?

12 .7 SUMMARY

Process modelling is a term used to define the process of describing the individual
tasks that make up a software project in some standardized way . Process models have a
number of uses : to support human understanding and communication ; to support pro-
cess improvement ; to support project management and quality assurance ; to provide
guidance to staff carrying out a software process ; and to aid automated execution sup-
port . Process assessment is concerned with making some judgement-usually carried out
independently-of a company's capability to develop software .

12 .8 FURTHER READING

Process modelling is a topic that has interested researchers since the early seventies .
However, the work which caused a huge flurry of interest in the subject is Humphrey
(1989) . This is an excellent book about managing the software development process . At
the heart of the book is the 1 to 5 categorization that Humphrey developed for the DOD .
Good descriptions of Humphrey's ideas in practice can be found in Gilchrist (1992) and
 STANDARDS AND PROCEDURES 175

standards and procedures tend to require a major effort to adapt them to a company's

13
circumstances-almost as much effort as writing the standards and procedures anew .
The second drawback-and probably the most important-is that the adaptation of
existing standards and procedures tends to be something of a solitary exercise, with
little interaction between the consultant and staff in the company : since the standards
STANDARDS AND PROCEDURES and procedures exist there is little incentive for the consultant to involve staff in their
construction .
This is a major mistake : it is an important principle that when a quality system is
developed, the staff who have to operate under that system must feel that they have had
a major say in its construction .

13 .2 THE STARTING POINT

The starting point in writing a standard or a procedure for a task is to question the
staff involved in carrying it out . The first step is to question the staff, either directly or
via a questionnaire, about the problems they have found with a particular task . Typical
questions that you should ask are :

• Do you consider part of the task to be too time-consuming?

AIMS • Where do you find yourself making errors?
• What do you enjoy doing in the task?
• To describe the nature of standards and procedures . • What do you not enjoy doing in the task?
• To give examples of standards and procedures . In the past, has an error committed in this task led to major problems in other tasks?
•
• To show how standards and procedures can be formally developed .
It is also worth while questioning project managers in order to see how poor performance
of that task has affected their work . Typical questions to ask are :
15
ƒ$
.1 INTRODUCTION
• Has the poor performance of a task badly affected any of your projects?
The aim of this chapter is practical : it is to show how standards and procedures are Is enough documentation issued from a task to allow you to monitor its effectiveness?
•
developed . Are you able to effectively monitor the completion of a task?
•
• If a task is a quality control, is there adequate evidence that the control has checked
the quality factor that it was meant to check?
Self Assessment Question 13 .1 Can you remember what the distinction
between a standard and a procedure is? It was detailed in Chapter 1 . It is vitally important that the process of questioning is also applied to staff carrying
out maintenance, as it is this category of staff who will provide the majority of material
Solution A standard specifies what a project document looks like, while a which enables you to develop a standard . Staff who carry out a task from new do not
procedure is a description of how a particular task is to be carried out . seem to be too concerned with standards and procedures ; however, it is the staff who
have to read specifications and program code for understanding, modify the documents
and program code, and chase errors, who seem keenest on standards and procedures, and
Something like 80 per cent to 90 per cent of the effort of implementing a quality system seem the most ready to answer questions about problems with software development .
is spent on writing documents which specify how certain tasks are to be carried out In order to demonstrate how standards and procedures are developed I shall look at
(procedures), and how information on documents is to be presented (standards) . One
two tasks : unit or module programming, together with its associated testing activities,
of the options taken by software developers when constructing a quality system is to and carrying out a design review .
hire a consultant in quality assurance to work on the task of writing standards and
procedures ; normally, what this consultant would do is to take a pre-written quality
manual, and adapt it to the company for whom he or she temporarily works . There
are a number of drawbacks in this approach . The first is that even general pre-written

174
176 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION STANDARDS AND PROCEDURES 177

13 .3 UNIT PRO RAMMIN 8 . I wish I understood what the program variables mean .
9 . I am often asked to apply some changes to a module . It can take me ages to find
This task involves the programmer receiving a specification of a module, programming the file which contains the module and the name of the directory in which the file is
that module, and then testing it . This is often one of the most poorly performed tasks held .
on a software project-mainly because of inadequate standards . In order to describe the 10 . I have to spend a large amount of time finding out what progress has been made in
process of developing the standards and procedures for this task it is worth looking at programming a system . We instigated weekly meetings during which the program-
some of the answers that might be obtained when staff are questioned about problems mers would report progress . However, I still feel it could be done better : meetings
with the task : are valuable and I don't like cluttering them up with routine reporting .
. I have major
11 . I often receive the specification of a module which is poorly written
1 I enjoy programming and I also enjoy testing the modules that I produce, but what I problems in finding someone who can explain to me what the specification means .
do not enjoy is the process of retesting . Often, a module is returned to me a number The design team are always so busy .
. In
of weeks later for reprogramming . There are a number of reasons for this : sometimes 12 . I don't know whether it is me, but I have real problems in debugging a module
the designer has made an error in the system design, more often there has been the end I have to insert a lot of debugging code and monitor the values of variables
a requirements change from a customer, and occasionally I have made an error in during execution . Can a quality system help me in this?
programming. I really do not enjoy trying to think up my original test data again .
2 . Sometimes I am asked to make a change to an existing module . I usually encounter
two problems with this . The first is that it is often quite difficult to understand what Self Assessment Question 13 .2 What facility would a quality system
the module is aiming to do . The second follows on from the first, in that when I have offer to alleviate the problems detailed in the first point above?
major difficulties in understanding the functionality of a module I have to find the
programmer who wrote the module in order to ask him or her about it . This often Solution It would insist that all test data was stored and a proper conven-
means I have to spend a lot of time looking through past project files . tion was used to name the files containing this data .
3 . Many of the modules which I have to maintain are littered with variables which act
as control flags . These pass information from one part of the module to another part .
Often these flags just have names such as fagl, flag2 and flag3, with little indication
of what they do . Moreover, they seem to contain so much information that when I
change one part of a module which contains references to these flags, I have to change Self Assessment Question 13 .3 What facility would a quality system
other parts which reference the same flags . offer to alleviate the problems detailed in the fourth point above?
4 . I have major problems understanding the program code of a module . Some of our
programmers nest their control structures to a horrendous depth . I once had to Solution It would specify as part of a programming standard that pro-
modify a program which had 16 loops and if statements . By the time I got inside grammers were not allowed to nest control structures to more than a cer-
the tenth control structure I was having great difficulty understanding what was tain figure .
happening in the module .
5 . In the past we have had some major problems with programmers who have inad-
equately tested the modules that we have given them to program . In one case, a
programmer felt so arrogant that all he did was to clean-compile a module and pass
it on to the system testers . We have found that rectifying errors during system test- Self Assessment Question 13 .4 What facility would a quality system
ing, which should have been detected during programming, is immensely expensive . offer to alleviate the problems detailed in the eighth point above?
6 . When I test a module I have to construct software which turns the module into an
executable program ; feeds test data to the program ; calls the module ; and displays Solution It would provide directions to the programmer for variable nam-
the results of the module execution . This extra coding is relatively easy to write, but ing in the programming standard .
it is a repetitive task which I think could be automated .
7 . One of the problems I have to face, particularly on the real-time systems that I
maintain, is that of coding tricks where a programmer has implemented some strange The first remark is quite a common one which emanates from staff who are involved in
piece of coding, either to increase response time or to save memory . Although this a wide variety of activities : it reflects the difficulty they have in retrieving a document,
trick is usually very clever, it is terribly difficult for me to understand it . It wouldn't source code or data which they believed was in a final state . Many quality systems do
be so bad if the programmer was readily available ; however, he or she has often left not provide direction to staff about the storage of these various entities .
the company.
 178 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION STANDARDS AND PROCEDURES 179

In this case, a programmer has not been instructed to store test data and test out- This is an example of where a grumble about a problem during programming has
comes in a file . Also, the names of these files have not been specified . The solution to ramifications for the quality assurance of another activity : in this case, the activity of
this problem is to include in the procedure an instruction to staff to create files to hold design . While it is obviously a good thing to insist that programmers use meaningful
the test data, test output and, if a test harness is used, a file to hold the test harness . In identifiers, it is also important that, as part of the design standards, staff should always
addition, the programmer should be instructed to place the files in the directory which be instructed to develop designs which only contain modules that implement one function .
has been assigned to the project in which the modules have been developed . Moreover, The quality control which would ensure this would be the signing off of a design by the
the programmer should be told to name the files in such a way that the module name can chair of a design review which, as part of its checklist, contains a directive banning
be traced from the file name and vice versa, for example by using the first n characters multi-functional modules .
of the module, where n is sufficiently large that duplication of file names will not occur . The fourth point, concerning the gross nesting of control structures, is connected
Also, the test data, test outcome and test harness files should each be distinguished in with the previous point about flags . It is obviously very difficult to understand a module
some way, for example by using the extension tda for test data, to for test outcomes, and which contains a large degree of nesting . However, the programmer responsible for de-
hr for the test harness . If these instructions were followed, then hours would be saved veloping the module could have been given a module specification which is too big, and
when a change has to be applied to an existing module . iven that this is a common which attempts to implement a number of functions . The programming standard should
activity on a software project it can lead to a large amount of resources being saved from obviously include directions about the maximum level of nesting ; however, it should also
this measure alone . contain directions about what to do if the programmer finds that he or she is unable to
The second remark is also a fairly common cri-de-coeur, which is not just associated write a module which is smaller in terms of nesting . This would involve reporting the
with program code, but with designs and the functional specification part of the require- fact to a senior member of staff .
ments specification . The solution to the problem is not easy : at a trivial level it requires Problem five-inadequate testing-is also a common one found in the software indus-
the function of the module to be embedded in the module as a comment . The major try . You will remember that in the first chapter I stated that the further into a software
problem here is that it is all too easy for staff responsible for specifying the functional- project an error remains undetected, the larger the amount of resources that are required
ity of a module-usually designers-to write a poor specification . The only solution to to rectify the error . It is vitally important that modules are tested thoroughly and that
the problem is to hold a design review, and for the checklist for the design review to evidence is presented that this has been done . There are a number of solutions to this
include a check that the functions of modules whose design is presented at the review problem : the first is to include, in the procedure governing module testing, an instruction
are understandable, unambiguous and can be interpreted in software terms . to document each test case, and provide a reason why each case was chosen . This, at
The sign-off of the parts of the system design considered in the review would be least, provides evidence that the programmer has thought up the test data . It also has
the quality control . It is also important that if functional descriptions of a module are the spin-off that it actually forces the programmer to think about the reasons why the
passed to a programmer, say when a design review has not taken into account part of test data was chosen-once someone starts thinking about a task, inevitably that task
the functional description, then there should be a clearly defined way of communicating will be more efficiently carried out .
any problems in the functional description . As you will see in the following description, Confirmation that the tests have been carried out can be shown in a number of
this means that the programmer should consult the member of staff to whom he or she ways . The simplest is for the programmer to provide a printout or a screen dump which
reports . The other point made concerns who to ask if the functionality of a module is not shows the results of each test . A more expensive method would be to use a dynamic
clear . This is relatively trivial ; however, it is surprising how much time that a programmer analyser. This is a tool which instruments a module and produces a report on which
can spend chasing up staff who he or she believes was the author of the module whose parts of a module a test has executed . The more sophisticated dynamic analysers also
functionality is difficult to discern . The solution to this problem is straightforward : it provide summary information for a number of tests ; for example, they often produce
is to have a standard for module documentation which insists that the designer of the histograms which show the frequency with which statements or branches are executed
module is included as a comment inside the module . and also identify any which remain unexecuted . The quality system could insist that the
The third remark concerns variables which have poorly chosen names . The quality report from this tool is bound in with the remainder of the module documentation .
system should insist that the names are well chosen and give examples . However, this Point six concerns software used to support testing . It is typical of something which
remark conceals something much more serious than misnamed identifiers . Very often often comes up during the process of asking staff to comment on the way in which they
when a module contains flags, that module is carrying out a number of functions, and currently produce software . It is not directly relevant to quality assurance ; however, a
the flags are used to communicate and pass information between these functions . A well procedure could be of help to programmers who find the development of such a harness
designed system will contain only modules which carry out one function and communicate software time-consuming . As part of the procedure governing module programming, staff
with other modules by means of small interfaces . Such systems contain modules which are could be told to use a pre-written piece of code which carries out many of the functions of
easy to read and understand in isolation, test in isolation, and integrate . The existence a test harness . Obviously, some extra coding would be required in order to cater for the
of modules with flags often indicates multi-functionality . These modules are horrendous input and output of values to and from a module but, nevertheless, there is still quite a
to maintain, since a change to a function in the module often means a change to one of degree of common coding which occurs and, hence, considerable savings can be achieved .
the flags, which further necessitates a change to other parts of the module .
 180 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION STANDARDS AND PROCEDURES 181

Point seven concerns devious coding tricks, and was obviously made by a maintenance The answers to these questions for module coding are fairly straightforward . The input
programmer . The effect of coding tricks can be totally eliminated by having a decent is the module specification taken from the system design . Another optional input might
programming standard which, for example, banned low-level coding . Probably a more be a project-specific standard for testing the module . For example, the normal standard
realistic standard would be to insist that if a programmer used a piece of tricky code, might insist that testing occurred with 100 per cent of all statements executed, and with
then it should be flagged and documented by means of a comment . 85 per cent of the branches executed, but the software that is being developed is so critical
Points eight and nine have already been covered . Point eight could be handled by that the standard requires modifying so that 90 per cent of all branches are executed .
means of a programming standard which gave examples of what meaningful variables The output is the coded module, which is then passed to the staff who are responsible
were . Point nine can be covered by means of standard naming conventions for files for integration .
.
Point ten was obviously made by a project manager who has problems tracking The main report that is generated to help project management is some form of
project progress . The answer to this problem is relatively simple : ensure that an item of document which indicates that programming of the module has been completed and
documentation is filled in when the programming of a module is complete . This docu- signed off. Another form of documentation which should be produced when problems
mentation could be processed by a clerical assistant, entered on a spreadsheet, together occur in programming and testing is some form of exception report . This explains why
with the estimated times of completion of modules, and displayed in an easily assimilable work on the module has ceased and what is happening about restarting the process .
form for the project manager . Normally, every developmental task is associated with a quality control, and module
Point eleven-the problems encountered with poorly written specifications-can be programming is no exception.
handled by having an explicit procedure whereby the programmer informs the member The main quality control will be the insistence that the tests which have been applied
of staff to whom he or she reports of a problem with a module . That member of staff will have been documented and that evidence that the tests have been carried out correctly
usually have sufficient seniority to make the design team sit up and take notice when the has been produced . This is normally found in the unit folder, and will be signed off by
problem is discovered . the senior member of staff responsible for the programmer who produced the module .
The final point about debugging can be covered by having an explicit defensive The question concerning archival material is usually easy to answer . In the case of
programming standard . Defensive programming is the term given to the insertion of module programming, the source code of the module should be stored together with its
debugging code into a module before the module is tested . A defensive programming object code, test data and test outcomes . It is also a good idea to store the harness which
standard will insist that the programmer insert this debugging code in places such as was used to test the module .
on entry to a module, after a module has been called, and after a particularly complex Question six, relating to module programming, is easy to answer : that there is no
calculation . intermediate checkpoint which requires permission to proceed . The answer to the seventh
Once the points made by staff in questionnaires have been processed, a standard and question will be virtually the same, irrespective of the task : that if the input to the task
procedure for a task can be developed . However, before actually doing this, there are a is defective, for example the module specification is ambiguous, then whoever carries out
number of questions that need to be asked in order to ensure that nothing is missed . the task has to report the problems that have been found to the senior member of staff
These questions are common to each of the activities on the software project, and in -to whom he or she reports .
many ways, they reflect quite a large number of the concerns addressed by the questions iven this simple algorithm of asking staff about the problems they encounter, and
discussed above . They are reproduced below : asking a few general questions, the process of producing a standard and procedure can be
carried out . It is worth stressing here that whenever you write a standard or a procedure
1 . What are the inputs to the task that is to be carried out? For example, the main it is important to try and minimize the amount of documentation that staff have to fill in .
input to the task of system design is the requirements specification . For example, try and use tools such as word processors, which provide glossary facilities .
2 . What are the outputs produced by the task that is to be carried out? The standard and procedure for module programming is shown below . It is important to
3 . What reports should be generated from the task which will help project management point out that the procedure is generic in that it covers any language X :
to monitor and control the task and the project?
4 . Is the task associated with a quality control? If so, how should evidence that the Procedure P17 . V3 (Module coding and testing)
quality control has been successfully carried out be documented? Who should sign Task
off the quality control? This procedure covers the coding and testing of modules written in the X programming language .
5 . What archival material should be produced for staff who are to carry out mainte- Tools used
nance? Borland compiler for language X . The dynamic analyser tool ANALYSE.
6 . Is there any checkpoint in the task which requires permission to proceed before the Input
task is completed? The input to this task will be a module specification . This will contain a list of parameters and
7 . What should be done when the input to a task falls below expected quality? global variables, together with the function of the module expressed in natural language . Further
details of this document can be found in Standard S13 .V2 (Module specifications) . A minor input to
the process may be a project-specific directive which overrides the current test completion criteria .
 182
SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION
STANDARDS AND PROCEDURES 183

Task description
The unit folder will contain a number of sections . The template for this document can be found
The programmer will take the module specification and produce program code in the X language
in Tempt :unitf.doc. This template should be copied to the programmer's directory and the sections
which satisfies the specification
. If the specification is deficient then the programmer will notify edited to include the information generated by module programming and testing .
whoever he or she reports to about the problem
. A simple form will be used for this . Details of this
form can be found in Standard S55 .V4 (Module incident form) Section 1 . Module specification
. Until the problems that have been
noted in the form have been cleared up the programmer will suspend work on the module This is the module specification that is provided by the designer . Normally this will be provided as
.
If the module specification is in a form which is adequate for programming, then the program a file so that it can be copied directly into the unit folder .
code is produced . This code must adhere to the programming standard used by the company for
Section 2 Source code
programming language X . This can be found in Standard S43
.V2 (Programming language X coding This section of the unit folder will contain the source code of the module that has been programmed .
standard)
. If in coding the module the programmer discovers that it is impossible to keep to the The display should be obtained by running the source code through the prettyprinter tool FORMAT,
standard, this fact must be reported to the manager the programmer reports to and directing the output to a file which can then be edited into the unit folder .
. That manager
may suggest a way in which the module could be coded which respects the coding standard, or may
Section 3 Test data
decide that the module is so functionally large that it has to be queried with the designer
. In the This section should contain the test cases that were used to test the system . You should have stored
latter case a error incident form has to be filled in by the programmer ; details of this form can be
found in Standard S55 .V4 (Error incident form) . these test cases on a file, so all that is required is for you to edit the file and annotate each test case
with TEST CASE N enclosed within square brackets.
Once the module is coded and clean-compiled it should be shown to a colleague, and the
After each test case you should describe why you chose it, for example that the test case
function of the module explained in terms of its source code
. If a substantial number of errors are
represents a boundary condition, or it was devised to bring the test coverage up to 100 per cent
discovered by your colleague, then another meeting should be scheduled after the errors have been
rectified and the module clean-compiled again statement coverage . Your description should again be enclosed within square brackets . An example
. Once no more errors have been discovered by this
static analysis, the colleague should sign off the module as being correct of a test case is shown below :
. The unit folder will contain
space for the signature to be inserted, see Standard S44 .V1 (Unit folder) . [TEST CASE 1] 33, 45, 22, 12 [This test case checks whether the first name on the database is
The next task is for the programmer to test the module recognized as being valid]
. A test harness will be built which
turns the module into an executable program
. It is then instrumented using the ANALYSE tool . The Once each case has been annotated, the resulting file should be edited into the unit folder . The
programmer will then generate test data which will test the functions of the module
. The various summary file of test coverage obtained by running the ANALYSE tool should also be edited into
testing strategies that can be used can all be found in the company's manual of good developmental
this section of the unit folder. Once this file has been completed, it should be stored in the project
practice . All the test outputs should be channelled to a file
. This file should be retained as it will library. It should be printed off and signed by the manager to whom the programmer who developed
eventually be stored in the project library .
the module reports and the programmer who statically checked it . The file should be given the
The main aim of testing is to ensure that a module works correctly in terms of the functional
name of the module followed by the extension ufo . If the name of the module contains more than
description provided by the designer
. However, the programmer should also provide proof that the 12 characters it should be truncated to 12 characters .
tests which have been carried out have executed all the module statements
. The ANALYSE tool
will produce a histogram which will show the statements that have been executed and the frequency
of execution of each statement . If any statement has not been executed, then more test data should The associated programming standard for language X might be as shown below :
be generated which achieves 100 per cent coverage
. Once this coverage has been achieved, the
programmer should complete the test data section of the unit folder
. Here, he or she should describe The start of the module should be annotated with bureaucratic information . This consists of the
why each test case was chosen in terms of the test strategies described in the company manual of
name and extension number of the programmer who developed the module, the name and extension
practice
. The histogram which is produced by the ANALYSE tool should then be placed in the unit number of the programmer who applied the last major change to the module, a list of the modules
folder ; finally, a printout of the source code of the module should be placed in the unit folder
. the module calls, a list of the global variables which the module references, and a list of the modules
Once these tasks have been completed the member of staff who the programmer reports to
which call the module .
signs off and confirms that the unit folder is correct . What this implies is that the member of staff
A module should also contain a comment which describes what it does . This can be copied
confirms that all the standards have been adhered to, and that adequate testing of the module has directly from the module specification produced by the designer of the system . This narrative
taken place .
should follow the bureaucratic information described previously . Normally, for the X programming
language, the following restrictions will hold for a module :
Outputs
There are a number of outputs from this task 1 . No goto statements are to be used .
: the source and object code of the module ; the test
data ; the test outcomes ; and the test harness 2 . No more than four boolean operators such as and and or should be used in a condition .
. These should be stored in separate files . Each file
should consist of the name of the module together with an extension . In the unlikely event of the
3 . On no account should more than two not operators be used in a condition . If possible, restrict
module name being more than 12 letters then it should be abbreviated to 12 letters . The extensions
boolean conditions to one not operator .
used are tda for test data, tou
for test outcomes, sor for source code, obj for object code, and tha
for the test harness 4 . Control structures such as if and repeat should not be nested to a depth greater than 8 .
. Thus, the test data for the module update will be stored in the file update.tda .
5 . Identifiers should be meaningful at all times . No identifiers such as ID009 should be written
unless, of course, ID009 has a specific meaning in terms of the application .
The associated standards for the unit folder and for the programming language X are 6 . Modules should be no bigger than 100 lines of code after they have been processed by the
shown below: FORMAT tool .

7. Variable initializations should be written at the head of a module, the only exceptions being
variables used to count loops .
 184 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION STANDARDS AND PROCEDURES 185

8 . If non-array variables in the program have a definite relationship with each other at a point in 9 . I am aware that reviews are rather a tricky thing to chair . Therefore, I would really
the module, for example the sum of variables a and b is greater than variable c, then defensive like data on the number of errors that a review threw up and the number of errors
programming code should be inserted which checks for this and displays an error message if the
condition is not true . that were missed . If this data is consistent-for example, reviews chaired by one
9. Arithmetic expressions should contain no more than 10 operators such as * or +
member of staff were consistently letting major errors go undetected-then I can do
. If they do, split something about it, such as sending the individual involved on a training course .
them into subexpressions and assign them to variables which have meaningful identifiers .
10 . Use only single letter variables for loop counters ; do not use them in any other context
10 . Sometimes my work is reviewed and problems discovered . Usually when a review
. takes place I am in the middle of another task, and when I return to rectifying the
If the programmer feels that these restrictions cannot be adhered to, he or she should communicate
the fact to the member of staff to whom he or she reports . errors I have forgotten what some of them were . I make notes, but because of the
nature of reviews they are always hurriedly done .
11 . When I started work for the company, reviews were a rude awakening . Here I was, a
13 .4 DESI N REVIEWS young graduate, thrust into meetings where two or three other members of staff tore
my work to pieces . I felt terribly dispirited, and almost left the company .
To conclude this chapter it is worth looking at another activity and writing a procedure
for it . This time I will use a slightly more rigorous notation for the procedure These are a selection of problems which I almost invariably encounter in companies which
one
modelled on a detailed design notation which has facilities for control flow specification . are trying to make reviews a success, though they are by no means all the problems I
encounter . It is worth considering them in order .
The first problem is quite a common one ; time and time again staff tell me that
Self Assessment Question 13 .5 This is more of the nature of an exercise whoever submitted an item for review has done it in a hurry, and hence it is incomplete
than a self assessment question . However, it is worth doing at this point in or too abstract . The solution to this problem is to direct whoever chairs a review to read
the text . Cover up the remainder of this page and write down the problems the item before deciding whether a review can take place ; if the item is not good enough,
which you think might afflict a design review . then it is returned to the originator for rework .
The second is the most common one that I find . A quality system's scope for im-
Solution The majority of the problems are detailed below . proving the chairmanship qualities of staff is limited . There are a number of points about
chairmanship that can be written into the company manual of good practices, such as
instructions to keep to an agenda, but the scope is limited . Where a quality system can
Some of the problems with design reviews are described below : help is in detecting whether a review-or, more realistically, a series of reviews-has not
gone well because of poor chairmanship . A proper error-reporting procedure, together
1 . Sometimes we are given a design to review which is nowhere near to being ready for with the minuting of the problems discovered during a review, will enable a project
review . 1Ve just waste our time looking at an incomplete document . manager to home in on a poorly executed review .
2 . The reviews which I attend are often appallingly chaired . WWWe seem to spend hours The third problem is one of poor planning . This can be solved by writing into the
just going over `pet' points that are brought up by the more loquacious members o project planning standards an instruction that all reviews should be explicitly planned
the review team. for and, furthermore, that staff should be allocated at least two hours in order to prepare
3 . Managers often seem to invite staff to a review at a moment's notice themselves for a review .
. I've been to
one review where all the staff, except myself, have only had an hour's notice that the The fourth problem is trivial, but it can give rise to a large amount of wasted time .
review was to take place . Consequently, they had not carried out enough preparation . The solution is to write an instruction into the procedure for holding the review : that
4 . I have turned up at the venue for a review which is occupied by another meeting . the chair should book a meeting room well in advance of the review taking place .
WW'e spent a considerable amount of time looking for an empty room which had not Again, a quality system cannot prevent the poor chairmanship which the fifth prob-
already been booked by another meeting . lem indicates . Obviously, a review is a much better medium for detecting errors than
5 . Some of the reviews that I have attended have degenerated into informal design solving problems, and a good chair should clamp down on any attempt to carry out
meetings where, in response to one problem, the whole design review team just redesign . Unfortunately, a quality system can only provide advice on good chairmanship
spent the time redesigning the system in order to eradicate the error . via a manual of good practice . Where a quality system is of help is in detecting the
6 . When a new member of staff first attends a review, he or she often has no idea what consequences of poor chairmanship . This is discussed with respect to problem nine .
a review is meant to achieve and what we should be looking for in a review . The sixth problem is one which not only affects reviews, but also almost every activity
7. Sometimes I do not receive material to review sufficiently in advance to be able to that a software company carries out . Although many companies have induction courses,
adequately prepare for a review . many are of insufficient duration to show new staff exactly what is required of them for a
8 . It would be nice for me to be able to discover progress on a project, in particular particular task . There is, in fact, a good argument against induction courses which only
how close we are to completing a system design . try to deal with minor detail . A quality system can generally help, as it is in effect a
 186 SOFTWARE QUALITY ASSURANCE--A STUDENT INTRODUCTION STANDARDS AND PROCEDURES 187

collection of detailed descriptions of the tasks which staff members have to carry out and FOR each problem DO
good practice governing these tasks . In the case of a design review, the design review The secretary notes down the reason [REV REPORT FORM]
checklist and the design review procedure would be the medium which a member of staff for the problem
would use to understand the nature of the review process . ENDFOR
The seventh problem is again a minor planning deficiency ; it can be remedied by CASE problems OF
including a directive in the review procedure to circulate review material at least two No problems : Review successful
days in advance of a review meeting . Chair signs off the design
The eighth problem is obviously voiced by a project manager . It calls for some form Small problems : Review mainly successful
of documentation to be issued after a review which gives some indication of its outcome . Designer modifies design
The ninth problem is obviously articulated by a project manager who is concerned Chair signs off design
with the quality of the processes which he manages . It is a call for documentation which Major problems : Designer modifies design
is issued after a review to give some idea of the quantity and seriousness of the errors ENDCASE
which were detected by the review . It is also a call for something much bigger : a proper UNTIL chair signs off design
defect reporting system which provides data on errors discovered throughout the soft- Review report form signed off [REV REPORT FORM]
ware project . Documentation which provides data on the number and incidence of errors Design added to project library
discovered by design reviews would be just one component of this system .
The tenth problem is again a common one . It can easily be solved by including a This looks very much like some program coding . However, it does convey the essence
directive in the design review procedure for a member of the review team to make detailed of the design review process . Obviously some things are missing ; for example, it does
notes on the errors that were discovered . This documentation can be read and checked not contain a description of best practice for reviewing . Nevertheless, it is an excellent
by the chair of the review, and signed off if he or she is satisfied that enough detail is medium for describing the step-by-step process whereby designs are produced .
included for the staff who are to rectify the errors . A design review will also need a standard for the documentation which is issued .
The final problem can be addressed by including material in design review guidelines Here, what is required is a document which tells the reader what was reviewed, who did
about the proper way to conduct a review ; it also requires some material to be inserted in the reviewing, and what was the result . A standard is shown below:
the induction material given to new technical staff . The first requirement is a procedure
for running a design review . It is written in a specific notation which resembles a detailed Design review documentation
design notation or programming language, with the subtasks which make up the task of The only piece of documentation issued from a design review is the design review minutes . This
running a design review specified in terms of control structures . The capitalized references will consist of a number of sections .
within brackets are to standards, CHECKLIST DES2 is the standard for the checklist The header
used for design reviews, while REV REPORT FORM is the standard for the report form This will contain bureaucratic information : the staff who attended, who chaired the meeting, who
which is issued when the review is completed : took minutes, the item that was reviewed, and the person or persons who produced the item . The
header will also consist of a unique identifier which distinguishes this review document from any
DESI N REVIEW PROCEDURE other document . Normally, a design review identifier is made up of the project name followed by a
letter d, followed by an integer one greater than the number of the last review .
REPEAT
Designer informs chair of review that design is ready for review The list of problems
This consists of a sequential list of problems which are each uniquely identified by means of the
REPEAT
design review identifier followed by an integer . The integer starts at one for the first problem, and
Chair reads design and checks for readiness is incremented by one for each subsequent problem . Each problem should be given a seriousness
IF the design is not ready THEN rating, starting at 1 (trivial) to 4 (serious) .
It is rejected Review outcome
ENDIF This section of the review minutes contains a description of the outcome of the review . This is
UNTIL design is ready for review filled in by the review chair . There are essentially three outcomes to a design review . The first is
Chair books meeting room that no problems were discovered ; this means that the review process is complete for the item being
processed . The second is that a relatively small number of not very serious problems were discovered ;
Chair announces time for review in this case the item being reviewed is reworked, then shown to the review chair and the review
Chair selects review participants signed off. The third outcome-and the most serious-is that major problems were discovered ; if
Chair generates review material this occurs the item has to be modified and completely re-reviewed .
Participants review material and make notes . [CHECKLIST DES2] Sign-off
Participants meet This part of the form just contains space for the chair to sign off the reviewed item if the review
Chairman reads the review material in textual order was successful, i .e. if the first two outcomes above occurred .
 188 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION STANDARDS AND PROCEDURES 189

The guidelines for a review are shown below : 13 .6 FURTHER READIN

Review guidelines There has been very little written on the development of standards and procedures .
We have found that reviews are a highly effective way of validating a project document or source However, two good documentation books which have helped me a great deal are Bell
code . In general, the earlier a review occurs the more effective it is . The following guidelines are not
and Evans (1989) and Williams and Beason (1990) . Strunk and White (1979) is a good
prescriptive, but our experience has been that if they are followed, then the vast majority of errors
pocket-sized introduction to writing clearly, and both owers (1988) and Jenkins (1992)
in an item are picked up well before expensive system testing starts . We would normally expect a
project manager to follow them for the vast majority of the time . are excellent reference books .
1 . A review should normally involve no more than five members of staff and no fewer than three,
including the chair of the review .

2 . Care should be taken in selecting staff for a review . The member of staff who produced the item
PROBLEMS
being reviewed should always attend, as should a member of staff from another project in order to . Such
.1 Write down a standard for a document which would be used to propose changes to a system
provide an independent viewpoint . A staff member who carries out the same task as the developer 13
a document would be processed by a configuration management system .
of the item being reviewed should be invited, as well as other members, depending on the type of
.2 Write down a procedure for costing a project using the conventional work breakdown structures
review. For example, the customer should be invited to attend requirements specification reviews, 13
staff responsible for system testing should also be invited to a requirements specification review, discussed on page 55 .
and an analyst who was concerned with the development of the requirements specification should .3 If you were going to develop a procedure and standard for system testing what sort of replies do
13
be invited to a design review . you think you might get after asking staff charged with this activity about the problems they encounter?
Consideration should be given to inviting new members of staff who have not experienced re- .4 Write down a procedure which might be used by staff charged with the improvement of a quality
13
views before . It is often a good idea to invite them as observers so that when, and if, their material
system .
is reviewed, the process of reviewing will hold no surprises for them . It is important that senior
members of staff do not attend reviews . Staff whose material is being dissected often feel that if
a senior member of staff is present, their performance is being evaluated . This often means that
they become very defensive and their attitude hinders the validation process . The company has
well-defined procedures for evaluating staff and a review is really not one of them .

3 . The chair of a review shall, at all times, ensure that the review concentrates on validation .
Discussion on possible solutions to a problem, similar problems which were encountered on other
projects, gossip, and over-concentration on one error, should be firmly discouraged .
4 . We have found that if reviews are working well, the participants become very tired and their work
becomes less effective after about two hours . The amount of work scheduled for a review should
therefore not result in meetings which last more than two hours .

5 . Having one's work reviewed can be a very painful experience, and the chair of the meeting must
ensure that the pain is reduced to a minimum . For example, if the chair feels that one member of
staff's validation contains some personal criticism of the originator of the document being reviewed,
then steps must be taken after the review to remind the member of staff that a review is meant to
be an appraisal of a document or source code, not a criticism of the person who produced it .

6 . It is important that the minutes produced after a review meeting are legible and understandable .
The chair should check them before they are issued .

13 .5 SUMMARY

The development of standards and procedures can consume up to 90 per cent of the
resources devoted to the implementation of a new quality system . The main point stressed
in this chapter is that the development of such standards and procedures should be driven
by the needs and problems of technical and managerial staff on projects, and that the
first step in writing new standards and procedures is to ask such staff what they feel is
required .
 iso 9001 191

14 • ISO 9004-2 Quality Management and Quality System Elements-Part 2 . This docu-
ment provides guidelines for the servicing of software facilities such as user support .

The requirements are grouped under 20 headings :

ISO 9001 Management responsibility Inspection, measuring and test equipment

Quality system Inspection and test status
Contract review Control of non-conforming product
Design control Corrective action
Document control Handling, storage, packaging and delivery
Purchasing Quality records
Purchaser supplied product Internal quality audits
Product identification and traceability Training
Process control Servicing
Inspection and testing Statistical techniques

Before discussing each heading it is worth looking at a small excerpt from ISO 9001 . This
gives the reader an idea of the level at which ISO 9001 addresses the QA and development
AIMS process . The extract chosen comes from section 4 .11 :

4 .11 Inspection, measuring and test equipment

• To introduce an important external standard : ISO 9001 .
• The supplier shall control, calibrate and maintain inspection, measuring and test equipment, whether
To interpret the twenty sections of ISO 9001 in software quality terms . owned by the supplier, on loan, or provided by the purchaser, to demonstrate the conformance of
product to the specified requirements . Equipment shall be used in a manner which ensures that
measurement uncertainty is known and is consistent with the required measurement capability .
14 .1 INTRODUCTION
The first thing to notice is its generality : it could apply to the developer of any product .
This chapter has a number of aims, the main one being to describe the increasingly impor- The second thing to notice is the difficulty in interpreting the paragraph-it is obviously
tant international standard ISO 9001 . The standard, which has been adopted for use by aimed at standard engineering processes where equipment such as thermocouples, cali-
more than 130 countries, is becoming increasingly important as the main means whereby bration gauges and potentiometers are the norm . An interpretation of the paragraph is
customers can judge the competence of a software developer . One of the problems with that the supplier shall ensure that any software tools used for testing are of at least
the ISO 9001 series standard is that it is not industry-specific : it is expressed in general the same quality as the software that is to be developed, and that any test equipment
terms, and can be interpreted by the developers of diverse products such as ball-bearings, which produces measurement values, for example, performance monitors, has an accuracy
hair dryers, automobiles, sports equipment, and televisions, as well as software . A num- which is acceptable when compared with the accuracy specified for performance in the
ber of documents have been produced which relate the standard to the software industry, requirements specification .
but do not go into a huge amount of detail . It is the aim of this chapter to describe what This, then, is the type of level of text contained in the ISO 9001 standard . The aim
ISO 9001 means in terms of the quality elements and development techniques described of the following 20 subsections is to examine each of the sections in the standard and
in the book . For the software industry the relevant standards are : produce an interpretation in terms of the contents of the previous chapters of this book .
Whenever numbers appear in brackets they refer to the specific paragraph in the ISO
9001 standard . The final section of this chapter summarizes the components of a quality
• ISO 9001 Quality Systems-Model for Quality Assurance in Design, Development, system which is able to satisfy ISO 9001 .
Production, Installation and Servicing . This is a standard which describes the quality
system used to support the development of a product which involves design .
• ISO 9000-3 Guidelines for the Application of ISO 9001 to the Development, Supply
and Maintenance of Software . This is a specific document which interprets ISO 9001
for the software developer .

190
 192 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION ISO 9001 193

14 .2 THE ELEMENTS OF ISO 9001

overall resource plan . Also, if certain quality controls are carried out using software tools,
these tools should be widely available, documented, and staff trained in their use .
14 .2 .1 Management responsibility
Staff should also be trained in the quality control activities used by the company.
This part of the standard (4 .1 .1) describes the fact that the management of the software If a company sometimes requires external staff to carry out some quality controls, for

developer should publish its commitment to software quality via company policy . The example complicated real-time testing, then there should be a procedure which describes
statement of this policy would appear in broad form in prominent company publications, how such staff should be contracted . There is also an implication that training provision

and in more detailed form in the introduction to the company quality manual . The should be regularly reviewed by management .
This section also states that there should be a designated member of manag
ement
broad form of a company's commitment to policy might appear in statements such as :
`The company believes that good quality assurance is something that occurs throughout who is responsible for ensuring that the requirements of ISO 9001 have been implemented
our projects, involves both our staff and the customer, and cannot be built in as (4 .1 .2 .3) . Normally, this should be a member of staff who is at a very senior level in the
an
afterthought .' The more detailed form might involve a statement such as developer's company-either at board level or just below it . Such a manager would
: `We believe
receive reports from staff carrying out the quality function about serious violations of
that an error discovered during development, no matter how trivial, should be rectified
and the cause of the error determined and documented .' Both the broad and detailed the standard . These would usually be reported as a result of project audits . In addition,

statements should be displayed prominently in documents which are widely available to the manager should have the power to order a thorough investigation of a project which
staff. starts emitting warning signals about budget exhaustion or late delivery .

Staff should be made aware of the quality policy of the company by means of training The final part of this section describes the fact that the quality system should be
and induction courses . Not only should a company produce a quality manual, but it is regularly reviewed . The frequency of review will depend on a number of factors . If the
also a good idea to produce a short guide to the quality manual which would reiterate the quality system is new, then it should be reviewed more frequently than if it was mature
and was seen to be coping well . A review could also be scheduled as part of the preparation
detailed quality aims and objectives of the company, and provide a rationale for company
quality procedures in terms of a history of projects which have had quality problems . for the implementation of new technology such as CASE tools . There are a number of

This section also details the fact that staff who are to carry out either a quality or a potential inputs into such a review : a report from the senior manager responsible for
developmental function should have their responsibility, authority and interrelationship ensuring that the quality system meets ISO 9001 requirements ; project debriefing reports
defined (4 .1 .2 .1) . This would include answering questions such as which highlight concerns about the operation of the quality system ; a staff questionnaire ;
:
summaries of audit reports ; quality records such as the level of defects discovered during
• How a quality assurance department is organized . operation, and the cause of the errors ; and data on project performance in terms of profit
• If a company is not big enough to have a separate quality assurance department,
and delivery time .

what is the managerial relationship between staff responsible for quality assurance
activities, the project manager, and the senior member of staff responsible for quality
matters . Self Assessment Question 14 .1 Would the fact that a company speci-
• fied that the skills and background of developmental staff is specified in the
What power a member of quality staff has when a problem is discovered, such as an
project plan be related to this part of the ISO 9001 standard?
audit discovering that a quality control has not been properly executed .
• Whether the team structure for a project is adequately defined, and its relationship
Solution Yes, it covers the fact that staff who are to perform a speci-
with the quality function and other external agencies specified .
• fied task should have the skills to carry out that task .
Whether an adequate development model is employed by each project .

Such details should be found in two sources : the job descriptions of staff categories
employed by a software developer, and the staffing section of the project plan which 14 .2 .2 Quality system
would detail project-specific responsibilities which may not be contained in broad job
descriptions . This section of the standard (4 .2) specifies that the software developer should establish

In this part of the standard (4 .1 .2 .1) there is an implication that as and maintain a documented quality system which satisfied ISO 9001 and that it should
far as possible be effectively implemented . The first requirement is really a reference to the fact that
quality staff should be independent of developmental projects . A further part of this
section (4 .1 .2 .2) specifies that the software developer should identify in-house verification there should be a widely available set of standards and procedures . These standards and
procedures should constitute a dynamic document, and should be updated in response to
requirements and provide adequate resources for these . This means that the developer
any management reviews and any developments in technology such as a new programming
should have identified all the main tasks used for checking both customer-related quality
language being adopted by the developer .
attributes and developer-related attributes, and have qualified and experienced staff who
can carry out these tasks ; normally, this information would be found in the company's The part of the section that deals with the effective implementation of the quality
system (i .e . its standards and procedures) has two implications for the software developer .
 ISO 9001 195
1 94 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

The first is that audits of software projects should be held to ensure that the agreed sounds innocuous to any reader who thinks that a contract is just a straightforward legal

quality controls in a project's quality plan have been implemented, and that the stand- document . The response of anyone reading this part of the standard might be to ensure
that there is a procedure which details how the legal department should be involved in
ards and procedures governing developmental activities and the quality controls have
. For
been correctly carried out . These audits should be carried out with a frequency that is this process . While this is an important activity, it is by no means the only one
example, a contract will specify a delivery date and a price and, usually, a document
determined by a number of factors, including the criticality of the activity that is being
audited (for example, requirements analysis is a much more important activity than which forms part of the contractual bundle associated with a contract is some form of

programming, and one would expect more auditing of tasks associated with requirements requirements specification . Thus, the developer has to ensure that customer requirements
analysis), and the experience of the members of staff (a quality manager may ask staff are adequately defined, properly specified, and that the software can be delivered on time

responsible for the quality assurance function to audit the work of a new member of and to budget .
staff) . These are major implications which have quite a lot of ramifications for the quality
. There
The second implication of this section of the standard is that adequate quality records system . First, let us examine those connected with the requirements specification
should be kept which enable staff concerned with quality to identify either that staff on a are a number of models of contracting in the software industry . One is to issue a tender
project are not complying with standards or procedures, or the existence of deficiencies for a system which is split into two phases : first, the software developer produces a

in the quality system . There are two types of quality records . detailed requirements specification (which he would then be paid for) ; he would then be
The first detail the number and seriousness of errors discovered during each phase contracted to produce a system which is based on this specification . A second model is
where a contract is awarded on the basis of the customer statement of requirements, and
and the phase in which the error was committed . By producing a plot of percentage
a third is where a contract is based on an outline requirements specification which may,
of errors committed, and percentage of errors discovered on a phase-by-phase basis, a
software developer will be able to detect which parts of the quality system may not have or may not, have been paid for by the customer . In the first and third cases, the quality
been implemented correctly by a project . system should provide direction on how the requirements specifications are validated with

The second consists of documentation which provides proof that a quality control has respect to the original customer statement of requirements . This will mean outlining the
various techniques which can be employed, for example technical reviews, walkthroughs,
been successful and has checked that a particular quality attribute has been implemented
in a system . A typical example is a test report . and prototyping . The main activity to ensure this will be a requirements specification
technical review, and the software developer should certainly have standards in place to
An important part of the quality system is that the company should document at
determine the conduct of these reviews (described in Chapter 4 of this book) .
least one process model which can be tailored to particular projects . This topic was
described in Chapter 12 . It is worth pointing out here that one model may not be ade- When the second model of contracting has been adopted, the quality system should
provide direction on how to ensure that the software developer incorporates a description
quate ; for example, a software developer may develop third-generation systems using a
of his or her perception of the system to be developed in the contract . This document
development philosophy which is process-driven, and may also develop commercial data
should be of at least the quality of an outline requirements specification, and should be
processing systems using a philosophy which is data-driven . In this case, one process
model may be inadequate and two would be called for . reviewed in the same way as a full requirements specification .
This section of the standard also emphasizes the fact that any requirements differing
Implicit in this section is that each project should produce a quality plan which
from those in a tender produced by the customer are resolved . A software interpretation
details how controls described in the quality manual are to be used to check that quality
of this is that there should be adequate documentation of contact with the customer, to
attributes specified at the beginning of the project are present in the final software .
show that differences between the contractual requirements document and the original
statement of requirements document have been notified to the customer and have been
Self Assessment Question 14 .2 Would the process of project auditing, agreed by a customer's representative .
i .e . the execution of detailed checks on software projects to ensure that they This part of the standard can also be addressed by documentation that links the
are adhering to their quality plan, be relevant to this part of ISO 9001? requirements specification document to the original statement of requirements produced
.
by the customer . Details of this form of documentation can be found in Chapter 4
Solution Yes, this activity is an important part of the quality system- The final part of this section concerns the fact that the supplier should ensure that
without it there would not even be a point in developing a quality plan for he or she has the capability to meet contractual requirements . The most important is,
a project . of course, delivery to the requirements detailed in the requirements specification, but
this also deals with the ability to deliver on time and to budget . Reviews and feasibility
studies are normally used to implement this part of the standard . The quality system
should provide standards and procedures for carrying out a feasibility study : a mini-
14 .2 .3 Contract review project which determines whether all the requirements including cost can be met .
This part of the ISO 9001 standard describes what the software developer should do to A requirements review is used to determine whether individual requirements can
ensure that the contract to produce a software system is a valid document (4 .3) . This be met . For example, the customer may insist in the statement of requirements that a
 1 96 ISO 9001 197
SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

particular requirement well beyond the state-of-the-art of software development has to be The third subsection (4 .4 .3) specifies that design input requirements relating to the
implemented . A requirements review should, as part of its remit, address this aspect . A system to be developed shall be identified and documented, and that poorly specified

project plan review is used to determine whether the set of tasks, resources and schedules requirements shall be resolved with the staff who drew them up . This is a reference
specified in the plan will meet the target date for delivery, and enable the profit called to the fact that the requirements specification, which is the main input into the design
for by the company's business plan to be met . The review should ask questions such as : process, should accurately reflect the customer requirements which were initially detailed
in the statement of requirements . Details of what exactly this means can be found in
• Do we have the expertise to carry out this task? Chapter 4 of this book . The implication for the quality system is that there should be a
• Is the project so risky that even if we built in risk avoidance techniques there is still standard for expressing a system design, and if detailed design forms part of a company's
a high probability that we will lose money on this project? development practices, a standard for detailed design ; there should also be a procedure
• Are the resource requirements for these tasks reasonable? which describes the processes of both detailed and system design .
• Can we have confidence that the customer will carry out specific tasks within the Another important issue touched upon by this part of the standard is that of poorly
time period that we have specified in the project plan? expressed requirements . The part of a quality system describing design should include a
procedure for describing what a designer should do when a poorly expressed requirement
Within this section there is an implicit requirement that the eventual requirements speci- is found .
fication produced is complete, and describes all customer requirements, and that even- The next subsection (4 .4.4) is concerned with the documentation of the design that
tually the final design and program code of the system can be checked to ensure that it has been produced by the software developer, and specifies that the output from the
meets those requirements . design process-the system design or detailed design-must meet the design input : those
requirements detailed in the requirements specification .
Subsection 4 .4 .5 deals with the output from the design process, and concentrates on
14.2 .4 Design control outlining the fact that the developer should plan, initiate, document and resource the
This part of the ISO standard (4 .4) is divided into six subsections . The first (4 .4 .1) process of verifying that a design meets its input . A software interpretation of this is
describes the fact that a supplier shall have standards and procedures which control and that standards and procedures should be available for activities such as design reviews,

verify the design of the software system in order to ensure that customer requirements prototyping, simulation, and response time calculation, and that proper planning should
are met . precede these activities .
There are a number of implications behind this . First, that adequate notations for The final subsection (4 .4 .6) is a reference to the configuration management practices
expressing the design are provided by the quality system, i .e . a notation which produces adopted by the developer : that all changes should be documented, reviewed, applied and
designs which can be checked against user requirements and which represents a global, checked, and that any related documents which also need to be changed are processed
physical view of the system . Second, that adequate standards for validation and verifica- correctly.
tion activities are detailed in the quality system ; for example, the quality system should Other implications in this part of the standard are : that the outputs from develop-
provide a standard and procedure which governs the conduct of system design and de- mental phases are documented and reviewed, and that when an item is identified as not
tailed design technical reviews . The quality system should also provide facilities whereby conforming to requirements it is tagged as such, and not used in subsequent development

traceability documentation, such as the verification matrix described in Chapter 5, is until it does conform .
constructed . Another feature which should be offered by the quality system is directives
which specify how the developer is to check the functional quality attributes and non-
14.2 .5 Document control
functional attributes, such as response time, against a design . The normal medium for
this is the technical review . This section of the standard (4 .5) is split into two subsections . The first (4 .5 .1) describes
Another important feature of this subsection of the standard (4 .4 .2) is the specifi- the fact that documents produced during a software project should be reviewed and

cation that the organizational and technical interfaces between different groups shall be approved for adequacy before they are used for developmental activities . The implication
identified and their interface documented . This is a reference to the provision in the qual- of this for a software quality system is that each task which produces a document, for

ity manual of details of how the relationship between the software development team and example the system design task, should be accompanied by a sign-off which acts as a
other agencies, such as hardware developers, should be managed . It should also detail quality control . Sometimes this sign-off will be carried out by a member of staff who has
the relationship between the design teams who are designing subsystems and how the re- inspected the document ; sometimes it will be signed off by the chair of a meeting such
lationship between the analysts and the designers on the project is handled . The quality as a technical review .
system should document the means of communication, the standards for the documents The second subsection (4 .5 .2) discusses the application of change control to project
used for communication, and aspects such as the querying of the functional properties documents . It specifies that changes to project documents such as the requirements
of the system by the designers . If there is any need for liaison between the designers and specification should be reviewed, approved or rejected, and the body which approves

the customer, this should also be the subject of standards and procedures . changes should have access to all the information on which it is necessary for them to
 ISO 9001 199
198 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

base their judgement . This includes the reason for the change ; the effect of the change tests need to be carried out on such software . Normally, the answer to this would be no ;
on the system ; and the resources required to implement the change . however, there are certain occasions when even software which is regarded by the indus-
This part of the standard also implies that a configuration management system try as reliable might need some acceptance tests to be carried out on it . For example,
capable of keeping data on versions of documents and the change history should be used a compiler used to produce code for an ultra safety-critical application, such as nuclear
by the developer . plant control, would need to be formally accepted by the software developer, either by
carrying out tests which check the generated code or by appraising tests carried out by
independent agencies .
14 .2 .6 Purchasing

This section details what issues a quality system needs to address when software projects
have to purchase or use external products-either software or hardware . These external Self Assessment Question 14 .3 Would a check on a compiler to en-
sure that it generated correct code-carried out as part of a safety-critical
products could either be subcontracted or could be provided by an external agency, such
as the customer or a government department . It describes what a software developer project-be relevant to this part of the standard?
should do to ensure that these external products have the same level of quality as the
software which will incorporate them . Solution Yes, even though the compiler might have been purchased far
The first subsection (4 .6 .1) is a general statement that all purchased products which in the past, it may only have been used on non-critical projects .
form part of a system should conform to specified requirements . This is a general direc-
tive . However, underneath this statement there are a number of major ramifications for
the software developer who is to handle a purchased product . The first is that the re- 14 .2 .7 Purchaser supplied product
quirements specification to be produced for someone like a subcontractor should be of the
same standard as the requirements specification used in the project, and should receive This short section (4 .7) simply re-emphasizes the important principle that software sup-
the same degree of validation . Also, the procedure whereby this software is system and plied by the purchaser should be checked for correctness as much as externally subcon-
acceptance tested should be of the same level of thoroughness as that employed within tracted software, and that configuration control should apply to these items as much
the project that uses the purchased software . The implication of both these requirements as to software developed within a project . An important implication of this is that the
purchaser should supply not only the software, but also the requirements specification of
is that the software developer should have carried out some form of assessment of the ca-
pability of a subcontractor to act according to the requirements specification and testing the software .
standards that will be used .
Subsection 4 .6 .2 deals with the assessment of subcontractors . It places an obligation 14 .2 .8 Product identification and traceability
on the developer to ensure that subcontractors who are competent to meet the quality
goals of the overall system are selected. There are a number of ways of carrying out this This is a direction that elements of the system to be developed-not only software, but
documents such as the requirements specification-should be adequately identified and
selection : keeping records of subcontractor performance during a software project which
documented and that traceability should be maintained between related documents .
can be consulted by a project manager who is thinking of employing a subcontractor,
carrying out an external quality audit by visiting the subcontractor, or by asking previous In order to illustrate this, consider an example taken from testing . At the beginning
clients of the subcontractor about their experiences . Ideally, all three measures should of the software project, the developer will have identified a number of quality factors .
These factors will each be uniquely identified and will be cross-referenced to statements
be employed .
Subsection 4 .6 .3 deals with the identification of product to be purchased . This is in the requirements specification . These will be documented in the quality plan and will
normally taken to be a directive to the software developer to specify any subcontracted be checked by a quality control . The quality control will be given a unique identification
software correctly and to ensure that adequate change control applies to this software as and will be cross-referenced to the quality factors . Those quality controls which represent
much as to the software and documentation produced in the project . user-related functional quality factors will give rise to a test design . This test design will
The final part of this section (4 .6 .4) deals with the verification of purchased product . again be identified and cross-referenced to the quality control, and may give rise to a test
In software terms it is interpreted as a directive to the software developer to provide an procedure . This test procedure again would be documented and cross-referenced to the
adequate set of quality controls, usually implemented as acceptance tests, to ensure that test design . Associated with the test procedure there may be a number of test files which
will be given a name that enables them to be cross-referenced to the test procedure . In
any software which has been purchased meets the requirements specified by the software
developer . this way, each element of the system is uniquely identified and traceability is built into
the system-from the requirements specification down to the test data used to check a
The issue of external software used on a project is a slightly complicated one, as
software projects which do not use subcontracted, purchased or customer supplied soft- customer-related functional quality factor .
Another requirement of this section is that some form of version information should
ware still, in a sense, use external software ; for example, testing tools and compilers are
external software . The software developer has to make a decision as to whether extensive be attached to components of a developed system .
 2 00 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION ISO 9001 201

14 .2 .9 Process control the customer may ask for early delivery of a system containing some subsystems which
This part of the standard (4 .9) describes the requirements upon development processes have not been acceptance tested . The developer should document those parts of the
delivered system which have not been adequately tested ; this documentation would be
such as system design . The major part of this section concerns the execution and planning
of these processes . used in a number of ways : as reporting data to high-level management, as information
to staff carrying out field support, or to staff who are responsible for the subsequent
A major interpretation is that the developer should adequately plan the development
process . This means that the project should be split into its various tasks, the right upgrade to the software which would be fully tested .
The second subsection (4 .10 .2) deals with process testing and inspection . It requires
people should be assigned to the tasks, task details such as duration and effort should
the developer to carry out testing during development, normally interpreted as integra-
be documented, and the resource estimate for the project should also be calculated . tion testing, unit testing, and system testing-and specifies that the developer should
The standard places stress on the fact that adequate documentation which describes
use mechanisms to ensure that such tests have been carried out adequately, and that
the work being carried out is given to the staff who are to execute a task ; for example,
the test results conform to what is expected . In software terms, the developer should
that adequate direction is given to staff who are to test a system based on their knowledge
have documentation standards which describe the tests that are carried out during unit,
of the system ; that the quality system provides adequate standards for a task ; that a
integration and system testing, and include sections that describe the actual test output .
proper design methodology is specified ; that in design not only functional attributes, but
Such documentation would be checked by a member of staff signing it off and also by
also any residual design directives that the customer has insisted on, are communicated
staff responsible for auditing .
to the designer . It also places great emphasis on standards for activities such as design
This subsection also specifies that a software system should not normally be released,
and programming, and the documentation in the quality system of good practice for
either to the acceptance testers or the customer, until all the tests and reviews which
developmental activities .
should occur during development have been carried out . The final part of this section
Another feature of this part of the standard is that adequate monitoring and control specifies that the software developer should be able to identify parts of the system which
of both the processes and products of a software project should occur . The main tools
are non-conforming, i .e . in terms of inspection and test, modules and partial versions
for implementing this aspect of the standard are the audit and the process of signing off
which have either not been tested or have failed their tests .
items by senior members of the project .
Subsection 4 .10 .3 concerns what it calls final inspection and testing, but which in
The final part of this section (4 .9 .1) describes special processes . These are tasks for
software terms should be interpreted as acceptance testing. It specifies that the final
which the result cannot be predicted during a software project and where deficiencies can acceptance tests should be carried out in accordance with the quality controls described
only be identified during actual use . The standard specifies that these processes should
in the quality plan for a. project, and that software should not be released until all the
be identified, and that even if they are regarded as special the same level of care should
quality controls have been carried out and confirmed .
be taken over them as over processes which can be checked .
The final subsection (4 .10 .4) states that the software developer should maintain
An example of this type of process is where a system is developed which has a records which provide evidence that a system, or parts of a system, has correctly passed
specified response time for a particular hardware configuration, but where an element of all its tests-that, for example, the project manager can call for module documentation
that configuration, say a new memory device, is not available well into the operational and see that the modules have been tested and that no problems have been raised by the
life of the system . The process of calculating the response time of the system with such
a device should be regarded as a special process, and although it would be carried out tester .
according to the developer's best practices, it would need to be identified during the
project planning . It should be marked as special, with the customer being informed that 14 .2 .11 Inspection, measuring and test equipment
although a guarantee of response time can be given for the first version of the system This section specifies that when any software tools such as dynamic analysers are used,
without the new memory device, no guarantee can be given of a better response time for
the developer should be confident that these tools work properly, and that records are
the full hardware configuration .
available-both internally and externally-to demonstrate that the developer has actu-
ally checked their functioning . This is vitally important not only for testing tools which
14 .2 .10 Inspection and testing are produced by the developer but also for bought-in tools . However, the degree of testing
applied to such tools will depend, in the main, on two factors : the maturity of the tools
This important part of the standard (4 .10) is split into four subsections . Subsection 4 .10 .1
and the criticality of the application . For a new software testing tool which has just been
describes the fact that a system or part of a subsystem should not normally be released to released and which is to be used on a safety-critical application, it is important that the
a customer unless it has been checked as conforming to requirements . This will normally
be carried out as system or acceptance testing . developer tests such tools thoroughly . Even for mature tools employed on non-critical
applications, it is still worth keeping test sets for checking out the tools ; such tools do
The final part of this subsection specifies the action to be taken if a system, or undergo revisions and can be issued in versions which include new errors . For tools which
part of a system, is released for urgent reasons . Typically, this might occur when the
are developed within a project, it is vitally important that the same quality regime used
business circumstances of a customer change during the development of a system, and to govern the developed software is used to oversee the development of the tools .
 ISO 9001 203
202 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

14 .2 .12 Inspection and test status 14 .2 .14 Corrective action

This section of the standard is related to that described in Section 14 .2 .13 of this chapter . This section (4 .14) describes the process of carrying out tasks when a defect is discovered,
It stipulates that software products such as modules should be properly identified in a defect being something which prevents a software system, or part of it, from meeting

order that staff can easily discern their test status ; for example, whether a module has requirements . This part of the standard has a number of themes .
passed tests which confirm that it satisfies certain requirements . This identification, which The first theme is that when a product is discovered as not meeting requirements-
is usually implemented at the subsystem level, should ensure that when a system is usually during system or acceptance testing-the developer should investigate the reason

assembled prior to release to the customer, no part of the system that is untested or for this and carry out any corrective action . Thus, if a test fails because of a fault intro-
poorly tested is included . duced during requirements analysis, then the developer should amend the requirements
specification, the system design, and the affected program code ; furthermore, there should
be an audit trail which leads into and out of the configuration management system that
Self Assessment Question 14 .4 Would a direction to the programmer documents this .
to include details of when a module was tested be relevant to this part of A second theme is that the developer should be continually monitoring the causes
ISO 9001? of defects via documentation such as test reports and customer complaints in order to
discover whether non-conforming software is being produced because of a deficiency in
Solution Yes, indirectly : it would at least give an indication that the mod- the quality system . The implication here is that a main driving force in the continual
ule had been tested . improvement of the quality system is that of monitoring non-conforming software .
A third theme is that the quality system should specify the actions to be taken when
a defect is discovered . For example, when an acceptance test fails, there should be a
clearly defined reporting procedure which results in a member of staff, usually a designer
14 .2 .13 Control of nonconforming product
or senior programmer, carrying out a mini post-mortem . The quality system should also
This section of the standard (4 .13) describes what is necessary in order to cope with a specify what should happen after the result of the post-mortem has discovered the cause
.
software product which does not satisfy specified requirements . Examples of this include : of any defects, i .e . that work instructions should be issued to eliminate the error

• A module which has failed some of its unit tests .

14 .2 .15 Handling, storage, packaging and delivery
• A subsystem which has failed some of its system tests .
• A design which has been reviewed and in which a number of design errors have been This part of the standard (4 .15) describes the process whereby the developer assembles
discovered by the review team . the final software product ready for delivery to the customer . This reads very much
like a set of directives to companies who produce non-software products, as it mentions
There should be adequate checks to ensure that subsequent development tasks do not deterioration and secure areas . However, it does have relevance to software development :
use items which have not satisfied requirements . For example, as part of the process of the product which is to be sent should be software which has been adequately tested,
installation, a check should be made that every module has been signed off by a senior and the methods for detecting non-conforming software detailed in the previous sections
member of the development team certifying that the module has been adequately tested . should have been applied and have easily been seen to be applied .
Another example is where the chair of a design review carries out a preliminary check The company for whom this section is the most relevant is the package developer .
that the part of the requirements specification which describes the design to be reviewed This company will perhaps have developed a small number of products but might have
has also been certified by a senior member of staff as having no residual errors . sold thousands of copies of each product, with every copy being different because it
One area where it is easy to produce non-conforming product is during maintenance satisfies the local circumstances of the customer . This part of the standard places an
or during the process of modifying a configuration item that has been baselined . Here, onus on the developer to ensure that the right version of a package is delivered and,
a designer or programmer may carry out the task which was specified, for example moreover, that when updates to parts of the system have been produced, the subsequent
rectifying a design fault, but may, at the same time, insert some extra code which affects deliveries of the system are the correct ones .
functions of the system which should not have been affected . A quality system which There is also an implication that software and project documents should be safely
detects a non-conforming product should address this issue . This means that checks stored and precautions taken against events such as fire, flooding and virus attack .
should be carried out to ensure that a change only addresses the area which it was
intended to address . There are a number of ways of doing this : one is to run a series of
14 .2 .16 Quality records
tests which check out the functions which should be unaffected ; another is to have all
changes reviewed, perhaps not by a large review team but by one or two other members Section 4 .16 places an obligation on the software developer to establish, store and main-
of staff . tain quality records . Such records are the visible confirmation that a software system
 ISO 9001 205
204 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

14 .2 .19 Servicing
meets the quality attributes specified in the quality plan . An important point made in
this section is that this includes not only quality records generated by the prime developer This section (4 .19) specifies that when a software system is to be serviced, there should
of a system but also those which arise from subcontractors . be standards and procedures in place which govern servicing, and the supplier should
Typically, these quality records will either be test reports (as described in Chapter have developed checks which ensure that the desired level of servicing has been achieved ;
7), the minutes of technical reviews, or documents which have been signed off by senior for example, a service contract may specify that all queries from the customer will be
members of the development team . This part of the standard effectively states that answered within 48 hours . The software developer should hence have developed some
for every quality attribute defined during the early stages of the project there should standards which require staff responsible for servicing to fill in the time of user request
be documentary evidence that the quality attribute has been met by the developed and the time when the request was answered .
software .
14 .2 .20 Statistical techniques
Self Assessment Question 14 .5 Would the production of documentation This section is really intended for the developer of mass-produced goods such as ball-
that showed that a module was clean-compiled be relevant to this part of bearings and hair dryers, where statistics are frequently used to check on product accept-
the standard? ability and the efficiency of the processes used to create a product . In software terms, its
interpretation is more problematic . There is a body of research which is attempting to
Solution No, this is not a quality record : it states that the module does predict metrics such as the mean time between failure from test results . However, this
riot have any syntax errors ; however, it could have many functional errors . work is still in its early days, and the standard does stress that, where appropriate, statis-
tical techniques should be employed . I would say that, except for the software developers
with the most mature quality systems, this paragraph currently has little meaning over
14 .2 .17 Internal quality audits and above the collection of error data from activities such as reviews, but that in the
next five years it will become increasingly important . Chapter 16 discusses these issues
Section 4 .17 specifies that the developer should initiate a series of quality audits which in more depth .
check that the quality system is being applied correctly ; that is, that standards and
procedures are being adhered to, and that the controls associated with the quality at-
tributes defined in the early part of the project have been applied . A subsidiary point Self Assessment Question 14 .6 Would the use of data which detailed
made in this section is that quality audits should be scheduled on the basis of the status the number of tasks carried out, the number completed on time and the
and importance of the activity ; for example, I would expect a company to audit more number over time be relevant to this part of the ISO 9001 standard?
frequently during requirements analysis than during programming, although the latter
activity should not be ignored . Solution Yes, this would be valuable data for staff who might want to
The standard describes the fact that the result of the audit should be documented improve the estimating process .
and brought to the attention of relevant staff who are concerned with quality policy .
Deficiencies that are found should then be followed up by staff responsible for quality
assurance . This follow-up may range from the harsh-a reprimand-to the less harsh-
an investigation as to why a particular facility of the quality system is being ignored by 14 .3 MAIN SUMMARY POINTS
a number of project managers .
This final section summarizes the points made in the previous 20 subsections . It outlines
the components of a quality system which should satisfy ISO 9001 .
14 .2 .18 Training
This section (4 .18) places a responsibility on the software developer to establish a system
Management responsibility
whereby the training needs, both of the company and of individuals, are identified and
sufficient training is put in place to meet these needs . The standard specifies that all • General quality aims and objectives in prominent company documents such as the
staff whose work affects quality should have training provided for them . I take this to annual report .
be a directive to provide training for all staff . The standard also specifies that training • More detailed quality aims and objectives in the quality manual .
records should be maintained . Such records would normally be consulted by managers • A short guide to the quality manual .
when they are carrying out the process of staffing a project . The training plans of the • Training courses for staff which introduce them to the quality system .
company should also be periodically reviewed .
206 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION
ISO 9001 207

• Accurate and up-to-date job descriptions being maintained by the company which
• The use of a project plan review to determine whether a project plan provides an
detail broad staff responsibilities .
adequate framework for the production of a software system with the desired quality
• A staffing section in the project plan which describes project-specific responsibilities
for all the staff involved in the project-both development and quality assurance staff . attributes .
• The production of a requirements specification which contains all customer require-
• A section in the company's overall resource plan, detailing staff levels for staff carrying
ments .
out activities such as testing .
• Traceability should be built into project documents so that customer requirements
• Adequate documentation of any software tools for implementing quality controls .
Adequate training for staff using software tools used for implementing quality controls . can be traced to program code via the system design and detailed design .
•
• Adequate training in quality control activities, such as system testing or technical
reviewing . Design control
• A procedure for hiring staff required for carrying out project-specific or one-off quality
• An adequate notation for design is employed by the developer .
controls .
• A senior manager responsible for overseeing the operation of the company quality • Adequate validation and verification activities which can be applied to a design are
described in the quality manual .
system and its adherence to ISO 9001 .
A regular review of the effectiveness of the quality system, based on documents such • Traceability between the requirements specification and the design should be built
•
into project documentation .
as project debriefing reports and summary project audit reports .
• A regular review of training provision . • The interface between the design team and other agencies should be documented in
the project plan .
• As far as possible, quality assurance being performed independently of the software
• Procedures should be available to govern the process of querying the requirements
projects .
which are fed into the design process .
• There must be explicit documentation produced which reflects the fact that the in-
Quality system put to the design process-the requirements specification-has been satisfied by the
design .
• The existence of documented standards, procedures and guidelines which have to be
• The design must come under configuration control, i .e . all changes to the design must
adapted and adopted by all the software developer's projects .
A standard and procedure which describes how auditing is used to check that project be rigorously reviewed, applied, checked and documented .
•
• The outputs from a development phase such as system design should be documented
standards and procedures are being adhered to by individual projects .
The collection and retention of quality records, which would normally describe defects and reviewed .
•
• If any developmental item such as a subsystem design is found not to conform to
on a phase-by-phase basis .
requirements, it should be tagged and not used until it does conform to requirements .
• The production of quality records which detail that a particular quality control has
been carried out .
• At least one process model which can be tailored to specific projects should be pro- Document control
vided as part of the quality manual .
• All the main documents generated during a software project should come under formal
• The production of a quality plan for every project which describes the quality at-
change control .
tributes for the software system that each project produces .
• There should be standards and procedures for notifying change and considering
whether a change should take place .
Contract review • Standards and procedures should be contained in the quality system to enable staff
concerned with evaluating change to decide whether a change should be approved or
• A technical review of the customer statement of requirements as preparation for
disapproved .
requirements analysis .
• The configuration management facilities offered by the quality system should be cap-
• A technical review of the requirements specification for adequacy against the customer
able of keeping version data and change history data .
statement of requirements.
• Documentation of the interaction between the customer and the developer.
• Traceability documentation, which relates requirements specified in the statement of Purchasing
requirements to those contained in the requirements specification .
• The same level of requirements specification used on a project should be employed to
• Standards and procedures for carrying out a feasibility study .
communicate with software subcontractors .
• The use of a requirements review to determine infeasible requirements .
• A proper acceptance procedure should be adopted for all software which is purchased .
208 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION iso 9001 209

• There should be explicit evidence that subcontractors are capable of producing soft- • The quality system should ensure that the developer documents those parts of a
ware of the same quality as the developed software . system which have not been adequately tested .
• The same level of change control should be adopted for purchased software that is • Adequate testing such as unit testing should be carried out in advance of acceptance
employed for the developed software . testing .
• An adequate acceptance testing regime should be applied to any external software . • Adequate documentation should be provided for testers which not only details the
test(s) to be carried out, but what the outcome of the tests should be .
• Software should not be released to acceptance testers until all testing activities which
Purchaser supplied product
precede it have been correctly carried out .
• Software supplied by the system purchaser should be checked as rigorously as the • Final acceptance tests should be derived from the quality attributes described in the
software that is being developed . quality plan .
• Software supplied by the system purchaser should be placed under configuration con- • Adequate documentation should be generated which confirms that a system or part
trol . of a system has passed all the tests which have been applied to it .
• Any purchaser supplied software should also be accompanied by its requirements
specification, and possibly its design .
Inspection, measuring and test equipment

• Where necessary, test and inspection tools are tested and test records are made gen-
Product identification and traceability
erally available .
• Each element of the system, be it documentation or code, should be uniquely identi- • For tools developed internally, the same quality regime that is used for the project
fied . on which the tools are to be used is employed during their development .
• As much traceability as possible should be built in between elements of a system .
• Version information should be held for elements of a software system .
Inspection and test status

• Parts of the system that are separately tested should be documented with their test
Process control
status .
• Tasks which affect quality should be identified and properly planned .
• Such tasks should be specified and associated with planning information such as
Control of nonconforming product
duration and sequence in time .
• Adequate documentation which describes the developmental tasks to be carried out Adequate checks should be implemented, prior to a task such as system design, that
•
should be provided for the staff responsible . all inputs into the task have been certified as being correct .
• Adequate facilities for monitoring the execution of a task vis-a-vis the project plan • That a maintenance change, or change to a baselined item, has not affected any other
should be provided . parts of the item which should not be affected by the change .
• Adequate facilities for monitoring the adequate completion of a task should be pro-
vided .
Corrective action
• Adequate facilities which enable quality staff to check that a quality control has been
carried out correctly should be provided ; for example, there should be standards for • When a defect is discovered, at whatever stage in a software project, the cause of the
auditing in place in the quality manual . defect should be investigated and documented .
• Standards should be provided for all tasks which affect quality . • Corrective action, when carried out, should not only affect the part of the system
• A project should specify an adequate development methodology- in which a defect was found, but should lead to the modification of all connected
• Good practice for tasks which affect design should be publicized in the quality manual . documents, such as designs .
• Special processes should be identified during planning, and marked as such in any • The developer should monitor the cause of defects in order to improve developmental
contractual documents . practices and the quality system.
• There should be well defined standards and procedures which govern what should
happen when a defect is discovered .
Inspection and testing
• There should be adequate standards and procedures which govern the reworking that
The quality system should provide checks that no system or part of a system is occurs when a defect is discovered .
•
released to the customer unless it has been checked as conforming to requirements .
 ISO 9001 211
210 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

PROBLEMS
Handling, storage, packaging and delivery
• When the final software is assembled for delivery, standards and procedures should 14 .1 Write down what you think is an adequate statement of policy on quality assurance which addresses
the management responsibility part of ISO 9001 .
be in place which ensure that it does not contain nonconforming product .
• The correct versions of the components of the system should be assembled into the 14 .2 Detail some of the questions that you might ask a subcontractor to judge their competence in
order to satisfy the part of ISO 9001 which deals with purchasing .
system that is to be delivered .
14 .3 Give examples of the type of documentation required in the part of ISO 9001 which deals with
quality records .
Quality records 14.4 Write down a procedure which might satisfy the needs of the part of ISO 9001 which deals with
internal quality audits .
• Quality records should be maintained by all projects . These records describe whether
a particular quality control has checked that a specific quality attribute has been built
into a software system .

Internal quality audits

• The quality system should contain standards and procedures which determine the
conduct, frequency and relevant documentation of quality audits .
• The quality system should also detail how the results of an audit are to be used .

Training
• The company should have a training plan which covers not only the training needs
of individuals but also those of the company .
• The training plan should be periodically reviewed .
• Training records should be readily available to staff carrying out project planning .

Servicing
• Where servicing of a software product is carried out, the developer should have stan-
dards and procedures which ensure that any services specified in a contract are being
delivered to the customer .

Statistical techniques
• Appropriate statistical techniques should be used for prediction . These techniques
should, however, be mature, and not the result of untried research .

14 .4 FURTHER READING

There has been very little written on ISO 9001 . However, a few case studies and short
articles containing advice have been published . Chan (1993) describes the implementation
of an ISO 9001 certified system, and Hemington (1993) provides excellent advice on
implementation .
 CASE STUDY 213

made up of as much as 20 per cent of software, but this type of system was something
15 of a rarity .
Because of the hardware-oriented nature of the MonitorMaster systems, the staff
CASE STUDY employed by the company were a mix of hardware engineers and process engineers . If
software was required then, to quote the words of the technical director : `somebody just
wrote a program' . The advent of the micro changed all that : during the early eighties the
systems supplied by MonitorMaster contained as much as 50 per cent software ; towards
the end of the decade as much as 80 per cent . Because of this increase, MonitorMaster
hired five software staff to replace hardware engineers who had left to form a company
geographically close to MonitorMaster .
It was in the late eighties that trouble started to occur . Three projects experienced
difficulty. The first of these used a new hardware device for which there was inadequate
documentation, and the project had to employ a simulator as the device was not available
until acceptance testing . The system tests for the project went well, including response
time tests . However, when the device was delivered and used in acceptance testing,
the software developer realized there was a major problem : the actual hardware device
differed in characteristics from the simulator . In particular, it led to the response time
dropping considerably. This meant that MonitorMaster had to redesign and reprogram
AIMS the system to increase the response time . This took eight months, and cost the company
a lot of overtime payments . The system was eventually delivered six months late, and
• To show how one company went about improving their quality system . MonitorMaster made only a tiny profit .
• To describe some of the quality problems encountered by one particular company . The second project involved a customer who continually changed his mind ; Monitor-
• To detail some important points about quality improvement which are universal . Master staff would meet with the customer at progress meetings, only to be faced with
some further change in requirements . This process continued well into the coding phase
of the project . The system which was delivered was received enthusiastically by the cus-
15 .1 BACKGROUND tomer; however, it differed quite radically from the system described in the customer's
original statement of requirements . The profit made was much less than was expected .
MonitorMaster PLC was a company which produced systems for industries where real- The final project was a major disaster . It was a large project contracted to a Ger-
time monitoring was involved . Its customers included chemical companies, avionics com- man chemical company, and the statement of requirements given to MonitorMaster was
panies and, more recently, water utility companies . It was set up by two men who worked extensive and detailed . It was an English translation of the original German document,
in the chemical industry and had seen the large amount of profits which had been gen- and in the translation process much of the subtle meaning behind a number of sentences
erated by real-time system subcontractors who had delivered what they regarded as had been lost . Also, one key member of the customer's staff with whom MonitorMaster
substandard systems . They left their employment with a large chemicals company and had to liaise spoke poor English . The system contained seven subsystems ; unfortunately,
started up MonitorMaster PLC. The first two or three years were very difficult : many of because of problems with the statement of requirements, three of these subsystems did
the systems suppliers were established, and had very good relationships with potential not meet requirements . This was only discovered during late system testing in Germany .
customers . However, the company managed to get two key contracts for metering systems The project was heavily delayed while these problems were rectified and, because of the
which went well : the systems were delivered on time, to budget and, more importantly, cost of reworking and of travel and accommodation for a software team in Germany, the
the two customers thought the systems were, in quality terms, far in excess of what had company made the first project loss in its history-and it was a big loss .
been supplied previously. After this success, other potential customers soon got to hear of During the third project the company was also coming under pressure in its strongest
the quality of the work of the company and started placing orders . Soon, MonitorMaster market : the United Kingdom . In particular, the company set up by the five hardware
was an extremely successful company. During the seventies it expanded its staff to 15, engineers who had left MonitorMaster had gained a number of medium-sized projects at
and the boom period in the eighties saw further expansion to around 35 staff . the expense of MonitorMaster .
Unfortunately, the eighties saw another phenomenon which gave rise to many prob- MonitorMaster was still financially healthy, but the three projects which were not
lems : the microprocessor . During the seventies, the systems produced by MonitorMaster successes had perturbed the senior managers of the company . I was called in to do an
were predominantly hardware-oriented; occasionally a system was developed which was autopsy on these projects . After about an hour it was clear what the problem was : that
MonitorMaster had no effective software quality system .
212 All the projects that failed exhibited signs in their early stages that trouble could
214 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION CASE STUDY 215

be expected : signs that a good risk analysis could have picked up-at least two of the planning, risk analysis and requirements specification . Because of this, the three projects
electronic engineers who worked for the company were aware of some hardware problems failed in that a profit well in excess of .£1200000 could have been made, when in fact for
that the supplier of the device for the first project had experienced in the past, and had the three projects a loss of £200000 was made . This alone made the managing director
formed the opinion that this was not a reliable supplier . It was clear at the initial meeting determined to implement some form of quality system .
with the customer who was involved with the second project that his requirements were
very fluid indeed, and it was also clear during the early stages of planning for the third
project that there were some worrying aspects of the statement of requirements provided 15 .2 FIRST STEPS
by the German customer .
The problem was that no adequate risk analysis was carried out which could have The senior management of the company decided that they wanted a software quality
resulted in proper project planning and quality controls and, furthermore, the quality system . My post-mortem report had listed about 50 deficiencies in developmental and
controls were not available as the software quality assurance system was rudimentary : quality assurance practices, and the result of these in terms of lost resources . The initial
the quality manual extracts for software occupied ten pages, while the hardware part of decision by the company was to contract the development of the quality system in two
the same manual was over 150 pages in length . stages . The first was a design study in which the broad outline of the components of the
I submitted an initial report which described what the problems were, and how they quality system were sketched out, together with an indication of how the quality system
could be avoided . The report did not concentrate on a large amount of technical details, would affect the working practices of the company ; the second was an implementation
but did provide detail on three of the 50-odd problems that I encountered to give a stage, in which the details of the quality system were specified in terms of a quality

flavour of the depth of the quality assurance difficulties . What the report did focus on manual and were introduced into the company software projects .
was how poor software quality assurance was affecting the business of the company-in The decision to carry out the introduction of a quality system in two stages was
the long term, its future growth, and in the short term, its ability to resist an economic made because the senior management of the company wanted to know what they were
downturn in British industry . I did not make the mistake which I often see in post-mortem committing themselves to, in both financial and organizational terms, before a huge
reports of hiding a very strong message about the business effects of poor quality behind amount of resources were spent . As a by-product of the process of carrying out the post-
a massive amount of technical detail . mortem on the three projects, I advised the company that their staff technical mix in
In the first project, a risk analysis procedure would have asked searching questions terms of hardware and software experience was wrong : that hardware staff dominated in
about the hardware and provided advice about what to do if there was a suspicion that terms of numbers at a time when the software component of their projects was increasing

the hardware was going to be delivered late . The results of the risk analysis would have led at a rapid rate . As a consequence of this observation, the company made a decision that
to MonitorMaster bidding more for the project to take into account possible manpower when hardware staff left they were to be replaced by software staff until the mix was
overruns at the end of the project . It would also have led to them concentrating on approximately 25 :75 per cent hardware to software . Also, when interviews for new staff
including a large amount of maintainability quality attributes in the system . For example, were held, the company made a big effort to interrogate potential staff members about

the programming language C was used for implementation, and a good information- their experience and views on quality systems . During 1988, when the first part of the
hiding standard could, I estimated, have reduced the rework costs by a minimum of 30 quality system design and implementation took place, four hardware engineers left, two
per cent . of them joining the company that had been set up by ex-hardware engineers who worked

In the second project, a good risk analysis would have detected the chance of volatile for MonitorMaster. They were replaced by four software engineers, three of whom worked
requirements being present . Again, this would have resulted in the developer building for large software developers who had well defined, mature quality systems .
contingency into the project cost, and also organizing the project to address the main- The first half of the quality system implementation-the design phase-was commis-

tainability quality attribute . Moreover, a proper change handling system incorporating a sioned by the company, and I started work in 1988 . I expected it to take about three to

costing procedure which gave each change a monetary cost that had to be confirmed and four weeks . The first step I took was to form a steering group, which would nominally be
paid for by the customer, would have made the project a very profitable one . It would responsible for the design of the quality system . This group consisted of four members
have dissuaded the customer from asking for some of the more outlandish changes that of staff .
The first member was a project manager who effectively represented the board of
were requested .
Again, in the third project, a good risk analysis would have shown that the project directors' viewpoint . I felt it was important that very senior members of staff were not
had a high probability of getting into trouble . As well as the measures which could have seen to take part in the design of the quality system . The main reason for this is that when
been adopted for the first two projects, the developer might have been prompted to use a company has problems with aspects of development such as errors, budget overshoots,
a limited form of prototyping, such as employing a screen generator to produce an early and unsatisfied requirements, the blame does not lie mainly with the technical staff: there

version of the human-computer interface, to check requirements well before development are often major problems with the attitude of senior staff to quality . For example, senior
started . staff who are neurotic about profit will order a project to skimp on system testing if a

The problem with the company was that the quality assurance manual, which sim- customer makes enough noise about delivery : in the short term this might seem to make
ply consisted of a few, not very well thought out standards, provided no guidance for economic sense ; however, in the long term it loses money .
 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION CASE STUDY 217
2 16

Both technical staff and senior staff should be actively involved in the development 15 .3 THE DESIGN STUDY
of the quality system, but I am a great believer in dealing with each group separately
as they can then be completely honest about each other's failings . I included the project It took me five weeks to complete the study . This comprised a number of sections .
manager to represent the directors' point of view. The person chosen was fairly close to
the financial circumstances of the company and could, if required, point out the financial 15 .3 .1 Introduction
implications of a decision to the steering group . The second reason for picking the project
manager was that he represented a major group of users of the future quality system. This introduced the remainder of the study ; gave an outline of what a quality system
The third reason, specific to the company, was that he had a hardware background consisted of (similar to that contained in this book) ; and described three case studies .
and had managed one of the projects that had failed . He was the link to the hardware These case studies outlined deficiencies in the rudimentary QA system currently used by
engineers who, at the start of 1988, when hardware staff were being replaced by software MonitorMaster . The three studies involved poor specification notations being employed
staff, were beginning to sense, wrongly, that they were being threatened by a software by company staff, the inadequate configuration management system that was currently
quality system . The project manager was there because he talked their language and being used, and the poor quality of the unit testing phase . They outlined the problems
could reassure them . which had occurred in the past ; for example, for the configuration management case
The second member of the steering group was one of the 1988 hirings . He was an study, the degree of miscommunication that led to different designers having different
analyst who had worked in London for a very large software company with a well de- versions of the functional specification was described . In each of the case studies the
veloped, if rather bureaucratic, quality system, and had moved back to the area where financial impact was outlined (I deliberately chose three areas where the financial impact
MonitorMaster was based . He was on the steering group because he could at least pro- was heavy) . Each of the three case studies concluded with an outline description of
vide confirmation to the sceptics that quality systems were not a hindrance to creativity, the future possible QA changes which would have led to significant savings . The message
but something to help staff . given by this section was that a quality system was an investment and, in a comparatively
The third member of the group was a hacker . He was a highly skilled electronic short time, could pay large dividends to the company .
engineer who produced some of the fastest C code that I had ever seen, but whose code
was grossly unreadable : he tended to reject any facility in C-and there are not many of 15 .3 .2 Major deficiencies
them-which promoted readability . He had, however, worked on the third project which
had gone very wrong . His presence on the steering group was important : MonitorMaster During my analysis of the company's approach to quality I discovered about 50 deficien-
produced the sort of fast-response system which required very efficient coding, and I cies, which were listed in this section . Each was described in no great length ; on average
thought that it was important to include someone who would be able to provide advice I devoted about 50 words to each deficiency . They were uniquely numbered so that I
on designing a quality system which would cope with the type of coding that sometimes, could refer back to them later in the plan . An example of one of the deficiencies is shown
of necessity, had to be carried out . below :
The fourth member was a woman who carried out the customer support function .
MonitorMaster transacted their business in two areas : they produced bespoke systems dl When staff carry out unit testing the test data for the unit test is not usually retained . It is a
simple matter to do this and place the file containing the test data in the project library . Failure
from scratch, and had also produced a number of packages which were created from the to retain test data leads to extra expense if a module has to be changed .
same design but which could, by means of altering a number of tables, be oriented towards
the needs of a particular customer . These packages generated a large proportion of the
As can be seen above, each failure was accompanied by a sentence which described why
income for MonitorMaster, and the customer support function was oriented towards
fielding problems and modifying the system whenever an error occurred . the failure was serious .
The two most popular packages were quite old and were both programmed in BASIC,
a language which was once an excellent teaching language, but which was not the ideal 15 .3 .3 A process model
medium to maintain, particularly since there were no designs for the two packages . The
The major problem with the company was that it did not have a defined process model .
packages were becoming increasingly popular, but there were ominous signs that the
Many of the staff worked to what they perceived was the process model, and because
rather rudimentary configuration management system being employed was gradually
communication was quite good in the company many had in effect agreed, via informal
breaking down . I wanted someone on the steering group who maintained other people's
code who could point out the consequences of poor quality assurance and who was also working practices, what the process model for the company was . This section described
what I perceived was the process model in terms of activities and the flow of documenta-
involved in maintaining multiple versions of systems .
tion in the company . This documentation was not only existing documentation, such as
unit test standards, but also documentation such as standards, procedures and specifi-
cations which were currently non-existent . These items were differentiated from existing
items by means of dotted lines .
218 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION CASE STUDY 219

An important point that is worth repeating many times is that it is an act of folly plan some cheap but effective measures which I was confident would produce some very
to develop a quality system which is not based on even a simple process model . Without visible savings that would persuade the company to commit fully to the second part of
an idea of the way in which a company works, or wants to work in the future, there is the plan-for example, I included a programming standard which contained instructions
little chance of developing the correct components of a quality system . In effect, what I on defensive programming .
am saving is that developing a quality system in this way is analogous to producing a The final plan was one which assumed that resources were very tight . This was the
software system with a poor or non-existent system specification . worst of the plans, and was the one I hoped would not be adopted . It represented the
dilemma faced by -a quality consultant time and time again in the software industry-
when faced with a shortage of resources provided by a customer, what do you do : tell
15 .3 .4 Required components of the quality system
the customer directly that the resources available for quality assurance were inadequate,
This section described what was required of a possible quality system for the company . It or make an attempt to solve some of the customer's problems? I made the decision to
outlined the standards and procedures that were required, the staffing that was needed do the latter, but provided plenty of written warnings about the problems of this third
and the relationship of staff who carried out quality assurance activities on software plan . The plan included tasks which would lead to the improvement of the requirements
projects to the senior management of the company . One of the problems with Monitor- specification process, together with the early generation of test data . This was accom-
Master was that it was a comparatively small company, and I felt that economically it panied by a number of small, cheap measures which would provide some large gains ; for
could only justify one extra member of staff responsible for quality . There were certainly example, I advocated a programming and unit testing standard to be implemented .
no resources available for a quality director, and so I suggested that a board member Happily, MonitorMaster adopted the second plan, and work started on the imple-
should have a part-time responsibility for overseeing the quality function . mentation of a quality system about nine months after receipt of the plan . The long
Each standard and procedure was listed and related back to the process model de- gap was occasioned by the fact that the company had received a major contract which
tailed in the previous section of the plan . They were also cross-referenced back to the diverted its attention from considerations of a quality system . This contract, which again
list of 50 deficiencies detailed in the second section of my plan, so that the reader could pointed up major quality problems within MonitorMaster, both paid for the first half of
see what deficiencies a standard or a procedure would address . It was in this section the implementation of the quality system and focused the minds of the management on
that I placed great emphasis on the fact that the member of staff who carried out the implementing the second half as soon as possible.
day-to-day quality function should have quite a degree of power over individual projects ;
for example he or she should be able to insist that a project adopt the mandatory parts
15 .3 .6 Tasks
of the quality system, such as the system specification standard that was to be developed
as part of the future quality system . However, I also stressed that, although individual This section included a list of tasks which had to be carried out either by consultants or
project managers had a right of appeal to the director who was partly responsible for by staff of the company. Typical tasks were :
quality assurance, it would be unusual for many appeals to be upheld .
• Write standard X
• Write procedure Y
15 .3 .5 Implementation plan • Write a statement of responsibility for Director Z
This section gave a broad outline of how the quality system could be implemented . It was, • Review procedures P, Q and R
in fact, three separate plans which catered for a number of possible scenarios of investment • Procure documentation software
in a quality system . The first plan was one which assumed that the company was able to • Develop templates for documentation W .
commit a substantial amount of resources in one chunk . It started with activities such as
placing a quality system infrastructure in place by hiring a quality specialist, writing the At this stage I did not include detailed timings for these tasks or when they were to be
job descriptions of some staff to give them quality responsibilities, and also organizing carried out . However, the three alternative plans outlined in the previous section gave a
lines of responsibility . This was followed by plans for the implementation of standards, broad indication of the progression of the implementation .
procedures and guidelines for developmental activities in the order in which they occur in
the life-cycle that I sketched out for the company . The development of the configuration
management system was also planned to take place in parallel with the implementation 15 .4 IMPLEMENTATION
of the QA functions associated with development, and I assumed that its implementation
would be complete before the sections of the QA manual dealing with design had been The company agreed to the implementation in two parts . The first stage of this imple-
finished . mentation involved producing a more detailed plan . I took the tasks which were specified
The second possible plan assumed that resources were available but that they were in the previous section of the plan, assigned staff to these tasks, and planned which tasks
stretched over a longer time period . In effect, I split the first plan into two halves, with had to be carried out in series and which could be carried out in parallel . I felt that
each half corresponding to a half of the first plan . However, I did include in the first half the quality system which was going to be devised would only succeed if there was full
 CASE STUDY 221
2 20 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

involvement of the MonitorMaster staff . To that end, I arranged that any tasks which ings are being made-savings which have prevented MonitorMaster, in a time of deep
I envisaged being carried out by a consultant would have a member of staff from the recession, from laying people off and retrenching their business commitments . However,
company shadowing him or her . The role of the shadow would either be to help in the the recession has had an effect in that it was originally envisaged that implementation of
task allocated to the consultant, for example help to write a standard for unit testing, the second half of the quality plan would start six months ago . The company made the
or have a role in reviewing the work of the consultant . MonitorMaster was distinguished decision to delay it by nine months . However, at the time of writing their commitment to
by the fact that many of its staff had very little experience of software quality systems . a start in three months time has already been shown by the fact that they are beginning
Consequently, I envisaged that most of the initial work of the development of the quality to plan longer projects to allow some of the key staff involved in the implementation of
system would be carried out by consultants . However, some of the recent signings to the the second half of the quality system not to have a full commitment to these projects .
company had excellent experience of modern quality systems, and so I allocated them to
the production of some of the elements of the quality system .
All the work produced-introductory documents, standards, procedures, guidelines, 15 .6 SUMMARY
statements of responsibility, etc .-was reviewed and commented on by staff in the com-
pany. Two of the best reviewers of this material were programmers who had been with There are a number of important points that arose from the MonitorMaster project
the company for some time and who, when I met them initially, were totally against the which are universal :
concept of a quality system . Both voiced objections that such a system would totally
stifle their creativity. In conversation with a senior member of staff, I discovered that the • As with many software projects, the problems encountered by MonitorMaster mainly
company was looking to assign staff to maintaining an existing system which had been occurred during the early stages of a project : during requirements analysis, system
developed with little concern for quality-a system which was grossly unstructured and specification, and project planning .
almost impossible to read . I suggested that the two members of staff might, as part of • Always concentrate on business issues in any evaluation of a software quality system .
their responsibilities, be asked to help out in the maintenance . This is what happened, By all means give technical examples of where, for example, money is being lost, but
and I found that within a short time they became pretty hawkish over quality . Days spent the important message that should come from a report on poor quality is how it can
poring over printouts which, to say the least, were unhelpful, developing new test data, radically affect the business health of an enterprise .
and trying to understand the functional characteristics of a monolithic software system, • From the first, it is important to involve staff in the development of a quality system .
turned them into hard-line quality propagandists . Indeed, their experience of maintain- As far as possible they should feel that it is their system-only then will they respect
ing a disastrous system made them less efficient than they could have been initially, as it and work under its constraints .
they tended to write standards and procedures which were exceptionally detailed, and • You cannot even think of implementing a quality system unless a company has at
which gave the users of those standards and procedures no leeway at all . However, they least one process model-it is equivalent to developing a software system without a
quickly became valued members of the team allocated to the process of developing a requirements specification .
quality system . • There are some cheap but effective quality changes such as a defensive programming
The process of writing standards and procedures was a three-stage one . First, the standard which will bring savings almost immediately and which, if implemented, can
staff involved in an activity were interviewed and asked what they thought the problems persuade a company to go for a full implementation of a quality system .
were in that activity ; a questionnaire was used for some of the tasks . A consultant then • The implementation of a software quality system is a project, and should be treated
went away and wrote a standard or procedure . This was later reviewed by the staff like any other project : it should be planned, monitored and controlled .
involved in the task . If the comments from the staff were minor, then the document was • Assignment to the maintenance function, particularly if the maintained software is of
revised and frozen . If there were major worries, the document was revised and then re- poor quality, can radically change development staff's perception of the utility of a
reviewed . It is worth stressing yet again the importance of always involving development software system .
and managerial staff totally throughout the construction of a quality system . It is those • If a quality system is to be implemented in a number of phases, it is vital that the
companies where the staff feel they played a part in the development which have had the elements of the system which deal with front-end activities such as requirements
most successful quality systems in terms of adherence and commitment . analysis are put into place in the first phase . These elements tend to have the best
potential for achieving large savings .

15 .5 THE CURRENT POSITION

The position in MonitorMaster at the time of writing (January 1993) is that the first half
of the plan has been fully implemented, and the company is working to a quality system
which covers planning ; system specification ; early generation of system and acceptance
tests; and configuration management . It is obvious, even after a year, that major sav-
 IMPROVING A QUALITY SYSTEM 223

• Chapter 9 discussed some of the ways in which new technology-both tools and
development methods-can give rise to new requirements on a quality system . It is

16 •
important that these new requirements are detected in advance of the implementation
of new technology.
Changes in business policy and strategy can give rise to the need for major changes in
IMPROVING A QUALITY SYSTEM a quality system . For example, a company whose main business is producing bespoke
software may decide that, with users increasingly looking to package-based solutions,
the time has come to develop software packages which can be tailored by the company
to a specific customer's requirements . This could have two major impacts on the de-
veloper . First, there will be a need to tighten up the configuration management parts
of the quality manual ; these parts may be adequate for bespoke system development
but, almost invariably, are inadequate for a situation where there are a number of
versions of a system in existence . Second, design standards and procedures will need
to be tightened up in order to cope with techniques such as table-driven development
which are used for producing packages .

16 .2 THE QUALITY IMPROVEMENT TEAM

AIMS One strategy which can be adopted to cope with the improvement and evolution of the
quality system is to appoint a quality improvement team . Its brief is to continually exam-
• To outline the need for quality improvement . ine the functioning of the quality system, business policy and technological developments,
• To describe one way in which quality improvement programmes can be implemented . and to specify tasks which need to be carried out in order to develop the quality system .
Such a team would come from diverse areas in a company . First, it should contain
a senior member of the company-preferably at board level . The reason for this is that
16 .1 INTRODUCTION it is important for senior management to show their commitment to improvement of the
quality system, and, after all, they are a user of the system .
A number of companies have expended a large amount of resources on installing expensive A project manager should also be part of the team . Project management use parts of
quality systems which have remained in place for many years, without any modification a quality system different from those used by senior management and technical staff . A
to them taking place . This is a mistake-a mistake which becomes increasingly costly as member of technical staff, say a designer or an analyst, should also be part of the team,
time proceeds . There are a number of reasons-over and above the very pragmatic one together with a programmer who currently works on the maintenance of a system . If the
that the ISO 9001 series insists on it-why a company should be monitoring, reviewing company is large, then it may have an R & D department : if so, then a member of that
and evolving their quality system : examining patterns of non-adherence to the system ; department should also be part of the team . An important input into the deliberations of
examining where software errors have occurred ; looking at the effect of new technology the team is an indication of technologies which will affect the company and which are just
on the quality system ; and discovering whether changes in business strategy will have an over the horizon . If the R & D department are doing their job properly, then they should
effect . These are : be continually monitoring and disseminating the spread and effect of new technology .
The final member of the team should come from the quality assurance department .
• There will always be a tendency for staff to skimp in terms of their adherence to the The team will base its deliberations on a number of inputs :
quality system . This is not because they are unprofessional, but rather because there
is a pressure on staff to deliver on time and to budget . Staff in a hurry to deliver a Project debriefing reports . A quality system should have a standard and procedure
•
system may ignore parts of a quality plan . If a company is certified to ISO 9001, then which directs a project manager to fill in a questionnaire after a project has been
a record of non-adherence is enough for the certification to be invalidated . completed . Such questionnaires are a useful repository of experiences, and an impor-
• The first version of a quality standard is never perfect . There will be parts of the tant component in such documents are questions about the effectiveness of the quality
system which still allow an unacceptable level of errors or require staff to carry out system .
too much work. It is important, during the early stages of the use of a quality system, • Reports on technology which will affect the company in one-, two- and five-year
that particular care is taken in monitoring what are perceived to be these weak spots . time spans . As mentioned previously, these reports should emerge from the R & D
Even quality systems which have been in place for a number of years contain processes
which can be considerably improved .

222
 SOME QA QUESTIONS 225
224 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

the quality system . Having a five-point rating when there are two outcomes might seem
department . If the company is too small to support such a department, then they
should be commissioned by external consultants . strange : the reason for having it is to give the quality department, which is responsible for
implementing change to the quality system, some idea of priorities ; for example, when
• Reports from validation activities, such as reviews, which detail the extent and level
resources are scarce and a major revision has to take place . A further task, over and
of errors that are being discovered at each phase of a project . These reports will not above identification and prioritization of a change, is to specify exactly what the change
be read in a raw form ; but will be presented to the team in summary form . Normally,
is and why it is being asked for . An example of such a specification is shown below :
staff responsible for the quality function will carry out limited post-mortems on these
reports and provide some indication of the reasons why certain categories of error . We have perceived that
There is a relatively urgent need to develop a C++ programming standard
have occurred . many of our potential customers are asking for systems written in this language
. Also, one of our
• Summaries of audit reports from quality staff which indicate where deviations from projects has already used C++, and it is obvious from the reports from system testing that a large
the quality system are occurring . number of errors which have been detected have arisen from an undisciplined use of a number of

• Unsolicited letters from staff on projects who, for example, may be having trouble C++ facilities. We have also had a report from staff who maintain this system that coding styles
. We have also had a number of complaints
range from the eccentric to the pragmatic and sensible
using a standard . and suggestions from programmers on the project that they felt they were producing poor quality
• Business policy statements from the board of directors . software because of the lack of a standard
. We suggest that the starting point for staff engaged
• Questionnaires filled in by staff asking them about their perceptions of the effective- in the development of this new standard is the report from the maintenance department and the
ness of the quality system. letters from the programmers .
• If the company has implemented quality- circles, then reports from these circles should Priority 4 . Date required 9 months from now .
be used .
• The contents of suggestion boxes . It is vitally important that lists of these changes are posted up in highly visible parts of
• Reports from consultants who have been called in to periodically review the quality
a company. Much of the input to the process of quality improvement comes from staff
system . on the ground, and a company should provide visible evidence that their suggestions are
• Reports from staff who provide a servicing function . Such staff will have a good idea being taken seriously .
of some of the problems with a software system, for example the poor quality of an
interface, which may be ascribed to deficiencies in the quality system .
• Reports from quality staff who have traced back selections of errors which have been
16 .3 SUMMARY
detected during validation to their source . It is important to point out that this is
quite a time-consuming activity, and only a small number of errors can be treated One of the indications that a company has an excellent quality system is, paradoxically,
in this way . Nevertheless, if it is possible to produce this report then it can be very that they do not think it excellent, but are continually attempting to improve the system .
valuable indeed . Another indication is that in improving a quality system the company involves staff at
every level, not just those who have to implement the quality system .
It is worth pointing out that very rarely do I find a company using all the above docu-
ments; the points described above merely constitute an exhaustive list which can be
tailored to each company's particular circumstances .
16 .4 FURTHER READING
In many ways, the quality improvement team functions in the same way as a Change
Control Board, the only difference being that it is responsible for changing the quality Two excellent books which concern process improvement are Humphrey (1989) and Har-
system rather than a software system and that version control is a much easier task . rington (1991) . The former describes the process of converting a rudimentary quality
A quality improvement team will usually meet relatively infrequently ; a realistic system into a sophisticated system which represents a zenith . The latter is a book about
interval between meetings is three months . Its task is to examine the type of reports business process improvement ; however, it can easily be adapted by staff who are inter-
detailed above and decide whether any changes to the quality system are required . Once ested in software process improvement .
it has made this decision, it then needs to categorize the importance of the changes . This
is one of the reasons why the spread of people on a quality improvement team should
be as wide as possible : a team consisting solely of developers will often prioritize those
changes which affect technical activities, and may be tempted to give a low priority to
changes which relate to the business strategy of the company .
I normally advocate that changes are given a five-point rating, together with an ideal
date by when the change should be implemented . Normally, a rating of five means that
the change has to take place immediately, with all other ratings used to indicate that the
changes should be batched up so that they can be implemented in the next revision of
 SOME QA QUESTIONS 227

A .2 .2 Project monitoring
A good question to ask a project manager concerns his or her ability to monitor the
progress of a project . I usually ask whether he or she can easily monitor the progress of a
project in terms of tasks completed, tasks which are slipping, and the amount of resources
expended at a certain point in a project as compared with the amount of resources which
APPENDIX A : SOME QA QUESTIONS were planned to be expended .
If a project manager can point to procedures which enable him or her to get this
information easily, then the part of the quality system dealing with planning and project
monitoring is excellent . It means that the planning is carried out at a task level, and
there are good facilities for staff to report on task completion . This topic is dealt with in
Chapter 3 [58] .

A .2 .3 Validation of designs
It is always worth asking a developer what efforts he or she takes to validate a design ;
for example, what quality controls exist which provide evidence that a design meets
its corresponding requirements specification-both functional and non-functional . An
A .1 INTRODUCTION
answer which simply states that a senior member of staff signs off the design is not
adequate . If the developer states that methods such as technical reviews and prototyping
This appendix contains a list of twenty-five questions which will usually enable someone are used and that standards and procedures exist for these activities, then that is a good
to detect major weaknesses in a quality system . Consider it, if you like, as a short version answer . Reviews, within the context of the validation of requirements, are described in
of the questionnaires discussed in Chapter 12 . I often use it when quickly assessing
Chapter 4 [75] .
companies for a customer who wishes to place a software contract . Each question is
cross-correlated to the chapter that it is relevant to and the page number of the most
relevant part of the book . The page number is written in bold and enclosed by square A .2 .4 Quality planning
brackets . A very revealing question to ask a developer concerns the construction of the quality
plan. If the developer simply states that each project uses the company quality manual,
then this is a poor answer . The correct answer, described in Chapter 1 [22], is that
A .2 THE 25 QUESTIONS the company has a quality manual, but that the elements of the manual relevant to a
project are extracted after quality factors have been identified and a risk analysis has
A .2 .1 Functional traceability
been carried out . This process is described in Chapter 3 .
One of the first questions I ask concerns traceability, a concept detailed in Chapter 5 .
I ask a developer to show me some of the listings of the program code of a completed
project . I place my finger on a module and then ask him or her to tell me which functions A .2 .5 Review of the quality system
in the project's requirements specification that module helps to implement . An important question to ask a developer who claims to have a quality system concerns
If they can point to the functions in a comparatively short time, say less than ten
its evolution . The question is : how does the company evaluate the effectiveness of the
minutes, then it is an important sign that the company has a good quality system . If quality system and, also, how does it evaluate the potential changes that new software
the retrieval time is measured in hours, then I normally suspect a poor quality system . development technology and business policy might have on the quality system? A quality
Chapter 1 [18] described why traceability is important : staff in a software project are
system is an evolving entity, and it is important that mechanisms are in place to evaluate
continually tracking between requirements specifications, designs and code, and a quality
what the degree of evolution should be . A company should receive information such as
system with standards which insist that a high degree of traceability is maintained will project debriefing reports, error statistics, and project audits which should be periodic-
encourage a reduction in errors and an increase in efficiency . Traceability is also dealt ally reviewed and then used as input into the process of evaluating the effectiveness of
with in more detail in Chapter 5 [91) .
the quality system-both in the present and in the future . This process is described in
Chapter 16 [222) .

226
 SOME QA QUESTIONS 229
228 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

A .2 .6 Unit test records A .2 .9 Configuration management practices

A question concerning unit testing is : what records exist which prove that a programmer It is worth investigating how the process of change is managed . A favourite question of
has actually carried out a test? Many quality systems do not provide enough standards mine is : show me what happens when an error is discovered during operation and it neces-
. This
for this part of the software project . If it is not obvious that a programmer has carried sitates changes to .,the requirements specification, system design and program code
is a question about configuration management : the process of documenting, appraising
out testing, but that all that might have happened is that a clean-compiled module has
been passed to staff who carry out integration, then there is a weakness in the quality and implementing changes to a system . In answer to this question, the developer should
system : it allows programmers to pass untested modules to staff who carry out system point to standards and procedures which describe how errors are notified to the software

and acceptance testing at a later stage in the project . developer ; how any proposed changes arising from the error are communicated to staff in
Discovering errors in a module when it is embedded in a whole system is many times charge of configuration management ; how a decision to carry out the change is made ; how
more difficult than discovering errors in a module when it is tested by itself . A good the change is applied and validated ; and, eventually, how staff who should know about
quality manual would include directives which ask the programmer to provide evidence the change have details communicated to them . This process is described in Chapter 8
that a module has been tested ; for example, by detailing the data that was used, and [122] .
including the output of the tests with the other module documentation that is generated .
Quality assurance and unit testing is dealt with in Chapter 7 [109] .
A .2 .10 Training

One of the aims of project planning is to ensure that staff who are qualified to carry out
A .2 .7 Planning for system and acceptance testing a task are experienced in that task or, as a minimum, have been trained in that task . If
a company does not keep adequate records of training-whether carried out externally
A very revealing question to ask about system and acceptance testing is : when does your
or internally-a project manager is unable to access important information needed for
quality manual insist that you start planning and thinking about system and acceptance
: do you keep training records and could
tests? If the answer is that this occurs during the later stages of the project, say, during planning . The question to ask in this case is
I have a look at them? It is worth looking at these records to see what training the
programming, then this is a poor answer . The correct answer is that as soon as the
requirements specification has been validated as being correct the system and acceptance company regards as useful : if staff are just sent on hacking courses on languages such as
tests should be developed in outline . C, then this is not a very good sign . However, if staff are sent on a variety of courses
There are a number of reasons for this : the project manager wants to know about on topics such as requirements specification, system design, project management, quality
the level of resources required for system and acceptance testing, and can only calculate assurance and testing, then that is a much better sign . Planning is dealt with in Chapter

this if he or she has a good idea of what the testing involves ; there may be a need for 3 [42] .
special-purpose hardware for testing which needs to be fabricated in advance ; and there
could also be a requirement for special-purpose testing tools which need to be specified
A .2 .11 Integration testing
and developed in advance .
Integration testing is a term used to describe the process of validating a system as it is
However, the main reason for developing the tests early, in outline form, is that they
give the project manager confidence in the requirements specification . If a good tester is being built up incrementally, with coded modules being added a few at a time . Developing
a system in this way enables errors to be discovered more efficiently than if the system was
able to generate a series of outline tests from a requirements specification, without too
not integrated and just collected together prior to system testing . The reasons for this are
many problems, then the requirements specification-a key document in the software
project-is in good shape . The derivation of these outline system and acceptance tests- described in Chapter 1 [19], while Chapter 7 [113] describes standards and procedures for
known as verification requirements-is dealt with in Chapter 4 [79] . testing . A good question to ask is : have you got standards and procedures to describe the
process of integration testing which, for example, provide advice on integration strategies
and describe the documentation to be generated from integration testing?
A .2 .8 Risk analysis

A question concerned with project planning is : what standards and procedures does your A .2 .12 Procedures for costing
quality manual contain which deal with the identification of risks? All projects are risky,
some more than others . A good quality system should provide standards and procedures A good question to ask is : are there standardized procedures which every project has to
use in establishing a cost estimate? It is important that projects use the same costing
which direct the processes of organizing and planning a project based on an appraisal of
. This may
the possible risks in the project . The effect of events which are risky can be mitigated technique, as the results from previous costings can be used for future projects
not seem to have much to do with quality ; however, projects which make erroneous costs
through careful planning. A quality manual should provide procedures for analysing the
usually start running out of budget during testing and often have to skimp on testing .
risks in a particular project and guidelines which provide advice on how to minimize the
effect of risk . This is dealt with in Chapter 3 [45] . This topic is dealt with in Chapter 3 [55] .
230 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION SOME QA QUESTIONS 231

A .2 .13 Checking costs A .2 .18 Traceability into testing

It is important that there are standards and procedures which enable a project manager A question which pinpoints problems in testing is : can you easily trace from a function,
to check the estimated cost of a project . Searching questions should be asked about the expressed in your requirements specification, to the individual tests which check out
existence of such standards and procedures . For example, are there modern metrics-based that function? If the company can retrieve these tests from past project documentation
costing methods whereby a project manager can compare the cost of part of a finished in under 15 minutes, then there is a good chance that the part of the quality system
system which is similar to part of a proposed system by looking at historical costing concerned with testing is in good shape . Traceability was dealt with in Chapter 1 [18],
data, or even the actual amount of resources spent on the finished system? and testing is dealt with in Chapter 7 [106] .

A .2 .14 Early checking of a system A .2 .19 Rationale for decision making

An excellent question to ask a company is : when is the earliest point in a project that a It is important that when a major decision is made in a software project the responsible
system and its documentation are checked against customer requirements? The answer to person provides a rationale, or set of reasons, for making the decision in the way that it
this should be that the requirements specification is checked just before it is frozen, and was made . You should closely question a developer to check whether standards for project
the developer is happy with it ; the validator may use a technique such as a requirements organization, system design and module testing include a directive which specifies that
review or prototyping . Errors detected early are much cheaper to rectify than if they were the staff carrying out these activities provide this rationale . For example, a designer
detected at a later stage in a project . Requirements reviews are described in Chapter 4 should be asked to explain why a particular design was adopted for a system . By asking
[75] ; prototyping is described in Chapter 9 [133] . staff to explain themselves, a quality system encourages them to think more deeply about
a task .
A .2 .15 The storage of test data
A .2 .20 Standards and procedures for auditing
A question that is intended to discover how seriously a company takes maintenance is :
do your standards insist that, as far as possible, test data for module, integration, system A question which often enables you to discover how seriously a company takes quality
and acceptance testing is stored? This ensures that if retesting occurs, then little extra is : do you have explicit standards, procedures and guidelines which govern the process of
effort is expended in redeveloping old test data . Quality assurance and testing is described checking that projects follow standards and procedures? They should cover the process
in Chapter 7 [106] . of auditing, when to audit, how frequently to audit, what reports are generated from the
auditing process, and what actions are taken when a project fails an audit . I have come
across quality manuals which contain good standards for activities such as system design
A .2 .16 The effect of new technology and requirements analysis, yet omit any mention of auditing .
As described in Chapter 9 [133], new technology can give rise to major new demands on
a quality system . It is worth questioning a developer as to whether he or she monitors
A .2 .21 The structure of the functional specification
the growth of new technology, and ascertains whether this technology threatens to affect
the quality system . This is dealt with in Chapter 16 [222], which details steps for the It is worth exploring the standards that a developer uses for the requirements specifica-
improvement of the quality system in response to a number of factors-one of which is tion . A good question to ask is whether the functional part of the requirements specifi-
new technology. cation is structured in such a way that all functions which are connected with each other
are physically adjacent-whether, for example, the functional requirements concerned
with stock ordering in a purchasing system can be found together . A requirements speci-
A .2 .17 Standards and procedures for the human-computer interface fication organized in this way is much easier to access by both software development
Systems are often built which, although they satisfy functional requirements, ignore the staff and the customer and, hence, gives rise to fewer errors . This topic is dealt with in
capabilities, environment and skills of the user . You should closely question a developer Chapter 4 [83] .
about standards and procedures which govern the process of discovering user characteris-
tics, and about guidelines for designing the human-computer interface. Chapter 10 [147]
A .2 .22 Defect data
details the quality assurance aspects of developing the human-computer interface.
Validation of a software system gives rise to defect data which describes the errors that
have occurred and their seriousness . A good question to ask is : what happens to this
data? Is it, for example, used to improve the technical processes employed by the software
developer and also the quality system? This topic is outlined in Chapter 16 [222] .
232 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

A .2 .23 Design guidelines

Design can have a major effect on the amount of resources expended during maintenance
and the number of residual errors that occur during this process . A good quality system
should have design guidelines which detail what practices are regarded as good or poor
design ; for example, it might state that a design which contains multi-functional modules
is poor . You should ask about the existence of these guidelines and read them thoroughly . BIBLIOGRAPHY

A .2 .24 Test failure

Ask the testers in the company what happens when an acceptance test fails . The wrong
answer is : we rectify the error, repeat the test and carry on . The correct answer is : we
rectify the error, repeat the test and then re-execute those tests which exercise modules
that have had to be changed in order to rectify the error . There is a high probability
that a change necessitated by the discovery of an error will have caused further errors
which can only be discovered by rerunning previous tests . The procedure for acceptance
testing should insist that this always happens . System testing and acceptance testing are
dealt with in Chapter 7 [106] .
A J Albrecht and J R Gaffney . Software function, source lines of code, and development effort
prediction : a Software Science validation . IEEE Transactions on Software Engineering,
A .2 .25 Project debriefings 9(6) :639--648, 1983 .
C Ashworth and L Goodland . An Introduction to SSADM. McGraw-Hill, 1990 .
Project debriefings are a useful source of information used for improving the quality `fir A Babich . Software Configuration Management . Addison-Wesley, 1986 .
system . You should closely question a company about the existence of standards and V R Basili and H D Rombach . The TAME project : towards improvement-oriented software
environments . IEEE Transactions on Software Engineering, 14(6) :758-773, 1988 .
procedures governing the conduct of a debriefing meeting and the associated documen- P Bell and C Evans. Mastering Documentation . John Wiley, 1989 .
tation that is produced . This topic is dealt with in Chapter 16 [223] . M Ben-Menachem . Software Configuration Management Guidebook . McGraw-Hill, 1994.
J L Bentley. Writing Efficient Programs . Prentice-Hall, 1988 .
D Benyon, J Preece et al . A Guide to Usability . Addison-Wesley, 1993 .
E H Bersoff . Elements of software configuration management . IEEE Transactions on Software
Engineering, 10(1) :79-87, 1984 .
B Boehm . Software Engineering Economics . Prentice-Hall, 1981 .
B W Boehm . Verifying and validating software requirements and design specifications . IEEE Software,
1 :75-88, 1984 .
B W Boehm . Software Risk Management . IEEE Press, 1989 .
G Booch . Object-oriented development . IEEE Transactions on Software Engineering, 12(2) :211-221,
1986 .
F P Brookes . The Mythical Man-Month . Addison-Wesley, 1975 .
P W Chan . Installing an ISO 9001 accredited software quality management system . In G Staples,
M Ross, C A Brebbia and J Stapleton, editors, Software Quality Management, Proceedings of 1st
International Conference on Software Quality Management, pages 13-26, 93 .
R N Charette . Software Engineering Risk Analysis and Management. McGraw-Hill, 1989 .
R H Cobb and H D Mills . Engineering software under statistical quality control . IEEE Software,
7(6) :44-54, 1990 .
J L Connell and L Brice-Shaffer . Structured Rapid Prototyping. Yourdon Press, 1989 .
P B Crosby . Quality is Free . McGraw-Hill, 1979 .
A M Davis. Software Requirements, Analysis and Specification. Prentice-Hall, 1990 .
R Dunn and R Ullman . Quality Assurance for Computer Software. McGraw-Hill, 1982 .
R C Durst, G E Stark and T M Pelnik . Evaluation of software testing metrics for NASA's mission
control centre . Software Quality Journal, 1(2) :115-132, 1992 .
C Easteal and G Davies. Software Engineering : Analysis and Design. McGraw-Hill, 1989 .
M E Fagan . Advances in software inspections . IEEE Transactions on Software Engineering,
12(7) :744-751, 1986 .
N E Fenton . Software Metrics : A Rigorous Approach. Chapman and Hall, 1991 .

233
 BIBLIOGRAPHY 235
234 SOFTWARE QUALITY ASSURANCE-A STUDENT INTRODUCTION

Prentice-Hall, 1993 .
J M Gilchrist . Project evaluation using the sei method . Software Quality Journal, 1(1) :37-44, 1992 . P Sully. Modelling the World with Objects .
C R Symons, Software Sizing and Estimating . Mk 11 FPA . John Wiley, 1991 .
A C Gillies . Software Quality- Theory and Management. Chapman and Hall, 1992 . Software Documentation . Scott, Freeman and Co .,
P A Williams and P S Benson . Writing Effective
P Goodman . Practical Implementation of Software Metrics . McGraw-Hill, 1993 .
E Cowers. Fowler's Modern English Usage . Oxford University Press, 1988 . 1990 .
. Prentice- all, 1979 .
R B Grady and D I Caswell . Software Metrics: Establishing a Company-wide Program . Prentice-Hall, E Yourdon and L Constantine . Structured Design
1987 .
M H Halstead . Elements of Software Science . Elsevier - North Holland, 1977 .
P G Hamer and G D Frewin . M . H . Halstead's Software Science - a critical evaluation . In Proceedings
6th International Conference on Software Engineering, pages 197-206, 1982 .
H J Harrington . Business process improvement. McGraw-Hill, 1991 .
W T Harwood. B Cohen and M I Jackson . The Specification of Complex Systems . Addison-Wesley,
1986 .
W Heitzel . The Complete Guide to Software Testing . QED Information Sciences, 1985 .
J Hemington. ISO 9001 : Moving from requirement to reality . In G Staples, M Ross, C A Brebbia and
J Stapleton, editors, Software Quality Management, Proceedings of 1st International Conference
on Software Quality Management, pages 3-12, 1993,
S Henry . Software metrics based on information flow . IEEE Transactions on Software Engineering,
7(5) :510-518, 1981 .
D S Hinsley and K H Bennet . A process modelling approach to managing software process
improvement . In G Staples, M Ross, C A Brebbia and J Stapleton, editors, Software Quality
Management, Proceedings of 1st International Conference on Software Quality Management,
pages 189-204, 1993 .
W S Humphrey . Managing the Software Process. Addison-Wesley, 1989 .
D C Ince . Software Development : Fashioning the Baroque . Oxford University Press, 1988 .
D C Ince . Object-oriented Software Engineering with C++ . McGraw-Hill, 1991 .
S Jenkins . The Times Guide to English Style and Usage . Times Books, 1992 .
C Kaner . Testing Computer Software. TAB Books, 1989 .
M I Kellner, B Curtis and J Over . Process modelling . Communications of the ACM, 35(9) :75-90, 1992 .
B A Kitchenham. Software metrics . In P Rook, editor, Software Reliability Handbook . Elsevier, 1992 .
R 0 Lewis . Independent Verification and Validation. John Wiley, 1992.
J Martin and C McClure. Structured Techniques: The Basis for CASE . Prentice-Hall, 1987 .
T J McCabe . A complexity measure . IEEE Transactions on Software Engineering, 2(4) :308-320, 1976 .
J A McCall, P K Richards, and G F Walters . Factors in software quality . Technical report, Rome Air
Development Center, 1978 .
J A McDermid . Software Engineer's Reference Book . Butterworth-Heinemann, 1990 .
S N Monanty . Software cost estimation : present and future . Software Practice and Experience,
11(2) :103-121, 1981 .
G J Myers. The Art of Software Testing . John Wiley, 1979 .
M A Ould. Quality control and assurance . In J A McDermid, editor, Software Engineer's Reference
Book . Butterworth-Heinemann, 1990 .
M Page-Jones . The Practical Guide to Structured Systems Design . Prentice-Hall, 1988 .
D L Parnas and P C Clements . A rational design process : how and why to fake it . IEEE Transactions
on Software Engineering, 12, 1986 .
R S Pressman . Software Engineering . A Practitioner's Approach-European Edition . McGraw-Hill,
1994.
F J Redmill . Dependability of Critical Computer Systems 1 . Elsevier Applied Science, 1988 .
P Rook . Project planning and control. In J A McDermid, editor, Software Engineer's Reference Book,
Butterworth-Heinemann, 1990 .
M J Shepperd and D C Ince . The multi-dimensional modelling and measurement of software designs .
In Proceedings ACM Annual Conference, 1990 .
B Shneiderman . Designing the User Interface . Addison-Wesley, 1987 .
T Snyder, W S Humphrey and R Willis . Software process improvement at hughes aircraft . IEEE
Software, 8(4) :11-23, 1991 .
D A Stokes. Requirements analysis. In J A McDermid, editor, Software Engineer's Reference Book .
Butterworth-Heinemann, 1990 .
W Strunk and E B White . The Elements of Style . MacMillan, 1979 .
 INDEX 237

Coding standard, 77 Data requirements specification standard, 84

Cohesion, 161 Data specification, 88
Combinatorial explosion, 19 Database . 12
Command and control system, 122 Database system, 47
Command-based system, 80 Debugging, 19, 158, 160, 180

INDEX Commercial data processing system, 82, 88, 194 Debugging code, 177, 180
Defect data, 170, 231
Communicational complexity, 57
Communicational overhead, 53 Defect reporting system, 186
Communications system, 143 Defect statistics, 10
Conceptual architecture, 138 Defence system, 122
Concurrency, 167 Defensive programming, 102, 180
Configuration auditing, 129 Defensive programming code, 102
Defensive programming standard, 221
Configuration bulletin, 129-131
Configuration control, 54, 127, 128, 199, 207, 208 Design control, 207
Configuration identification, 127 Design directive, 28, 65, 86
Configuration item, 54, 127-129 Design evaluation, 137
Configuration item change trail, 129, 131 Design guideline, 233
Configuration item change trail standard, 131 Design methodology, 200
Configuration management, 53-54, 123-124, 129, Design metric, 50
Design notation, 88, 96
197, 207, 229
Acceptance test, 18, 36, 38, 51, 65, 75, 92, 116, Configuration management system, 18, 53-54, Design review, 36, 94, 175, 178-179, 184,
Boolean condition, 102, 166
198, 203, 209, 228 123-124, 126, 131, 165, 198, 203, 216, 217 186-187, 202
Bottom-up integration, 37, 113
Acceptance test data, 127 Design review procedure, 186
BS5750, 55 Constraint, 3, 12, 66, 109
Acceptance test suite, 36 Design specification, 71, 103
Bug bounty hunter, 17 Constraint acceptance test, 117
Acceptance testing, 15, 17-19, 34, 38, 92, Constraint system test, 117 Design standard, 179
Business manager, 13
106-109, 116-117, 120, 134, 142, 144, 201, Contingency, 10 Design validation, 91
208, 228, 233 Contract review, 194, 206 Design validation procedure, 97
C, 1, 8, 31, 86, 88, 107, 141, 214, 216, 229
Acceptance testing procedure, 120 Detailed design, 30, 34, 60, 99-101, 104, 126, 197
C function, 2 Contradiction, 12, 65, 80, 143
Accounting package, 5, 17, 45, 123 Control board minute, 129 Detailed design metric, 160
C++, 8, 141, 225
Activity network, 59, 60 Control board minute standard, 131 Detailed design notation, 87, 101, 104, 186
Calling structure, 87
Ada, 30, 107, 141 Control construct, 30, 102 Detailed design procedure, 104
CASE technology, 144, 151
Adaptability, 83 Detailed design review procedure, 104
CASE tool, 145, 193 Control flag, 176
Adaptive change, 5 Detailed design review standard, 104
Change authorization note, 129-130 Control flow, 158
Air-traffic control application, 69 Control of nonconforming product, 202 Detailed design standard, 100, 104
Change authorization standard, 131
Algorithmic complexity, 93 Control structure, 103, 176, 179 Development method, 61
Change control, 198
Ambiguity, 6, 12, 65, 80, 143 Controlling, 9-10 Device handler, 5
Change control board, 54, 124-126, 128-130, 224
Analyst, 13, 53 Direct observation, 154
Change control board minute, 130 Corrective action, 203, 209
APL, 135 Corrective change, 5 Document control, 197, 207
Change control procedure, 131
Application-oriented programming language, 135 Correctness, 5-6, 8, 23, 48 Documentation, 60, 61
Change history data, 207
Assembler language, 30, 100, 101 Cost accounting, 56 Documentation service, 46
Change notification note standard, 131
Attribute, 69-71 DOD, 171
Change proposal, 129 Cost checking, 230
Audit, 77 Cost estimate, 142, 229 Dynamic analyser, 179, 201
Change request note, 124, 129, 130
Audit report, 224 Costing, 43, 56 Dynamic memory management, 140
Change validation procedure, 131
Audit trail, 13, 203 Coupling, 161
Change validation report, 129-130
Auditing, 21, 194, 206, 231 E-R diagram, 31, 71
Change validation report standard, 131 Critical path, 59
Auditory feedback, 147 Efficiency, 7
Checklist, 13 Cross-referencing, 50
Availability, 83 Customer requirements, 5, 13, 15, 20, 34, 230 Emergency display, 72
Chemical monitoring system, 102
Emergency operation, 71
Class, 86, 135, 136, 139, 140, 142 Customer-developer interaction procedure, 84
Banking, 20 . 31 Entity, 31-32, 69-71
Class library, 142 Customer-developer interaction standard, 84
Baseline, 127, 131 Entity life-history, 32
Class review, 142 Cyclomatic complexity, 160
Baselining, 54 Entity-relationship diagram, 31
Clerical application, 8
BASIC, 216 Data architecture . 12, 31, 93, 96, 165 Equivalence partitioning, 111
COBOL, 11, 29, 31, 33
Bespoke software, 223 Data design, 101 Error condition, 102
Code metric, 160
Bidding, 4 Error guessing, 111
Code review, 92, 104 Data design standard, 97
Black box testing, 106 Data flow diagram, 33, 68, 158 Error rate, 80
Code review procedure, 104
Black team, 17 Data invariant, 144 Error report, 13
Code review standard, 104

236
 238 INDEX INDEX 239

Error reporting procedure, 185 Key field, 33 Operating system, 22, 45, 49, 118
HCI design, 151-152, 155
Error statistic, 227 Outline architecture, 48
FICI design rationale . 150
Estimating, 10 Layout convention, 11, 102 Outline design, 48
HCI design rationale procedure, 155
Evolutionary prototyping, 136, 137 Life-cycle, 44 Outline design standard, 48
HCI design rationale standard, 155
Expenditure record, 10 Life-cycle model, 60 Outline requirements specification, 195
HCI design review, 152-153
External hardware, 72 Line-based command, 68 Outline system test, 38
HCI design review procedure, 156
External interface, 94 Line-by-line interface, 4 Outline test, 228
HCI design review standard, 156
External milestone, 61 HCI design specification, 152 LISP, 35, 135 __
External quality standard, 21 Package, 86-87, 107, 113
HCI evaluation, 153
External software, 72 Mainframe computer, 15-16, 44 Package testing, 107-109
HCI evaluation report, 155
External software contractor, 10 Maintainability, 5-6, 17-18, 23, 48, 93, 165 Parameter, 158
HCI evaluation report standard, 156
External software interface, 72 Maintenance, 5-6, 53, 102, 122, 139, 159, 175, Pascal, 28, 86, 88, 89
Help facility, 72, 80
External standard, 50, 60 220-221,223 Penalty clause, 45
High-level language, 30
Maintenance support, 28 Perfective change, 5
Historical cost data, 230
Fagan inspection, 77 Management responsibility, 192 Performance, 83, 93
Human interface, 72
Fagan, M .E., 77 Management review, 193 Personnel department, 10
Human-computer interface, 45-46, 113, 137, 154,
Feasibility study, 195, 206 Marketing department, 10, 15 Planning, 9, 43
214, 230
Federal Systems Company, 170 Memory constraint, 36 Platitude, 6, 12, 65
File comparator, 134 Metric, 4, 6, 158, 159 Polymorphism, 139-140
IBM, 170-171
Financial application, 7, 49 Implementation directive, 28, 86 Metric-based costing technique, 57 Portability, 4, 6, 22-23, 30
Financial contingency, 45 IncrementaLprototyping, 136-137 Metrics research, 158 Portability testing, 22
Financial law, 65 Middle-out integration, 113 Predictor metric, 159
Independent test group, 119
Financial package, 65, 126 Milestone, 61 Procedure, 3, 9-10, 17-18, 21-22, 24-25, 43, 50,
Independent validation, 16
Fitness for purpose, 3-4, 6 Military command and control, 69 79, 94, 167, 174-175, 181, 188, 194-195, 218,
Information hiding, 22, 139, 214
Flow chart, 30 Moderator, 75-77 220, 227, 229, 231
Information system, 27
Formal method of software development, 143 Information system development, 69 Modifiability, 5 Process, 165
Formal specification, 143 Modula 2, 107 Process architecture, 12
Inheritance, 135, 140
FORTRAN, 29, 86 Module, 12, 17, 28, 39 Process assessment, 171
Innovating, 9-10
Forward traceability, 18 Module hierarchy, 48 Process constraint, 66
Inside-out integration, 37
Module review, 142 Process control, 200, 208
Fourth-generation language, 31, 33, 34-35, 47, 86, Inspection and test status, 202
88, 135 Module specification, 30, 99, 109, 157, 168, 181 Process description, 169
Inspection and testing, 200
Function, 39, 66-67, 109 Module testing, 22, 231 Process improvement, 13, 166
Inspection, measuring and test equipment, 201,
Functional acceptance test, 117 Monitoring, 9-10 Process model, 165, 166-167, 171, 172, 194, 206,
209
Functional checking, 93 Multi-functional module, 92 217, 221
Instrumentor, 47
Functional cohesion, 102 Process modelling, 164, 167, 169, 172
Integration, 19, 37, 113, 115-116, 228
Functional design standard, 97 Naming convention, 102 Process modelling notation, 168-169
Integration and test plan, 37
Functional requirement, 74, 83, 87, 91, 231 Natural language, 82 Process step, 165
Integration plan, 115
Functional requirements specification standard, Non-functional requirement, 83 Product constraint, 66
Integration strategy, 19, 115, 229
84 Non-functional requirements specification Product identification, 199
Integration test, 38, 117
standard, 84 Product identification and traceability, 208
Functional specification, 33, 71, 97, 141, 178 Integration test data, 127
Functional system test, 117 Non-functional quality attribute validation Product metric, 159
Integration test procedure, 120
Functional test, 116 document, 93 Program, 11
Integration test standard, 120
Functional testing, 37, 108 Non-functional requirement, 83 Program design language, 30
Integration testing, 19, 37-38, 92, 107-109,
Non-initialized variable, 77 Programmer, 11
113-114, 120, 201, 229
Genericity, 139 Programmer's notebook, 110
Integrity, 7
Goto statement, 102 Object code, 11 Programming, 24, 30, 33, 39, 45, 99, 179, 200,
Interactive observation, 154
GQM, 161-162 Object model, 141 204, 228
Interface evaluation guidelines, 156
Granularity of integration, 37 Object-oriented technology, 141 Programming language, 21, 30, 88, 100, 133, 167,
Interface evaluation procedure, 156
Graphical notation, 68, 167 Object-oriented development, 142 169, 186
Internal milestone, 61
Guideline, 9, 21, 24-25, 50, 167, 220, 231 Object-oriented programming language, 135, 139, Programming procedure, 104
Internal standard, 60
140 Programming standard, 11, 14, 23, 101-104, 107,
International standard, 22
Handling, storage, packaging and delivery, 203, Object-oriented system, 142 142, 180
Interoperability, 7, 49, 93
210 Object-oriented technology, 141-143, 145, 167 Progress meeting, 14, 24
ISO 9000, 169
Hardware, 96 Observational evaluation, 154, 155 Progress reporting, 56
ISO 9000-3, 190
Hardware system, 3 One-to-n relation, 69 Project assumption, 44
ISO 9001, 22, 43, 55, 190, 193, 206, 222
HCI acceptance criteria, 154 One-to-one relation, 69 Project audit, 193, 227
ISO 9004-2, 191
 240 INDEX INDIIx 241

Project contract, 60 Real-time software, 16, 27 System design review checklist, 97

Security. 96
Project costing, 55, 57 Real-time system, 102, 176, 212 S- stern design review procedure, 97
Security requirement . 72
Project debriefing, 233 Real-time testing . 193 System design review standard, 97
Senior management . 13
Project debriefing report, 193, 206, 227 Recovery operation, 71 System design specification, 93, 93, 122 . 144
Servicing, 205, 210
Project deliverable, 60 Relation, 31, 33 . 69, 70 System design standard, 145
SETL, 135
Project library, 11, 15, 18, 38, 135 Relational database, 88 System designer, 12
Shutdown operation . 71
Project management, 229 Relational database management system, 33 System documentation . 28
Simulation . 20, 47, 75 . 93, 19 7,
Project manager, 8-10 . 13, 15, 17, 20 . 28 . 13 . 169 System environment specification standard, 84
Reliability, 6, 83 Sizing, study, 48
Project monitoring, 227 Reliability testing . 50 System function, 71
Smalltalk . 141
Project organization, 9, 43-44, 52, 231 Representing, 9-10 System software, 27
Software architecture. 20
Project plan, 10, 14, 22, 44 . 50, 60 . 61, 75. 83 . 81, Reprogramming, 11 System specification, 3 . 28
Software deliverable . 60
154,206-208 Requirements analysis, 19, 27-2S, 32, 34, 36, 38, Software development methodology, 21 System test, 15, 22, 23, 35-38, 51, 65, 75, 108,
Project planning, 20, 44, 4S, 50, 56, 167, 200 . 21 .1 . 51, 64, 65, 73, 75, 79, 141-112 . 167, 194, 116, 128, 134, 228
Software Engineering Institute, 171
227 203-204, 206, 221, 231 System test data . 127
Software logging, 154
Project planning standard, 51, 185 Requirements change, 10 .3 System test specification, 123, 128
Software metrication, 55
PROLOG, 35, 135 Requirements elicitation, 34, 137 System test suite . 36
Software tool, 21, 61, 87, 167 , 206
Prototype, 134, 136 Requirements error, 106 System testing, 6, 10, 15, 19, 34, 38-10 . 45, 80,
Source code, 11
Prototype evaluation, 138 Requirements review, 34, 36 . 145 . 230 83, 92, 106-109 . 116-117, 120, 134 . 142, 1-14,
Special process, 200
Prototyping, 20, 34, 35 . 47, 50, 54 . 75, 133-137, Requirements review checklist, 84 154, 176, 201 . 213. 215, 228
Spot-check, 77
1-13, 153, 165, 195, 197, 227, 2.30 Requirements review procedure . 84 System testing procedure . 120
Spreadsheet . 7, 10, 48, 49, 72, 82 . 93 . 180
Prototyping procedure, 84 Requirements review standard, 84 System tests
. 40 . 113
SSADM . 90
Prototyping standard, 84 Requirements specification, 3-7 . 12-13, 15, 18-21, Staff questionnaire, 193
Purchaser supplied product . 199 . 208 23, 27-28, 33-36, 38-40, 45 . 51 . 61, 64-67 . Staffing department . 1-1
Purchasing manager, 12 69-75 . 77, 79-80, 83-84, 86-87, 91, 94-96, Standard, 9-10 . 17-18, 2'_-22, 24-25 . 43, 50, 79, Table-driven processor, 135
Purchasing system, 231 99, 116, 126-128, 134, 138, 141 . 143-145, 94, 167 . 174-175, 1S1, 188 . 19-1-195 . 218, Task identification, 48, 58
147, 151, 155, 178, 180, 195-199, 202-203, 220 . 227, 229, 231 Task identification standard, 60
Quality assurance, 6
0 06-207,215,226-231
- Statement of requirements . 28, 35, 42, 48, 60, 65, Task specification, =18, 150-152, 155
Quality assurance department, 3, 10, 15 Requirements specification checklist, 84 Task specification procedure. 155
73-75, 79-80, 195 . 206, 213-214
Quality assurance function, 17 Requirements specification review, 79, 195 Statement of requirements traceability standard . Task specification review, 153
Quality attribute, 3 Requirements specification standard, 24, 68 . 73, Task specification review procedure . 156
84
Quality audit, 198, 204, 210 145 Task specification review standard, 155
Statement of responsibility, 220
Quality circle, 224 Requirements volatility, .15 Task specification standard, 155
Statistical technique . 205, 210
Quality control, 7-9, 15, 21-23, 25 . 49-51 . 9-1, Response time, 3, 7, 12, 28 . 87 Status accounting, 128 Taxation law, 17
106 . 109, 142, 175, 180-181, 193 . 194, Technical implementation strategy, 48
Result metric, 159 Stock control application . 13, 60
197-199, 201, 206, 208, 214 Retailing . 20 . 31 Technical review . 20, 23 . 34 . 75 . 103 . 151, 169 .
Stock control system . 12
Quality factor, 3-8, 22-23, 25, 42-43 . 48-50, 166 . Retesting, 15 Structural metric . 107 195, 197 . 227
175 . 199 Reusability, 4, 7-,8, 142 Structured software development method . Telecommunication system, 88
Quality factor checklist, 7-8 Reusable library, 8 Telecommunications software, 27
Structured walkthrough . 101
Quality improvement team, 223-224
Reuse, 170 Sub-sub-function . 67 Test case, 117, 142
Quality management system. 21 Reverse traceability, 18 Test case specification, 117
Subcontracting plan, 43
Quality manual, 2, 7-8, 21-22 . 25, -'.3 . 50. 53-54, Risk, 43 Test case specification standard, 120
Subcontractor, 43-44, 46, 54, S0 . 198
57, 79, 82, 88-90, 101, 12-1 . 164 . 166, Risk analysis, 10, -45, 48, 60, 137
. 155, 214, 215 . Subcontractor evaluation, -15 Test coverage, 110, 1 ,12
171-172, 174, 192, 196 .. 206-207, 214-215, 228 Test data, 2-3, 11 . 178, 181
Subcontractor performance, 198
M .227-228 Risk analysis procedure, 214 Subcontractor selection procedure . 5 t Test data selection . 110
Quality plan, 15, 20, 22, 25 . -12 . 49-51, s
.3, 94-95, Risk assessment . 46. 57 Test department, 61
Sub-function . 67
166, 194, 206, 227 Risk factor, 10 Subroutine, 11 . 86, 87 Test design, 82
Quality planning, 227 .catiou, 117
Risk minimization guideline, 47 Subsidiary software. 60 Test design specil
Quality policy, 192, 201 Risk monitoring, 48 Test design specification standard . 120
System architecture, 29, 97
Quality record . 193-194. 203-20-1 . 206 Risk questionnaire . 46, 48, 138 system design, 9 . 19, 28, 30, 36-39 . .1 56 . 50-31 . Test failure, 233
Quality system, 2-4, 7, 9-12, 14-15 . 17-21, 25,
65-66 . 71 . 75, 81 . 53 . 86-87. 91, 93, 96-97 . Test file, 15, 18
42. 47, 54, 57, 70, 92, 226-228, 233 Safety-critical application, 4, 199 . 201 99, 101 . 107, 126-127 . 134, 1 .13, 152 . 157-15s . Test harness, 178
Questionnaire. 180 Safety-critical system, 6, 8, 80 . 102 . 154 176, 178, 180-181, 197, 200, 203 . 231 Test log, 117 . 119
Scaffold software, 115 System design metric, 160-161 Test leg standard . 120
R & D department, 223-224 Scenario analysis, 79 System design notation, 87 Test oracle, 134
Readability index, 23
Screen and window design guidelines, 156 System design procedure, 97 Test outcome, 2 . 11, 15, 178 . 181
Real-time response, 46 Screen and window specification standard, 156 System design review, 145 Test problem report form, 11 7 . 119
242 INDEX Further titles in this Series

Practical Formal Methods with VDM Andrews and Ince

Test problem report standard . 120 User characteristics, 230 Portable Modula-2 Programming Woodman, Griffiths . Souter
Test procedure, 38, 81, 117, 119, 168, 199 User documentation, 20 and Davies
Test record, 23 User manual, 3, 28, 65
Software Engineering : Analysis and Design Easteal and Davies
Test report, 194 User requirement, 35
User/environment questionnaire, 155 Systems Construction and Analysis : A Mathematical and
Test status, 209
Test summary, 117 Logical Framework Fenton and Hill
Testability, 6 Validation, 17, 19, 30, 34-35, 38, 50 . 75-76 . 87, SSADM Version 4 Project Manager's Handbook Hammer
Testing, 43, 52, 167, 176, 199, 201, 228-231 92, 104, 124, 130, 142-143, 153, 207 Introduction to Compiling Techniques :
Testing tool, 10, 36, 79, 228 Variant, 128 A First Course Using ANSI C, LEX and YACC Bennett
Third-generation language, 31, 47, 86-88, 107 Variant number, 128
Third-generation system, 194
An Introduction to Program Design Sargent
Verbal protocol, 154
Throw-away prototyping, 136, 137 Verification, 30, 34, 38, 50, 76, 207
Expert Database Systems : A Gentle Introduction Beynon-Davies
Top-down integration, 37, 113 Verification matrix, 91-94, 96 A Structured Approach to Systems Development Heap, Stanway and Windsor
Traceability, 18, 56, 92-93, 97, 172, 199, 206-208, Verification requirement, 51, 79-82, 116, 228 Rapid Information Systems Development Bell and Wood-Harper
226, 231 Verification requirements procedure, 84 Software Engineering Environments :
Training, 60, 137, 204, 210, 229 Verification requirements standard, 84
Automated Support for Software Engineering Brown, Earl and McDermid
Training plan, 210 Version data, 207
Training record, 204 Version number, 128-129
Knowledge Engineering for Information Systems Beynon-Davies
Transaction-based system, 8 Video recording, 154
Virus attack, 203
Unit folder, 110, 181
Unit folder standard, 120
Walkthrough, 104, 195
Unit testing, 37, 103, 107, 109, 112, 120, 201, 209,
White box testing, 106
217
WIMP interface, 4
Unit testing procedure, 120
Windows interface, 68, 80
UNIX, 35, 135
Word processor, 49, 181
Unsafe programming construct, 77, 102
Work breakdown structure, 55-60
Usability, 4, 6, 8
Usability review, 50
Usability study, S YACC, 135
Usability testing, 8 Yourdon Structured Development, 90
Related titles are available in McGraw-Hill's International Sofnvare Quality Assurance Series