Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

Investigation of the Metrology Concepts in ISO 9126 on

Software Product Quality Evaluation
ALAIN ABRAN*, RAFA E. AL-QUTAISH*, JUAN J. CUADRADO-GALLEGO**
*
École de Technologie Supérieure
University of Québec
1100 Notre-Dame Street West
Montréal, Québec H3C 1K3,
CANADA
**
Department of Computer Science
University of Alcalá
28805 Edificio Politécnico
Alcalá de Henares,
SPAIN

Abstract: - The ISO International Vocabulary of Basic and General Terms in Metrology (VIM) represents the
international consensus on a common and general terminology of metrology concepts. However, until recently,
it was not usual practice in software engineering measurement to take into account metrology concepts and
criteria in the design of software measures. Using the ISO 9126-4 Technical Report on the measurement of
software quality in use as a case study, this paper reports on the extent to which this ISO series addresses the
metrology criteria typical of classic measurement. Areas for improvement in the design and documentation of
measures proposed in ISO 9126 are identified.

Key-Words: - Metrology, VIM, ISO 9126, Quality in Use metrics, Software Measurement, Software Quality.

1 Introduction proposals [6-8]. The inventory of software metrics is

In the field of software engineering, the term at the present time so diversified and includes so
“metrics” is used in reference to multiple concepts, many individual proposals [9, 10] that it is not seen
whether in terms of the quantity to be measured to be economically feasible for either industry or the
(measurand1), measurement procedures, research community to investigate each of the
measurement results or models of relationships hundreds of alternatives proposed to date.
across multiple measures, or of the objects While metrology has a long tradition of use in
themselves. In the software engineering literature, physics and chemistry, it is rarely referred to in the
the term is applied, for instance, to a measure of a software engineering literature. Carnahan et al. [11]
concept (e.g. McCabe cyclomatic complexity [1]), are among the first authors to identify this gap in
to quality models (ISO 9126 – software product what they referred to as “IT metrology”; they
quality [2]) and to estimation models (e.g. highlight the challenges and opportunities arising
Halstead’s equations [3], COCOMO I and II from the application of the metrology concepts to
estimation models [4, 5]). This has led to many information technology. In addition, they have
curious problems, among them a proliferation of proposed logical relationships between metrology
numerous publications on metrics for concepts of concepts, consisting of four steps to follow to obtain
interest, but with a very low rate of acceptance and measured values: defining quantity/attribute,
use by either researchers or practitioners, as well as identifying units and scales, determining the
a lack of consensus on how to validate so many primary references and settling the secondary
references. Moreover, Gray [12] discusses the
applicability of metrology to information
1 technology from the software measurement point of
A measurand is defined as a particular quantity subject
view.
to measurement; the specification of a measurand may
require statements about quantities such as time, Abran [13] has highlighted some high-level
temperature and pressure [19]. ambiguities in the domain of software measurement,
Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

and proposed substituting the appropriate metrology Terms in Metrology as the field of knowledge
terms for the current ambiguous and peculiar dealing with measurement [19]. More specifically, it
software metrics terminology unique to the domain has been defined in [20] as: “that portion of
of software engineering. In metrology, the term measurement science used to provide, maintain, and
“metrics” is never used. In addition, the availability disseminate a consistent set of units; to provide
of the metrology concepts in software engineering support for the enforcement of equity in trade by
has been investigated in [7, 14, 15]. Abran and weights and measurement laws; or to provide data
Sellami [8] have documented the metrology for quality control in manufacturing”. Following the
concepts addressed in ISO 19761 (COSMIC-FFP), above definitions, metrology forms the basis of all
both in the design of this measurement method and measurement-related concepts in the natural
in some of its practical uses. Moreover, Sellami and sciences and engineering, and to each of the
Abran [16] have investigated the contribution of different interpretations of a software metrics is
metrology concepts to understanding and clarifying associated a related distinct metrology term with
the framework for software measurement validation related metrology criteria and relationships with
proposed by Kitchenham et al. in [17]. other measurement concepts. In 1984, the ISO, with
The ISO 9126 series of documents on software other participating organizations (BIPM, IEC and
product quality evaluation proposes a set of 120 OIML), published their first edition of the
metrics2 for measuring the various characteristics international consensus on the basic and general
and subcharacteristics of software quality. However, terms in metrology (VIM) [21]. Later, in 1993, this
as is typical in the software engineering literature, publication was reviewed, and then the ISO, in
their set of so-called metrics in ISO 9126 refers to collaboration with six participating organizations
multiple distinct concepts which, in metrology, (BIPM, IEC, OIML, IUPAC, IUPAP and IFCC),
would have distinct labels (or naming conventions, published the second edition of this document [19].
e.g. terms) to avoid ambiguities. The ISO is now working on its third edition of this
To help in understanding and clarifying the document to integrate, in particular, concepts related
nature of the metrics proposed in ISO TR 9126-4 to measurement uncertainty and measurement
[18], each is analyzed in this paper from a traceability.
metrology perspective and mapped to the relevant The second VIM edition on metrology presents
metrology concepts. Such a mapping will also 120 terms in six categories and in increasing order
contribute to identifying the measurement concepts of complexity, and describes each term individually
that have not yet been tackled in the ISO 9126 series in textual format (in parentheses, the number of
of documents. Each of these gaps represents an terms in each category): Quantities and Units (22
opportunity for improvement in the design and terms), Measurements (9 terms), Measurement
documentation of the measures proposed in ISO Results (16 terms), Measuring Instruments (31
9126. terms), Characteristics of the Measuring Instruments
This paper presents an overview of the relevant (28 terms) and Measurement Standards – Etalon (14
metrology concepts in section 2, and an overview of terms).
the ISO 9126 series and the quality in use metrics in To facilitate an understanding of these more than
Section 3. Section 4, 5, 6, and 7 present the analysis one hundred related terms, Abran and Sellami [22]
of the “effectiveness”, “productivity”, “safety”, and proposed a modeling of all the sets of measurement
“satisfaction” metrics, respectively. A discussion in concepts documented in this ISO document.
section 8 concludes the paper. Two of the categories of terms in the VIM deal
with some aspects of the design of measurement
methods, that is, category 1: “quantities and units”,
2 Metrology and category 2: “measurement standards – etalon”.
The term “metrology” is defined in the ISO The other four categories are related to the
International Vocabulary of Basic and General application of a measurement design with a
measuring instrument, and to the quality
characteristics of the measurement results provided
by this measuring instrument [22]. More
2
While the term “metrics” is used in ISO 9126, the use of specifically, we use the first category, which deals
this term will be abandoned and replaced by with the design of the measurement methods, that is,
“measures” in the next ISO version currently in quantities and units and, in particular, the system of
preparation as an initial step towards harmonizing the
software engineering measurement terminology with
the metrology terminology.
Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

quantities that consists of two types of quantities; them and examples of how to use them in the
that is, base and derived.3 software product life cycle [18].
In ISO 9126-4 [18], fifteen metrics have been
proposed for the quality in use metrics. They have
3 ISO 9126 & Quality in Use Metrics been classified into four collections of metrics based
In 1991, the ISO published its first international on the characteristics presented in ISO 9126-1:
consensus on the terminology for the quality 1. Effectiveness: task effectiveness, task completion
characteristics for software product evaluation: ISO and error frequency
9126 – Software Product Evaluation – Quality 2. Satisfaction: task time, task efficiency, economic
Characteristics and Guidelines for their Use [23]. productivity, productive proportion and relative
From 2001 to 2004, the ISO published an expanded user efficiency
four-part version, containing both the ISO quality 3. Safety: user health and safety, safety of people
models and inventories of proposed measures for affected by use of the system, economic damage
these models. The current version of the ISO 9126 and software damage
series now consists of one International Standard 4. Productivity: satisfaction scale, satisfaction
(IS) [2] and three Technical Reports (TR) [18, 24, questionnaire and discretionary usage
25]. These fifteen metrics are analyzed using a
The first document of the ISO 9126 series – metrology concept structure from the VIM category,
Software Product Quality Model – contains two Quantities and Units [19], based on four
structures of quality models for software product characteristics, that is: system of quantities,
quality [2]: a structure for both the internal and dimension of a quantity, unit of measurement and
external quality models, and a structure for the value of a quantity.
quality in use model. The first structure of the ISO
9126-1 Quality Model includes six characteristics,
subdivided into twenty-seven subcharacteristics for 4 Effectiveness Metrics
internal and external quality [2]. These In ISO 9126-4, the claim is that the three
subcharacteristics are related to internal software Effectiveness Metrics assess whether or not the task
attributes, and are noticeable externally when the carried out by users achieved the specific goals with
software is used as part of a computer system. The accuracy and completeness in a specific context of
second structure of the ISO 9126-1 model includes use [18]. This section presents the outcomes of the
four quality in use characteristics [2]: effectiveness, mapping of the set of Quantities and Units
productivity, safety and satisfaction. These metrology concepts to the 2004 description of
characteristics and subcharacteristics are defined in Effectiveness Metrics in ISO 9126-4. A summary of
the ISO 9126-1 international standard document. this mapping is presented in the Appendix.
The second document of the ISO 9126 series –
Software Product External Quality Metrics –
contains a set of metrics for each external quality 4.1 System of quantities for Effectiveness
subcharacteristic, explanations of how to apply and 4.1.1 Base quantities
use them, and examples of how to apply them First, it can be observed that these three
during the software product life cycle [24]. Effectiveness Metrics are not collected directly
The third document of the ISO 9126 series – by a measurement system, but are derived from
Software Product Internal Quality Metrics – a computation of four base quantities that are
contains an inventory of metrics for each internal
themselves collected directly, that is: task time,
quality subcharacteristic, explanations of the
application of these metrics, and examples of how to number of tasks, number of errors made by the
use them in the software product life cycle [25]. user and proportional value of each missing or
Finally, the fourth document of the ISO 9126 incorrect component.
series – Software Product “Quality in Use” Metrics The first three of these base measures in the
– contains a basic set of metrics for each quality in Appendix refer to terms in common use, but this
use characteristic, explanations of how to apply leaves much to interpretation on what constitutes,
for example, a task: it does not ensure that the
measurement results are repeatable and reproducible
3 across measurers, across groups measuring the same
In ISO 15939, the terms “base quantities” and “derived
quantities” were replaced by equivalent terms: “base software and, as well, across organizations where a
measures” and “derived measures”. task might be interpreted differently and with
Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

different levels of granularity. This leeway in their 4.3 Units of measurement for Effectiveness
interpretation makes a rather weak basis for either The metrology concepts related to units of
internal or external benchmarking. measurement are:
The third base quantity, number of errors made ƒ Symbols of the units
by the user, is defined in Appendix F of ISO TR ƒ Systems of units
9126-4 as an “instance where test participants did ƒ Coherent (derived) units
not complete the task successfully, or had to attempt ƒ Coherent system of units
portions of the task more than once” [18]. This ƒ International system of units
definition diverges significantly from the one in the ƒ Base units
IEEE Standard Glossary of Software Engineering ƒ Derived units
Terminology [26] where the term “error” has been ƒ Off-system units
defined as “the difference between a computed, ƒ Multiples of a unit
observed, or measured value or condition and the ƒ Submultiples of a unit
true, specified, or theoretically correct value or The mappings of these metrology concepts for
condition; for example, a difference of 30 meters Effectiveness Metrics are presented in the Appendix.
between a computed result and the correct result.” Two metrology concepts must be analyzed in more
The fourth base quantity, referred to as the detail, base units and derived units.
“Proportional value of each missing or incorrect
component” in the task output is based, in turn, on
another definition, whereas each “potential missing 4.3.1 Base units
or incorrect component” is given a weighted value Of the four base quantities, a single one, i.e. task
Ai based on the extent to which it detracts from the time, has an internationally recognized standard
value of the output to the business or user [18]. base unit, i.e. the second, or a multiple of this unit. It
These expansive, embedded definitions contain a also has a universally recognized corresponding
number of subjective assessments for which no symbol (‘s’). The next two base units (tasks and
repeatable procedure is provided: the value of the errors) do not refer to any international standard of
output to the business or user, the extent to which it measurement, and must be locally defined (which
detracts, the components of a task and potential means that they fit poorly, for comparison purposes,
missing or incorrect components. when measured by different people, unless local
measurement protocols have been clearly
documented, and they are implemented rigorously
4.1.1 Derived quantities in a specific organization). The fourth base quantity,
The proposed three Effectiveness Metrics, which are proportional value of each missing or incorrect
defined as a prescribed combination of these base component, is puzzling because it is based on a
quantities, are therefore derived quantities. The given weighted value (number) and has no
ranges of the results obtained from implementing measurement unit.
the corresponding measurement function are
introduced in the upper part of the Appendix for
each of these derived quantities. These quantities 4.3.2 Derived units
inherit the weaknesses of the base quantities of The derived quantity, task effectiveness, leads to a
which they are composed. derived unit that depends on a given weight (i.e. (1 –
a given weight)). Therefore, like the base unit, its
derived unit of measurement is unclear.
4.2 Dimension of a quantity for Effectiveness The derived quantity, task completion, is
Emerson [27] states that the concept of dimension is computed by dividing two base quantities (task/task)
particularly applicable to the derived quantities: two with the same unit of measurement.
of them, i.e. task effectiveness and task completion, The definition of the computation of the derived
can have values between 0 and 1, and would be quantity, error frequency, provides two distinct
considered as dimensionless quantities, since a ratio alternatives for the elements of this computation.
of quantities with the same dimensions is itself This can lead to two distinct interpretations, i.e.
dimensionless [27]. errors/task or errors/second. Of course, this in turn
leads to two distinct derived quantities as a result of
implementing two different measurement functions
(formulas) for this derived quantity. Of course, this
leaves open the possibility of misinterpretation and
Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

misuse of measurement results when combined with 5.1 System of quantities for Productivity
other units: for example, measures in centimeters One of the five proposed productivity metrics in
and measures in inches cannot be added or ISO 9126-4 is a base quantity (task time) while the
multiplied. other four are derived quantities (task efficiency,
This lack of clarity, as well as the lack of economic productivity, productive portion and
references to international units of measurement, relative user efficiency).
could explain why there has been no attempt to In addition, task efficiency refers explicitly to
integrate the proposed base and derived quantities another derived quantity, task effectiveness, which
into a system of units, including references to was analyzed in the previous section.
coherent units and a coherent system of units. It is to be noted that these derived quantities are
themselves based on five base quantities: task time,
cost of the task, help time, error time and search
4.4 Value of a quantity for Effectiveness time.
The four types of metrology values of a quantity
are: true value, conventional true value,
numerical value and conventional reference 5.2 Dimension of a quantity for Productivity
scale. All the productivity metrics, except task time, are
Numerical values are indeed obtained for each dimensionless quantities.
base quantity based on the defined data collection
procedure; for each derived quantity, the numerical
values are obtained by applying their respective 5.3 Units of measurement for Productivity
measurement function. For instance, the derived In the lower-middle part of the standard, for the base
quantities, task effectiveness and task completion, and derived quantities, there are five base units and
are both percentages, and are interpreted as the no explicit derived units. However, it can be
effectiveness and completion of a specific task observed that the measurement unit for task
respectively. effectiveness is not completely clear, since it
For task effectiveness in particular, anyone depends on an ill-defined “given weight”:
would be hard pressed to figure out both a true value ' task effectiveness' unit
' task efficiency' unit = .
and a conventional true value; for task completion sec ond
and error frequency, the true values would depend 1 - ' a given weight' unit ?
= = . (1)
on locally defined and rigorously applied second second
measurement procedures, but without reference to Similarly, the measurement unit of economic
universally recognized conventional true values (as productivity depends on the measurement unit of
they are locally defined). task effectiveness, a derived quantity which is
Finally, in terms of the metrological values of a unknown:
quantity, only task time refers to a conventional ' task effectiveness' unit
' economic productivity' unit =
reference scale, that is, the international standard- currency unit
etalon for time, from which the second is derived. =
1 - ' a given weight' unit
None of the other base quantities in these currency unit
effectiveness metrics refers to a conventional ?
reference scale, or to a locally defined one.
= . (2)
currency unit
Since there is no measurement unit for the
productive proportion (it has the same measurement
5 Productivity Metrics unit in both the numerator and the denominator), the
In ISO 9126-4, the claim is made that the five result is a percentage:
productivity metrics assess the resources that users second
' productive proportion' unit = . (3)
consume in relation to the effectiveness achieved in second
a specific context of use. In this standard, the time Finally, for relative user efficiency, there is no
required to complete a task is considered to be the measurement unit either, since the measurement
main resource to take into account [18]. This section units in both the numerator and the denominator are
presents the outcome of the mapping of the set of the same here as well (the task efficiency
Quantities and Units metrology concepts to the measurement unit), and therefore the result of this
2001 description of Productivity Metrics in ISO derived quantity is also a percentage. This point can
9126-4. be clarified as follows:
Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

' task efficiency' unit themselves depend on four base quantities (i.e.
' relative user efficiency' unit =
' task efficiency' unit population average, number of responses, number of
' task effectiveness' unit times that specific software function / application /
second systems are used and number of times that specific
=
' task effectiveness' unit software function/application/systems are intended
second to be used). Two of the proposed satisfaction
1 - ' a given weight' unit metrics are dimensionless quantities, i.e. satisfaction
= second questionnaire and discretionary usage.
1 - ' a given weight' unit Regarding the measurement units, there are four
second base units and no derived units; however, the
?
measurement unit, satisfaction scale, is not clear,
= second . (4) since it depends on a “questionnaire producing
?
second
psychometric scales”. The clarification of this point
is as follows:
psychometric scale unit
' satisfaction scale' unit = . (5)
people
6 Safety Metrics
In ISO 9126-4, the safety metrics claim to assess the
level of risk of harm to people, businesses, software,
property or the environment in a specific context of
8 Conclusions
The ISO International Vocabulary of Basic and
use; their scope includes the health and safety of
General Terms in Metrology (VIM) represents the
both the users and those who affected by use, as
international consensus on a common and general
well as unintended physical or economic
terminology of metrology concepts. However, until
consequences [18].
recently, it was not usual practice in software
To evaluate the safety characteristics of a
engineering measurement to take into account
software product, four derived quantities must be
metrology concepts and criteria, either in the design
quantified (i.e. user health and safety, software
of software measures or in their use and in the
damage, economic damage and the safety of people
interpretation of measurement results.
affected by use of the system). Each of these derived
This paper has presented an analysis of the ISO
quantities is the result of a computational formula
9126-4 Technical Report on quality in use metrics,
(function), which consists of a combination of pre-
and has investigated the extent to which it addresses
collected base quantities (i.e. number of usage
situations, number of people, number of occurrences the metrology criteria found in classic measurement.
Based on the analysis in sections 4 to 7, the
of software corruption, number of occurrences of
following comments and suggestions can be made:
economic corruption and number of users). It can be
- Identifying and classifying the quality in use
observed that the resulting values of all the derived
metrics into base and derived quantities makes it
quantities should be between 0 and 1.
easy to determine which should be collected (base
All the safety metrics are dimensionless
quantities) to be used subsequently in computing
quantities; there are five base units and two derived
the other (derived) quantities.
units for these quantities. In addition, two of the
- Based on equations (1) and (3 to 5), some of the
derived quantities have no measurement units, since
derived units are ambiguous, since they depend on
the measurement unit is the same in both the
other quantities with unknown units.
numerator and the denominator, i.e. user health and
- None of the quality in use metrics refers to any
safety and safety of people affected by use of the
system of units, coherent (derived) unit, coherent
system, whereas none of the measurement units has
system of units, international system of units (SI),
a symbol.
off-system units, multiple of a unit, submultiple of
a unit, true values, conventional true values or
numerical values.
7 Satisfaction Metrics - None of the base and derived quantities, except for
The satisfaction metrics in ISO 9126-4 claim to task time, has symbols for their measurement
assess the user’s attitudes towards the use of the units.
product in a specific context of use [18]. It is to be noted that the ranges of the results of
All three proposed satisfaction metrics are many of the derived metrics in ISO 9126-4 are
derived quantities (i.e. satisfaction scale, satisfaction between 1 and 0. Therefore, it is easy to convert
questionnaire and discretionary usage), which them to percentage values. However, from our point
Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

of view, these results will be easier to understand if Conference on Enterprise Information Systems
they are ranked in terms of qualitative values; for ICEIS 2004, Porto, Portugal, 2004.
example, for task completion, if the percentage [9] Kaner, C. and Bond, W. P., "Software
result is 100%, then the completion of the task is Engineering Metrics: What Do They Measure
labeled “excellent”; if the result is 80%, then the and How Do We know?" in Proceedings of the
completion of the task is labeled “very good”; and 10th International Software Metrics
so on. Symposium (METRICS 2004), Chicago,
Using the ISO 9126-4 Technical Report on the Illinois, USA, 2004.
measurement of software quality in use as a case [10] Zuse, H., A Framework of Software
study, this paper has investigated and reported on Measurement, Walter de Gruyter, Berlin,
the extent to which this ISO series addresses the Germany, 1998.
metrology criteria typical of classic measurement. [11] Carnahan, L., Carver, G., Gray, M., Hogan, M.,
Areas for improvement in the design and Hopp, T., Horlick, J., Lyon, G., and Messina,
documentation of measures proposed in ISO 9126 E., "Metrology for Information Technology,"
have been identified. The analysis methodology StandardView, Vol. 5, No. 3, 1997, pp. 103-
developed to investigate ISO TR9126-4 could also 109.
be of use to analyze the metrological strengths and [12] Gray, M. M., "Applicability of Metrology to
weaknesses of close to 120 metrics proposed by the Information Technology," Journal of Research
ISO in TRs 9126-2 and -3. of the National Institute of Standards and
Technology, Vol. 104, No. 6, 1999, pp. 567-
578.
References: [13] Abran, A., "Software Metrics Need to Mature
[1] McCabe, T. J., "A Complexity Measure," IEEE into Software Metrology (Recommendations),"
Transaction on Software Engineering, Vol. 2, presented in the NIST Workshop on Advancing
No. 4, 1976, pp. 308-320. Measurements and Testing for Information
[2] ISO/IEC, ISO/IEC 9126-1: Software Technology (IT), Maryland, USA, 1998.
Engineering - Product Quality - Part 1: [14] Abran, A., Sellami, A., and Suryn, W.,
Quality Model, International Organization for "Metrology, Measurement and Metrics in
Standardization, Geneva, Switzerland, 2001. Software Engineering," in Proceedings of the
[3] Halstead, M. H., Elements of Software Science, 9th International Software Metrics Symposium,
Elsevier North-Holland, New York, 1977. Sydney, Australia, 2003.
[4] Boehm, B., Software Engineering Economics, [15] Bourque, P., Wolff, S., Dupuis, R., Sellami, A.,
Prentice-Hall, Upper Saddle River, NJ, USA, and Abran, A., "Lack of Consensus on
1981. Measurement in Software Engineering:
[5] Boehm, B., Clark, B., Horowitz, E., Westland, Investigation of Related Issues," in
C., Madachy, R., and Selby, R., "Cost models Proceedings of the 14th International Workshop
for future software life cycle processes: on Software Measurement (IWSM),
COCOMO 2.0," Annals of Software Magdeburg, Germany, 2004, pp. 321-333.
Engineering, Vol. 1, No. 1, 1995, pp. 57-94. [16] Sellami, A. and Abran, A., "The contribution
[6] Jacquet, J. P. and Abran, A., "Metrics of metrology concepts to understanding and
Validation Proposals: A structured Analysis," clarifying a proposed framework for software
in Proceedings of the 8th International measurement validation," in Proceedings of the
Workshop on Software Measurement, 13th International Workshop on Software
Magdeburg, Germany, 1998. Measurement (IWSM), Montreal, Canada,
[7] Abran, A. and Sellami, A., "Measurement and 2003, pp. 18-40.
Metrology Requirements for Empirical Studies [17] Kitchenham, B., Pfleeger, S. L., and Fenton,
in Software Engineering," in Proceedings of N., "Towards a Framework for Software
the 10th International Workshop on Software Measurement Validation," IEEE Transaction
Technology and Engineering Practice (STEP), on Software Engineering, Vol. 21, No. 12,
Montreal, Canada, 2002. 1995, pp. 929-944.
[8] Abran, A. and Sellami, A., "Analysis of [18] ISO/IEC, ISO/IEC TR 9126-4: Software
Software Measures Using Metrology Concepts Engineering - Product Quality - Part 4:
- ISO 19761 Case Study," presented in the Quality in Use Metrics, International
International Workshop on Software Audits Organization for Standardization, Geneva,
and Metrics SAM’2004, 6th International Switzerland, 2004.
Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

[19] ISO/IEC, International Vocabulary of Basic Guidelines for Their Use, International
and General Terms in Metrology (VIM), Organization for Standardization, Geneva,
International Organization for Standardization, Switzerland, 1991.
Geneva, Switzerland, 1993. [24] ISO/IEC, ISO/IEC TR 9126-2: Software
[20] Simpson, J. A., "Foundations of Metrology," Engineering - Product Quality - Part 2:
Journal of Research of the National Bureau of External Metrics, International Organization
Standards, Vol. 86, No. 3, 1981, pp. 36-42. for Standardization, Geneva, Switzerland,
[21] ISO/IEC, International Vocabulary of Basic 2003.
and General Terms in Metrology (VIM), [25] ISO/IEC, ISO/IEC TR 9126-3: Software
International Organization for Standardization, Engineering - Product Quality - Part 3:
Geneva, Switzerland, 1984. Internal Metrics, International Organization for
[22] Abran, A. and Sellami, A., "Initial Modeling of Standardization, Geneva, Switzerland, 2003.
the Measurement Concepts in the ISO [26] IEEE, Std. 610.12-1990: Standard Glossary of
Vocabulary of Terms in Metrology," in Software Engineering Terminology, the
Proceedings of the 12th International Workshop Institute of Electrical and Electronics
on Software Measurement (IWSM), Engineers, New York, USA, 1990.
Magdeburg, Germany, 2002. [27] Emerson, W. H., "Short Communication on the
[23] ISO/IEC, ISO/IEC IS 9126: Software Product Concept of Dimension," Metrologia, Vol. 42,
Evaluation - Quality Characteristics and No. 2, 2005, pp. 21–22.
Proceedings of the 10th WSEAS International Conference on COMPUTERS, Vouliagmeni, Athens, Greece, July 13-15, 2006 (pp864-872)

Appendix: “Quantities and Units” Metrology Concepts in

ISO 9126-4 – Effectiveness Metrics

Metrology Concepts ISO 9126-4 (Effectiveness Metrics*)

5.1 System of Quantities:
- Base Quantities: 1. Task Time.
2. Number of Tasks.
3. Number of Errors Made by the User.
4. Proportional value of each missing or incorrect
- Derived Quantities: 5. Task Effectiveness*. 0 ≤ Task Effectiveness ≤ 1
6. Task Completion*. 0 ≤ Task Completion ≤ 1
7. Error Frequency*. Error Frequency ≥ 0
5.2 Dimension of a Quantity:
- Quantities of Dimension 5. Task Effectiveness.
One (Dimensionless 6. Task Completion.
Quantities):
5.3 Units of Measurement:
- Symbols of the Units: - s (Second)
- Systems of Units: - None.
- Coherent (Derived) - None.
- Coherent System of - None.
- International System of - None.
- Base Units: 1. Second. 2. Task. 3. Error. 4. Non (ill-defined)
- Derived Units: 5. (1- a given weight). 6. Task/Task = %
7. Error/Task or Error/Second.
- Off-System Units: - None.
- Multiple of a Unit: - None.
- Submultiple of a Unit: - None.
5.4 Value of a Quantity:
- True Values: - None.
- Conventional True - None.
Values:
- Numerical Values: - Results of applying the measurement functions of the
above base and derived quantities.
- Conventional Reference 1- Task Time.
Scales (Reference-
Value Scales):