A Goal-Based Software

Measurement Framework
A Goal-Based Software Measurement
Framework
1. Classifying software measures
2. Determining what to measure

2
 Classifying Software Measures
In software there are three such classes
 Processes: Collection of software-related activities.
 Products: Artifacts, deliverables or documents that
result from a process activity.
 Resources: Entities required by a process activity.
 Internal and External Attributes
• An internal attribute can be measured by examining
the product, process, or resource on its own,
separate from its behavior. (Program size,
complexity, dependencies).

• External attributes are those that can be measured

only with respect to how the product, process or
resource relates to the environment.
(Ease of Navigation, Number of failures)
 Internal and External Attributes
Internal External
 Size, Effort, Cost  Usability
 Code Complexity  Integrity
 Functionality  Efficiency
 Modularity  Testability
 Redundancy  Reusability
 Syntactic Correctness  Portability
 Reuse  Interoperability
 Importance Of Internal Attributes
• Many software engineering methods proposed and
developed in the last 25 years provide rules, tools,
and any heuristics for providing software products. It
is claimed that this structure makes them easier to
understand, and tests.
• It is assumed that good internal structure leads to a
good external quality. This connection has rarely
been established.
 Processes
• We often have questions about our software-
development activities and processes that
measurement can help us to answer.
• We want to know how long it takes for a
process to complete, how much it will cost,
whether it is effective or efficient, and how it
compares with other processes that we
could’ve chosen.
 Products
• Products are not restricted to the items that management is committed to
deliver to the customer. Any artifact or document produced during the
software life cycle can be measured and assessed. For example developers
often build prototypes for examination only, so that they can understand
requirements or evaluate possible designs; these prototypes may be
measured in some way.
• External product attributes depend on both product behavior and
environment, each attribute measure should take these characteristics
into account.
• Internal product attributes are sometimes easy to measure. We can
determine the size of a product by measuring the number of pages it fills
or the number of words it contains. Other internal product attributes are
more difficult to measure, because opinions differ as to what they mean
and how to measured them. For example the complexity of codes.
 Resources
• The resources that we are likely to measure
include any input for software production.
• Resources include personnel, materials, tools and
methods. Resources are measured to determine
their magnitude, cost and quality.
• Cost is often measured across all types of
resources, so that managers can see how the cost
of inputs affects the cost of the outputs.
• Resource measure combines a process measure
(input) with a product measure (output).
A Goal-Based Software Measurement Framework

Determining what to measure

– Measurement is useful only if it helps understand the
underlying process or one of its resultant products
– Goal-Question-Metric (GQM) has been proven to be
effective in selecting and implementing metrics
• List the major goals of the development project
• Derive from each goal the questions that must be answered to
determine if goals are being met
• Decide what must be measured in order to be able to answer
the questions adequately

10
 GQM
The Goal/Question/Metric (GQM) framework for
software measurement is goal-oriented.
Firstly, the goals need to be described clearly so that it
can be measured during the software development. It
consists of the following 3 basis elements:
• Goal
• Question
• Metric
 GQM
• hierarchical model that follows a top-down
approach where first the goals are specified, then
questions are written and collected, and finally,
metrics are associated with each question
 GQM
• Goal – improve Products, Processes or
Resources (Conceptual level)
• Question – should be specific to the goal
(Operational level)
• Metric – quantitative way in which data can
be collected to address the question you are
asking (Quantitative level)
• What is the purpose of GQM?
To ensure that you use metrics effectively
Example of GQM Approach :
A goal should specify the following things in it :
Its purpose
A process (or object)
A viewpoint
A quality issue
• Objective metrics
Based on data that does not rely on anyone’s
individual viewpoint or interpretation
Examples
Number of product downloads, processing time,
LOC, size of module, size of program
• Subjective metrics
Are those that could depend on interpretation
Examples
Product ratings by users, code quality
 Example for GQM
Say you are developing a software product and you want
to know your product is “Done Right”
• Goal (G):Customer Satisfaction
Few question to address the goal
• Questions(Q):
How much of my product’s code is covered by unit tests?
How much defects are in my product?
• Metric(M):
--percentage of rejections to acceptances
--number of bugs that are placed in your product backlog
at any given time
Example 2
A Goal-Based Software Measurement Framework

GQM example – goal is to evaluate

effectiveness of coding standard
Goal Goal

Questions Who is using standard? What is coder productivity? What is code quality?

Proportion of coders Experience of coders Code size Effort Errors

Metrics • Using standard • With standard (lines of code,
• Using language • With language function
• With environment, etc. points, etc

18
A Goal-Based Software Measurement Framework
• GQM example 2 – AT&T goals, questions, metrics
Goal Questions Metrics
Plan How much does the inspection process Average effort per KLOC
cost? Percentage of reinspections
Average effort per KLOC
How much calendar time does the Total KLOC inspected
inspection process take?
Monitor and What is the quality of the inspected Average faults detected per KLOC
control software? Average inspection rate
Average preparation rate
To what degree did the staff conform to Average inspection rate
the procedures? Average preparation rate
Average lines of code inspected
What is the status of the inspection Total KLOC inspected
process?
Improve How effective is the inspection process? Defect removal efficiency
Average number of faults detected
per KLOC
Average inspection rate
Average preparation rate
Average lines of code inspected
What is the productivity of the inspection Average effort per fault detected
process? Average inspection rate
Average preparation rate
Average lines of code inspected 19
A Goal-Based Software Measurement Framework
• Templates for goal definition
– Purpose – to (characterize, evaluate, predict, motivate, etc.) the (process,
product, model, metric, etc.) in order to (understand, assess, manage,
engineer, learn, improve, etc.) it.
• Example – To evaluate the maintenance process in
order to improve it.
– Perspective – Examine the (cost, effectiveness, correctness, defects,
changes, product measures, etc.) from the viewpoint of the (developer,
manager, customer, user, etc.)
• Example – Examine the cost from the viewpoint of
the manager
– Environment – The environment consists mainly of the following: process
factors, people factors, problem factors, methods, tools, constraints, etc.
• Example – the maintenance staff are poorly
motivated programmers who have limited access to
tools.

20
 Sample Goal Trees (1/2)
Project planning Improve software Project tracking goal area:
goal area: Predict Evaluate the project in
the process in order process predictability Execute order to manage it
to manage it
successful project

Improve effort Improve defect

predictability predictability
Achieve business
Improve schedule value
predictability
Deliver a quality product
within cost and schedule
constraints
21
 Sample Goal Trees (2/2)
Process change Improve the software process
management goal area:
Evaluate the process in
order to improve it

Improve quality
Improve productivity Minimize schedule
Maximize reuse
Defect prevention goal area:
Characterize defects in order to
prevent them
Characterize the process to
understand it Prevent defects
(minimize defect fixing costs,
improve product quality)
22
 Sample GQM Map for Inspections
Organizational Process Improvement Inspection Process
Goal Increase quality Increase productivity Efficiently use
inspections to
improve process

Is defect density Is productivity

Question being reduced increasing over time?
How well are inspections
performing?
over time? What is the return
of inspections?
What are high-leverage
opportunities for defect What are optimal
prevention? How effective have inspection parameters? Is the inspection
process changes been? process under control?

Metric inspection defect removal

inspection ROI efficiency effectiveness

defect density trends productivity trends inspection rate

effectiveness vs. control chart
defect
inspection rate
categorization effectiveness vs. defect finding rate
# of inspectors control chart
23
 Determining what to measure
Measurement and process improvement
• Measurement offers visibility into the ways in which the process, products
and resources, methods and technologies of software development relate
to one another.
• The measurement is useful for:
1. Understanding
2. Establishing a baseline
3. Assessing and predicting
Some development processes are more mature than others, changing
significantly with the people who work on the projects.

24
 Determining what to measure
• The SEI has suggested that there are five levels of process ranging from ad
hoc, repeatable, defined, managed and optimizing.
• The SEI distinguishes one level from another in terms of key process
activities going on at each level.
Overview of process maturity and measurement
Maturity level Characteristics Type of metrics to use
5. Optimizing Improvement feedback to Process plus feedback for
the process changing the process
4. Managed Measured process Process plus feedback for
control
3. Defined Process defined and Product
institutionalized
2. Repeatable Process dependent on Project management
individuals
1. Initial Ad hoc Baseline 25
Level 1 : Initial
- Inputs to the process are ill-defined; while outputs are expected, the
transitions from inputs to outputs is undefined and uncontrolled.
- Similar projects may vary widely in their productivity and quality
characteristics because of lack of adequate structure and control.
- Baseline measurements are needed to provide starting point for measuring
improvement as maturity increases.

26
Level 2: Repeatable
- It identifies the inputs and outputs of the process, the constraints and the
resources used in the final product.
- The process is repeatable: proper inputs, proper outputs, but there is no
visibility into how the outputs are produced
Control
(Budget Schedule
Standards)
Input Construct the Output code
Requirements system Documentation

Resources
(Staff Tools)

A Repeatable process
27
Level 3 : Defined
- The defined process provides visibility into the “construct the system” box
- In this level the intermediate activities are defined and their inputs and
outputs are known and understood.
- All process can be examined, measured and assessed.

A defined process

Design Method Inspection criteria Test Plans

Require Define Code, System S/W

integrate
ments design unit test

Tools, Staff Tools, Staff Tools, Staff

28
Level 4: Managed

- A managed process adds management oversight to a defined process.

- You can compare contrast, the effects of changes in one activity can be traced in
the others.

- The feedback determines how resources are deployed.

- You can evaluate the effectiveness of process activities: how effective are
reviews? Configuration management? Quality assurance?.

- A significant difference between level 3 and 4 is that level 4 measurements

reflects characteristics of the overall process and of the interaction among and
across major activities.

- Management oversight relies on a metrics database that can provide

information about such characteristics as distribution of defects, productivity
and effectiveness of tasks, allocation of resources, and the likelihood that
planned and actual values will match.

29
Level 5 : Optimizing

- Here measure from activities are used to improve process, possibly by removing
and adding process activities, and changing the structure dynamically in response
to measurement feedback.

30
•Combining GQM with process maturity
For example
Is the set of requirements are maintainable?
Level 1: ? – ill defined requirements
Level 2 : ? – well defined and you can collect additional requirements
Level 3 :? – your visibility into the process is improved
Level 4: ?
Level 5: ?

31
 Applying the framework
• In high level view of software measurement and its
objectives, many of the activities may have seemed
unrelated.

• We need to look at which processes, products and

resources are relevant in each case, which attributes
we are measuring and whether these are internal or
external and whether the focus is on assessment or
prediction.

32
 (Contd)
– Cost and effort estimation
– Productivity measures and models
– Data collection
– Quality models and measures
– Reliability models
– Performance evaluation and models
– Structural and complexity metrics
– Capability-maturity assessment
– Management by metrics
– Evaluation of methods and tools

33
1. Cost and effort estimation

34
 (Contd)
• Where a and b are parameters determined by the
type of software to be developed.
• The model is intended for use during requirements
capture, when estimates of effort are needed.
• To use this model we need to determine the
parameters a and b.
• Bohem provides three choices for these constants,
- Determine the type of software system it is classified
it to be;
- Determine the size S of the eventual system;
- Since the system has yet to be built, we must predict
S in order to predict E.

35
• Thus it is more correct to view COCOMO as a
prediction system rather than as a model.
• Cost models often reflect a variety of attributes,
representing all of the entity types.
• For example, for more advanced versions of
COCOMO as well as other cost estimation
approaches, the inference procedure involves
numerous internal and external attributes of the
requirement specification (product) , together
with process and resource attributes subjectively
provided by users.

36
2. Productivity measures and
models

37
3. Data Collection

38
 4. Quality models and measures
• Quality models usually involve product
measurement, as their ultimate goal is
predicting product quality.
• In chapter 1, we saw the cost and productivity
depends on the quality of products output
during various process.

39
5. Reliability models

40
6. Performance evaluation and models
• Measuring efficiency and product
attribute(time efficiency , space efficiency).
• Algorithmic algorithm:-high level algorithm
determine good predictor of efficiency of the
• implemented code , time efficiency
• predictor

41
 7. Structural and Complexity metrics
•Single internal attribute representing
complexity can be accurate predictor of many
external quality attribute as well as process
attribute like cost and effort.
• Predictor

42
8. Capability – maturity assessment

43
 9. Management by metrics
•Management use metrics to set targets for
their development projects.
• Target derived from best practice.
• E.g.->Existing project

44
 10. Measurement of Methods and
Tools
•Investing new method or tools but hesitate
because of uncertainty about cost , utility,
effectiveness.
• Proposed tool tried in small project and result
are evaluated and used for later project.

45
 11. A Mathematician View of Metrics
• In mathematical analysis ,a metric has a very
specific meaning: it is a rule use to describe how
far two points
• Metric is a function m defined on pairs of objects
x and y such that m(x,y) represents the distance
between x and y.
• It should satisfy some properties:
• – M(x,y)=0
• – M(x , y)=m(y , x)
• – M(x,z)<=m(x,y)+m(y,z)

46