SOFTWARE TESTING

(CSE440)
 Contents – UNIT E
Controlling and Monitoring
Unit E Test metrics and measurements –project,
progress and productivity metrics – Status
Meetings – Reports and Control Issues – Criteria
for Test Completion – SCM , Developing a review
program – Components of Review Plans–
Reporting, Review Results. – evaluating software
quality – defect prevention – testing maturity
model
 Software Test Metrics
Software Testing Metrics are the quantitative measures used to estimate the
progress, quality, productivity and health of the software testing process.
A Metric defines in quantitative terms the degree to which a system, system
component, or process possesses a given attribute
Test metrics example:
How many defects exist within the module?
How many test cases are executed per person?
What is Test coverage %?

Why Test Metrics are Important?

• Take decision for next phase of activities
• Evidence of the claim or prediction
• Understand the type of improvement required
• Take decision or process or technology change
Types of Test Metrics
• Process Metrics: It can be used to improve
the process efficiency of the SDLC ( Software
Development Life Cycle)
• Product Metrics: It deals with the quality of
the software product
• Project Metrics: It can be used to measure the
efficiency of a project team or any testing tools
being used by the team members
 Process Metrics:
• Software Test Metrics used in the process of test preparation and test execution phase
of STLC.
• The following are generated during the Test Preparation phase of STLC:
• Test Case Preparation Productivity:
• It is used to calculate the number of Test Cases prepared and the effort spent for the
preparation of Test Cases.
• Formula:
Test Case Preparation Productivity = (No of Test Case)/ (Effort spent for Test Case
Preparation)
E.g.:
No. of Test cases = 240
Effort spent for Test case preparation (in hours) = 10
Test Case preparation productivity = 240/10 = 24 test cases/hour
• Test Design Coverage:
• It helps to measure the percentage of test case coverage against
the number of requirements
• Formula:
Test Design Coverage = ((Total number of
requirements mapped to test cases) / (Total number of
requirements)*100
• E.g.:
Total number of requirements: 100
Total number of requirements mapped to test cases: 98
Test Design Coverage = (98/100)*100 = 98%
• The following are generated during the Test Execution phase of STLC:
• Test Execution Productivity:
• It determines the number of Test Cases that can be executed per hour
• Formula:
• (No of Test cases executed)/ (Effort spent for execution
of test cases)
• E.g.:
No of Test cases executed = 180
Effort spent for execution of test cases = 10
Test Execution Productivity = 180/10 = 18 test cases/hour
• Test Execution Coverage:
• It is to measure the number of test cases
executed against the number of test cases
planed.
• Test Execution Coverage = (Total no. of test cases executed / Total no. of
test cases planned to execute)*100
• E.g.:
Total no. of test cases planned to execute = 240
Total no. of test cases executed = 160
Test Execution Coverage = (180/240)*100 = 75%
• Test Cases Passed:
• It is to measure the percentage no. of test cases passed
• Test Cases Pass = (Total no. of test cases passed) /
(Total no. of test cases executed) * 100
E.g.:
Test Cases Pass = (80/90)*100 = 88.8 = 89%
Test Cases Failed:
It is to measure the percentage no. of test cases failed
Formula:
• Test Cases Failed = (Total no. of test cases failed) / (Total no. of test cases
executed) * 100
• E.g.:
• Test Cases Failed= (10/90)*100 = 11.1 = 11%
• Test Cases Blocked:
• It is to measure the percentage no. of test cases
blocked
• Test Cases Blocked = (Total no. of test
cases blocked) / (Total no. of test cases
executed) * 100
• E.g.:
• Test Cases Blocked = (5/90)*100 = 5.5 = 6%
 PRODUCT METRICS
• Product metric:
• Software Test Metrics used in the process of defect analysis phase of
STLC.
• Error Discovery Rate:
• It is to determine the effectiveness of the test cases.
• Error Discovery Rate = (Total number of defects found
/Total no. of test cases executed)*100
• E.g.:
• Total no. of test cases executed = 240
• Total number of defects found = 60
• Error Discovery Rate = (60/240)*100 = 25%
• Defect Fix Rate:
• It helps to know the quality of a build in terms of defect fixing.
• Formula:
• Defect Fix Rate = (Total no of Defects reported as fixed - Total no. of
defects reopened) / (Total no of Defects reported as fixed + Total no.
of new Bugs due to fix)*100
• E.g.:
• Total no of defects reported as fixed = 10
• Total no. of defects reopened = 2
• Total no. of new Bugs due to fix = 1
• Defect Fix Rate = ((10 – 2)/(10 + 1))*100 = (8/11)100 = 72.7 = 73%
• Defect Density:
• It is defined as the ratio of defects to requirements.
• Defect density determines the stability of the application.
• Formula:
• Defect Density = Total no. of defects identified / Actual Size
(requirements)
• E.g.:
• Total no. of defects identified = 80
• Actual Size= 10
• Defect Density = 80/10 = 8
• Defect Removal Efficiency:
• It allows us to compare the overall (defects found pre and post-delivery) defect removal
efficiency
• Defect Removal Efficiency = ((Total no. of defects found pre-delivery) /( (Total no. of
defects found pre-delivery )+ (Total no. of defects found post-delivery)))* 100
• E.g.:
• Total no. of defects found pre-delivery = 80
• Total no. of defects found post-delivery = 10
• Defect Removal Efficiency = ((80) / ((80) + (10)))*100 = (80/90)*100 = 88.8 = 89%
 What is Quality Management
• Also called software quality assurance (SQA)
• Serves as an umbrella activity that is applied throughout the
software process
• Reduces the amount of rework, which results in lower costs and
improved time to market
• Encompasses
– A software quality assurance process
– Specific quality assurance and quality control tasks (including
formal technical reviews and a multi-tiered testing strategy)
– Effective software engineering practices (methods and tools)
– Control of all software work products and the changes made
to them
– A procedure to ensure compliance with software
development standards
– Measurement and reporting mechanisms
 Quality Defined
• Defined as a characteristic or attribute of something
• Refers to measurable characteristics that we can
compare to known standards
• In software it involves such measures as cyclomatic
complexity, cohesion, coupling, function points, and
source lines of code
• Includes variation control
– A software development organization should strive
to minimize the variation between the predicted
and the actual values for cost, schedule, and
resources
– One goal is to ensure that the variance in the
number of bugs is also minimized from one release
to another
 Quality Defined (continued)
• Two kinds of quality are sought out
– Quality of design
• The characteristic that designers specify for an item
• This encompasses requirements, specifications, and the
design of the system
– Quality of conformance (i.e., implementation)
• The degree to which the design specifications are
followed during manufacturing/ development
• This focuses on how well the implementation follows the
design and how well the resulting system meets its
requirements
• Quality also can be looked at in terms of user satisfaction for
software
User satisfaction = compliant product + good quality
+ delivery within budget and schedule
 Quality Control
• Involves a series of inspections, reviews, and tests used
throughout the software process
• Ensures that each work product meets the requirements
placed on it
• Includes a feedback loop to the process that created the
work product
– This is essential in minimizing the errors produced
• Combines measurement and feedback in order to adjust
the process when product specifications are not met
• Requires all work products to have defined, measurable
specifications to which practitioners may compare to the
output of each process
 Quality Assurance Functions
• Consists of a set of auditing and reporting
functions that assess the effectiveness and
completeness of quality control activities.
• Provides management personnel with data that
provides insight into the quality of the products
• Alerts management personnel to quality problems
so that they can apply the necessary resources to
resolve quality issues
 The Cost of Quality
• Includes all costs incurred in the pursuit of quality or in performing
quality-related activities
• Involves various kinds of quality costs
– Prevention costs: Quality planning, formal technical reviews, test
equipment, training
– Appraisal (Evaluation) costs: Inspections, equipment calibration and
maintenance, testing
– Failure costs – subdivided into internal failure costs and external
failure costs
• Internal failure costs: Incurred when an error is detected in a
product prior to shipment & Include rework, repair, and failure
mode analysis
• External failure costs: Involves defects found after the product
has been shipped & Include complaint resolution, product return
and replacement, help line support, and warranty work
• Increases dramatically as the activities progress from
– Prevention Detection Internal failure External failure

"It takes less time to do a thing right than to explain why you did it wrong." Longfellow
 Software Quality Defined
• "Conformance to explicitly stated functional and performance
requirements, explicitly documented development standards, and implicit
characteristics that are expected of all professionally developed software“
• This definition emphasizes three points
– Software requirements are the foundation from which quality is
measured; lack of conformance to requirements is lack of quality
– Specified standards define a set of development criteria that guide the
manner in which software is engineered; if the criteria are not followed,
lack of quality will almost surely result
– A set of implicit requirements often goes unmentioned; if software fails
to meet implicit requirements, software quality is doubtful
• Software quality is no longer the sole responsibility of the programmer
– It extends to software engineers, project managers, customers,
salespeople, and the SQA group
– Software engineers apply solid technical methods and measures,
conduct formal technical reviews, and perform well-planned software
testing
 The SQA Group
• Serves as the customer's in-house representative
• Assists the software team in achieving a high-quality
product
• Views the software from the customer's point of view
– Does the software adequately meet quality factors?
– Has software development been conducted according to
pre-established standards?
– Have technical disciplines properly performed their roles
as part of the SQA activity?
• Performs a set of activities that address quality assurance
planning, oversight, record keeping, analysis, and
reporting.
 SQA Activities
• Prepares an SQA plan for a project
• Participates in the development of the project's software
process description
• Reviews software engineering activities to verify compliance
with the defined software process
• Audits designated software work products to verify
compliance with those defined as part of the software
process
• Ensures that deviations in software work and work products
are documented and handled according to a documented
procedure
• Records any noncompliance and reports to senior
management
• Coordinates the control and management of change
 Software Reviews: Purpose of Reviews
• Serve as a filter for the software process
• Are applied at various points during the software process
• Uncover errors that can then be removed
• Purify the software analysis, design, coding, and testing activities
• Catch large classes of errors that escape the originator more than
other practitioners
• Include the formal technical review (also called a walkthrough or
inspection)
– Acts as the most effective SQA filter
– Conducted by software engineers for software engineers
– Effectively uncovers errors and improves software quality
– Has been shown to be up to 75% effective in uncovering
design flaws (which constitute 50-65% of all errors in software)
• Require the software engineers to expend time and effort, and
the organization to cover the costs
 Formal Technical Review (FTR)
• Objectives
– To uncover errors in function, logic, or implementation for any
representation of the software
– To verify that the software under review meets its requirements
– To ensure that the software has been represented according to
predefined standards
– To achieve software that is developed in a uniform manner
– To make projects more manageable
• Serves as a training ground for junior software engineers to observe
different approaches to software analysis, design, and construction
• Promotes backup and continuity because a number of people become
familiar with other parts of the software
• May sometimes be a sample-driven review
– Project managers must quantify those work products that are the
primary targets for formal technical reviews
– The sample of products that are reviewed must be representative of
the products as a whole
 The FTR Meeting
• Has the following constraints
– From 3-5 people should be involved
– Advance preparation (i.e., reading) should occur for each participant but
should require no more than two hours a piece and involve only a small
subset of components
– The duration of the meeting should be less than two hours
• Focuses on a specific work product (a software requirements specification, a
detailed design, a source code listing)
• Activities before the meeting
– The producer informs the project manager that a work product is
complete and ready for review
– The project manager contacts a review leader, who evaluates the
product for readiness, generates copies of product materials, and
distributes them to the reviewers for advance preparation
– Each reviewer spends one to two hours reviewing the product and
making notes before the actual review meeting
– The review leader establishes an agenda for the review meeting and
schedules the time and location
 The FTR Meeting (continued)
• Activities during the meeting
– The meeting is attended by the review leader, all reviewers, and the
producer
– One of the reviewers also serves as the recorder for all issues and decisions
concerning the product
– After a brief introduction by the review leader, the producer proceeds to
"walk through" the work product while reviewers ask questions and raise
issues
– The recorder notes any valid problems or errors that are discovered; no
time or effort is spent in this meeting to solve any of these problems or
errors
• Activities at the conclusion of the meeting
– All attendees must decide whether to
• Accept the product without further modification
• Reject the product due to severe errors (After these errors are
corrected, another review will then occur)
• Accept the product provisionally (Minor errors need to be corrected but
no additional review is required)
– All attendees then complete a sign-off in which they indicate that they took
part in the review and that they concur with the findings
 The FTR Meeting (continued)

• Activities following the meeting

– The recorder produces a list of review issues that
• Identifies problem areas within the product
• Serves as an action item checklist to guide the producer
in making corrections
– The recorder includes the list in an FTR summary report
• This one to two-page report describes what was
reviewed, who reviewed it, and what were the findings
and conclusions
– The review leader follows up on the findings to ensure that
the producer makes the requested corrections
 FTR Guidelines
1) Review the product, not the producer
2) Set an agenda and maintain it
3) Limit debate and rebuttal; conduct in-depth discussions
off-line
4) Enunciate problem areas, but don't attempt to solve the
problem noted
5) Take written notes; utilize a wall board to capture comments
6) Limit the number of participants and insist upon advance
preparation
7) Develop a checklist for each product in order to structure and
focus the review
8) Allocate resources and schedule time for FTRs
9) Conduct meaningful training for all reviewers
10) Review your earlier reviews to improve the overall review
process
 Statistical Software Quality Assurance:
Process Steps
1) Collect and categorize information (i.e., causes) about software
defects that occur
– Incomplete or erroneous specifications
– Misinterpretation of customer communication
– Intentional deviation from specifications
– Violation of programming standards
– Errors in data representation
– Inconsistent component interface
– Errors in design logic
– Incomplete or erroneous testing
– Inaccurate or incomplete documentation
– Errors in programming language translation of design
– Ambiguous or inconsistent human/computer interface
2) Attempt to trace each defect to its underlying cause (e.g.,
nonconformance to specifications, design error, violation of standards,
poor communication with the customer)
3) Using the Pareto principle (80% of defects can be traced to 20% of all
causes), isolate the 20%
 SQA Plan: Purpose and Layout
• Provides a road map for instituting software quality assurance in an
organization
• Developed by the SQA group to serve as a template for SQA activities
that are instituted for each software project in an organization
• Structured as follows:
– The purpose and scope of the plan
– A description of all software engineering work products that fall
within the purview of SQA
– All applicable standards and practices that are applied during the
software process
– SQA actions and tasks (including reviews and audits) and their
placement throughout the software process
– The tools and methods that support SQA actions and tasks
– Methods for assembling, safeguarding, and maintaining all
SQA-related records
– Organizational roles and responsibilities relative to product
quality
Quality Control vs Quality Assurance
 ISO 9000 Quality Standards
• The International Organization for Standardization (ISO) is an
independent, non-governmental organization made up of
members from the national standards bodies of over 160
countries that set international standards related to
products and services.
• ISO 9000 is defined as a set of international standards on
quality management and quality assurance developed to
help companies effectively document the quality system
elements needed to maintain an efficient quality system.
• ISO has published over 13,000 standards.
 Objectives of ISO Standards
• Achieve, maintain and improve product quality
• Improve quality of operations to continually meet
customers and stakeholders needs
• Provide confidence to management, employees,
customers, and stakeholders that quality
requirements are fulfilled.
ISO 9000 Series
ISO 9000 Series
ISO 9000 Series
Capability Maturity Model (CMM)

Reactive: Reacting to the past rather than anticipating the future.

Proactive: Acting before a situation becomes a source of confrontation or crisis.
 Capability Maturity Model (CMM)
• The Capability Maturity Model (CMM) is a methodology used to develop
and refine an organization's software development process.
• At the initial level, processes are disorganized, even chaotic. Success is
likely to depend on individual efforts, and is not considered to be
repeatable, because processes would not be sufficiently defined and
documented to allow them to be replicated.
• At the repeatable level, basic project management techniques are
established, and successes could be repeated, because the requisite
processes would have been made established, defined, and documented.
• At the defined level, an organization has developed its own standard
software process through greater attention to documentation,
standardization, and integration.
• At the managed level, an organization monitors and controls its own
processes through data collection and analysis.
• At the optimizing level, processes are constantly being improved through
monitoring feedback from current processes and introducing innovative
processes to better serve the organization's particular needs.
 CMM vs ISO
• The CMM is similar to ISO 9001.
• The main difference between the two systems
lies in their respective purposes:
– ISO 9001 specifies a minimal acceptable quality
level for software processes, while the CMM
establishes a framework for continuous process
improvement.