Mini Guide Functional Testing :
Governance based on testware
metrics
What to do when you want to outsource
your testing service to a Test Factory?
Ton Dekkers
Ignacio Lopez Carrillo
1
Why NESMA
About LEDAmc
The cuestion
Want to improve
Typical Problems
What to do
Examples
And in Agile
2
About Nesma
Ton Dekkers
Nesma|President The renewed website is
Interdependent | Software Cost Engineer organized into 6 themes:
ISBSG | Past President - Board
COSMIC | IAC member Benchmarking
Outsourcing
Partners Productivity
Galorath | US Project Control
Leda | ES Estimation
Metrieken.nl | NL Sizing methods
PRICE systems | UK
Vision
independent from customers and suppliers
nesma is the not-for-profit organisation in the area of predictability of the cost of the
delivery and the maintenance of software
nesma joins as much as possible existing standards with a different focus than measurement
nesma connects surrounding attention domains
nesma is independent from customers and suppliers
3
Nesma Miniguides Software metrics in contracts
1. Guideline for metrics in contracts (resume)
2. Development methodologies;
3. Maintenance;
4. Management;
5. RFP Questions;
6. Functional Quality;
7. Pricing Mechanisms;
8. Technical Quality;
9. Assessing Suppliers Performance;
10. Software Metrics in Contracts; [13] Mini
Guide:
English -- Free 10,= 20,= 40,= in progress
11. Requirements for Supplier organizations; Functional
Testing
12. Requirements for Customer organizations.
13. Functional Testing
http://nesma.org/publications/downloads/guides/software-metrics-in-contracts/
4
About LEDAmc
Spanish company
> 100 employees
Office in Madrid and Bogota
Customers in Europe, USA, LATAM and ASIA
Quantify Focussed in Outsourcing Management
Optimize > 70% FP certified consultants in Spain
ROI service commitment (47M 2015)
Projections
5
Risk SW Gobernance
Content Management
Lack of Risk Type
Methodology Complexity Schedule Over Effort Preassure
Information
Size
Persons
Technology Process Factors Risk SW Consecuences Economical
Time
Technology Gobernance
Management Imagen Performance
Experience Productivity Quality
Model
Test Design Productivity
Estimate
Test Productivity Test Execution Productivity Test Process Diagnostic
OPP Test Effectiveness Prevent Predict Agile
Projects
Score Card Testing
Test Gobernance Predictions
Decision Making Optimize Project Risk Sizing Team
Office
TMO Test Audit & Test QA
Quality Project Navigator
Strategy &Test Planning
Test Management Tracking reports
Delivery reports Gobernance Test Process Maturity
Test Design
Agile Testing
Testing Test Execution
Inside_
Estimaciones Testware
Functional Quality Benchmarking SOAPUI
ROI Pruebas
6
The question
7
The question
Esfuerzo de pruebas:
Disminucin del coste por reduccin de tarifa
Produccin de Pruebas (TFP):
Incremento de coste por perdida de
productividad de pruebas
8
Test Teams / Test Factories
Lack of this
Testware metrics
Some goes directly
enhance
?
capability
to this model
Lack of reference
values
Testing
factories Lack of results expected
enhance
?
effectiveness
RFP Testing Factory
Select wrong Testing
Factory supplier
enhance
performance ? Loss of time, money
and quality
Functional Testing
Miniguide
9
Typical Problems
Dont worry
Test Management
Different Test and Defect tracking Be busy
environment common tools
management
Different
Highly detailed Location
documentation
Organizational
Non Not enough
professional information in
profiles Quality Gates
Capability
?
High level Human Decision
of rotation resources making
Performance
No information
Effectiveness Subjetive in earlier stage
Payment Model
Savings (MH, TC)
Economical
Project
Planning delay
Over sizing
of testing
resources Duplication of Not objective
cost for getting testing estimation
good quality model Not control
Not measurable of outsourced
target of service
testing results
10
What to do to mitigate problems?
Four Stages during the testing outsourcing process to mitigate the problems :
RFP preparation: before the RFP adjudication
RFP adjudication: during the RFP adjudication
Service Operation: during the operation of the outsourced testing service
Close/Renewal of service: during the termination or renovation of the outsourced service
11
Activities
Getting reference Define testing Justify savings with
Define testing service Define Testware Sizing of human Selection of possible
values of testing ecosystem conditions outsourcing to testing
payment model estimation model Resources needed suppliers
activities of outsourced service factories
RFP preparation
Getting reference Analyze the size and
Define Non subjective Define size estimation Define outsourcing Sizing of testing Justify savings
values of testing business focus of the
payment model model of the project location resources in time
productivity suppliers needed
Getting reference Define model of
Define testing service Sizing of defect Ask for a RFQ to no Justify savings
values of testing Define size of testware outsourcing testing
ROI model removal resources more than six suppliers in Quality
effectiveness factory
Getting reference Define test
Define bonus/penalty Define testing level to Justify savings
values of current effort estimation Test design, Test execution,
scheme be outsourced in
testing cost model Test automation
Define Defect removal
Function Points tested Enhancement detail of
effort estimation
or any other functional documentation needed
model
Function Points or Designed test case , expected to be
size metrics
any other detected defects, expected incidence in
functional size production (during the guarantee period
metrics and Quality debt)
12
Activities
Define Testing Factory Define Testing Factory Prepare testing
RFP MUST RFP NTH ecosystem
RFP adjudication
Request for a Prepare organization of
Define ANS continuous operation with testing
improvement model tools Requirements, Test
Management, defect Tracking
Demand the use of a Guarantee access to
Define Quality Gates
test process reference different test
metrics
model not proprietary environments
Define protocol to start Select different blocks
Guarantee semi real
and finish the testing for two different
test data
service suppliers
ISTQB,
Facilitate communication
Request of a Control Include testing service
with developers: Defect
Panel for the TKPIs ROI model
removal
Request for a service
tracking model
13
Activities
Establish model to Extrapolate possible
Control testing metrics Audit the outsourced
reduce the testing evolution of the final
(TKPIs) testing service
resource needed quality results
Service Operation
Audit the level of Enhancement of the Related with Quality defects in the next
Control testing testing phases
details of the test case testing activities in the Gate and compared
productivity
designed testing levels with initial estimation
Audit the test Automatize tracking Propose evolution
Control testing
coverage, depth and service, quality and needed to get the
effectiveness
impact in the risk of test designed governance reports committed results
quality,
dates and quality
performance and Compare the TKPIs Propose possible new
effectiveness of the Audit the effectiveness
metrics by area (block), and more realistic
testing service of the testing service
supplier, technology, commitments
Compare the TKPIs Control Impact of Translate TKPIs
metrics by month and rotation of the hhrr evolution to economic
year assigned to the service data
14
Activities
Prepare next period of
Economical review Close Renewal
service
Close/ Renewal of service
Fix new TKPIs target
Estimate new needs for Execute transfer plan
Analyze ANS results for next period of
next period to new supplier
service
Perform continuous
Perform ROI report of Define target of
improvement model
the testing service enhacement
(TMMi)
Perform bonus/penalty
scheme
15
Examples RFP Preparation: Test-ware estimation
Estimaciones de Testware
Early test-ware size and effort estimations
16
Examples RFP Preparation: Sizing of testing resources
Estimaciones de Testware
17
Examples RFP Preparation: Testing Productivity figures
Caso de xito
18
Saving money Using the Mini Guide
Caso de xito
KPI Indicator Control No control KPI Indicator Control No control GAP %
Time to create a TC 24 39 Total TC designed by person/year 3.575 2.200 1.375 62,5%
Productivity Productivity
Time to execute a TC 32 48 Total TC executed by person/year 2.681 1.788 894 50,0%
% Bugs detected 81% 49% Total Bugs detected by person/year 203 75 127 168,6%
Effectiveness % Wrong bugs 8,0% 15,7% Efectiveness Total Wrong Bugs reported by person/year 16 6 10 168,6%
Bug Fixing 67,0% 26,7% Total Bugs fixed by person/year 136 20 116 574,8%
Project Size 5.236 PF No
# TC 28.275 Control control GAP %
Test size
Testware # Bugs 3.677
estimation TC Design 4.166 Size Test Team 8 13 -5 62%
Test effort
TD execution 5.024
Total Cost 417.600 588.120 170.520 29%
TC
28.275
17.400
Control
No control
0
1 year 1,6 years Time
19
Software Testing metrics in Agile projects
Estimaciones de Testware
Relation between History Point and Function Point
Quality debt by team (current and future)
Quality team ranking Agile yes, but not
Sprint amount needed by 100 implemented History Point kamikaze
Testing and SW
Cooperation Test automation effectiveness
metrics always
of all team
needed
Agile Test
Team: ATT
History Dont
Factories Point with compare
different persons
criteria and teams
Average test cases designed by person [/ day] / 1 designedTest Case Agile teams
Average test cases executed by person [/ day] / 1 executed Test case Releases
Average defects checked by person [/ day] / 1 detected defect Sprints
# detected defects by History Point / 1 avoided defect
% total errors detected QA / UATs / Production / 1 tested History Point
20
Software Testing metrics in Agile pojects
Estimaciones de Testware
Be agile,
flexible, but
not fragile
21
Estimaciones de Testware
22
