Agile Testing Project Estimation – Some Thoughts

“Agile Testing Project Estimation – Some Thoughts”

Govind Kulkarni
Test Manager,
Testing Center of Excellence (TCOE)
Zensar Technologies Ltd, Pune, India
Kharadi Knowledge Park,
Off. Nagar Road,
Pune -411014
India
www.zensar.com
[email protected]

“2nd Annual International Estimation Colloquium 2007, Bangalore, August

03, 2007, India

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 1 of 12
Agile Testing Project Estimation – Some Thoughts

Abstract:-

Due to business pressure and hurry to market the products, many of the projects executed
these days make use of agile development methodology. Unlike the waterfall model,
where completion of a predecessor phase becomes a mandated input for the subsequent
phases, agile development methods use an iterative development approach. In this
methodology, the whole scope of system development is divided in to several iterations.
Each iteration is a time boxed activity and tries to incorporate” Value Added” business
functionality. This gives a chance for the end users to use the partial/fully developed
system and provide an early feedback. The change requests and the defect fix
implementation are managed in iterations by prioritizing as per business needs.

The agile development methodologies are basically used for the complete system
development, and testing forms only one of the phases. Normally, the estimation would
consider for the complete system development including the testing phase. However, off
late, the customers engage multiple partners for system development. In this model, the
development is done by one partner and the testing is done by an independent laboratory
or an outsourced company.

In such situation, it becomes necessary for the testing partner to arrive at a separate
estimate for all the testing activities. The Testing partner needs to work in tandem with
other partners in completing the whole system development and provide adequate inputs
for testing activities. It is quite likely that, the client might ask for one single estimate
from multiple partners.

In this tutorial, we discuss some of the thoughts that could be used while arriving an
estimate for testing phase considering that a partner is needed to provide separate
estimates. We used these thoughts while working in a project which used SCRUM as an
agile development methodology.

The test managers could use these thoughts while coming up with an estimate for a
testing phase in an agile development project.

Key words: - Agile, Changes, Estimation, Iteration, System, Defects.

1.0 Introduction:-

1.1. What is agile development?

Many different software development methodologies have evolved over last fifty years.
Waterfall, Spiral and now we have reached a stage where agile methodologies have
emerged as most promising for system development. There are several agile

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 2 of 12
Agile Testing Project Estimation – Some Thoughts

methodologies which have come up such as XP (Extreme Programming), SCRUM,

Rational Unified Process (RUP), DSDM. Software development projects use these
methodologies as per the client advice or as agreed mutually with other partners. The
agile methodologies are considered as light weight, faster and reliable to develop and
deliver the system to the business community.

The following schematic diagram explains about agile methods implementation for
system development.

In the agile development projects, the whole system development scope is divided in to
several iterations. Each iteration :-

- Is planned in advance in mutual agreement with business community.

- Receives a set of requirements as an input. This forms the scope of the iteration.
Normally the scope is limited and which can be completed in a few days of time.
- Requirements are analyzed.
- Requirements are converted in to a design - UI, Database, program
- Design is converted in to a code.
- The code is tested – more thrust on developers doing unit tests using a test driven
approach. The testing team concentrates on conducting all types of tests needed
for the iteration.
- And the iteration is delivered as a release once it passes through the acceptance
criteria set out by the testing team.

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 3 of 12
Agile Testing Project Estimation – Some Thoughts

This iterative cycle gives flexibility. By the time a particular iteration has reached a
testing stage and the application is under testing and correction, next iteration may
commence with analysis and design activities. This parallel activity is encouraged to
ensure that the time is utilized properly.

This iterative process cycle (Analysis, Design, Code and Test) keeps on rolling till the
whole system is developed and continues even after the product is in implemented in
production.

Each iteration is a time boxed activity and tries to incorporate value added functionality
for the end users. The scope of the iteration is decided with the business users in advance
and features, change requests, defect fixes are prioritized.

Small but frequent release to the end users, and their involvement during UAT at each
iteration, gives them an opportunity to use the partial or a something “working” system
and provide feedback to the business analysts. The user’s early feedback is incorporated
in the subsequent iterations along with change requests or the defect fixes found while
testing.

As the iterations are undertaken for development and released to production, the software
is enhanced to correct defects and to incorporate new functionality. It depends on the
business which feature is needed urgently and which one can wait, so is the case with a
defects fix, a defect may be severe, may be throwing an error but business hardly uses it,
so fixing it may not make sense in current iteration, however a defect which irritates the
user for a regularly used functionality must be corrected and deployed. This is
fundamental point to consider for the scope of iteration.

One of the cornerstones of agile development is that each iteration can have length at the
most a month. Kent Beck [8] suggests the iteration length as even two weeks.

As for the testing is concerned, agile methodologies focus more on conducting early tests
particularly at the unit tests level. The developers are encouraged to write test cases
before they write the code. The dedicated testing team will complete required tests and
once their opinion of “Go” is taken the system is released to the production.

All iterations may not run exactly as planned, some iteration, particularly incorporating
complex functionality, may run behind the schedule and might create a “backlog”. In
order to control “Huge backlog” agile methodologies (SCRUM) suggest having daily
“stand up” meeting with the team. The stand up meeting form one of the important status
tracking tools to find out what the development team, test team has achieved, what issues
they are facing and the bottlenecks and corrective actions. Normally, for a testing team,
involvement of developers, configuration management team is absolutely necessary to
discuss any burning issues faced during testing.

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 4 of 12
Agile Testing Project Estimation – Some Thoughts

1.2. Testing in the context of agile development

Where as, agile implementation fundamentally does not change the basic activities of
testing but the way testing estimation is to be provided will certainly change. Because the
scope of iteration in I1 (first iteration) will be different than in the subsequent iterations like I2,
I3 … and In.

In the initial phases of system development, iteration (I1) might have a very limited scope
with minimum features, but as the system development proceeds and more iteration are
released to production, there will be a need for new features, defect fixes both requiring
to do additional regression testing..

Each of the iteration will contain at least 4 activities followed by an implementation.

- Analysis - analysis of a requirements

- Design -
- Code - Coding of required scope
- Testing – Test case generation, reviews, test execution and defect re testing.

√ The tests at the initial iterations are more rigorous. Until the system gets ready, all
types of testing must be done to ensure that the system is stable enough. This will
involve testers spending effort in various types of tests with a clear focus on
functionality testing. If initial iterations result in several defects and those remain un
attended, the subsequent iterations will need more testing efforts.

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 5 of 12
Agile Testing Project Estimation – Some Thoughts

√ When doing testing of subsequent iterations, the test team has to consider not just for
conducting tests for new functionality but also regressing previous functionality
releases such as R1, R2 … Rn-1 also verifying the defect fixes of current release and
the retesting of important fixes of previous releases.
√ The larger the scope the more the testing will be needed. Normally in agile
implementation, the iteration scope is limited to implement in less than 1-2 weeks or
so.
√ Some iteration may have shorter duration of less than fortnight or other iterations may
take exactly a moth or so depending on the scope of iteration. Therefore, the test
activities have to be appropriately planned as per iteration length.
√ As needed, each iteration will require the testers to create or modify test cases to meet
business scope.
√ The coding and testing phases in each iteration may end up having “code and fix” or
“test and re test” sub iterations in it. This is due to the fact that, the application may
be buggier than expected and may need several code drops (builds) requiring
functionality tests.
√ Performance, security and Usability tests are planned as the application functionality
is delivered and is testable.
√ The test team will prioritize what tests are relevant for the iteration. It may so happen
that, performance testing is not needed in the current release and can be deferred to
next iteration. These decisions are mutually agreed with the business.
√ Since the testing phase comes as the last phase before implementation, the tester’s
focus will be more on the current iteration and making it successful than planning for
the subsequent iteration. However, the test manager might decide to keep aside a few
resources to exclusively work on next iteration planning, understanding the scope,
creating test cases and reviews with business.
√ Tester’s main consideration in agile implementation is
ƒ What is in “scope” of this iteration”
ƒ Do we have test cases? Or do we need to create them? Or existing test
cases need modification?
ƒ How much time we have for testing?
ƒ What tests are planned? Is it just functional or other type of tests?
ƒ How many testers will focus on next iteration and how many will
concentrate on current iteration?
o The test manager arranges to provide answers for all above questions in a
day to day meeting with the testing team.

2.0 Agile Test Estimation Thoughts

The following are the important points to consider while we do estimation for testing
only activities in an agile development. The test manager has to work closely with all
other stakeholders in compiling necessary information for making plan and
communicating the decisions in terms of scope change, feature implementation,

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 6 of 12
Agile Testing Project Estimation – Some Thoughts

priorities, and test types to the testing team members. In any case, the test manager has to
provide the effort for following headings

1. Test management effort

2. Test case generation effort
3. Testing effort
4. Regression effort
5. Review effort

The estimation is usually required on two cases as mentioned below.

Case 1: - Estimating testing effort for the complete system development.

Normally, the stakeholders, including the client, will like to know how much testing
effort will be required to test the system. This kind of approximation is generally required
at the beginning itself. If a test manager is estimating for the complete testing effort then
the following points can be helpful. Also, the normal estimation methods such as
analogy, wideband Delphi, three point estimates could be employed to arrive at total
testing effort.

√ Size of the whole system in terms of Function Points, Lines of code, Use case
points.
√ Divide the system, screens, and modules in Simple, medium and complex.
√ Arrive at possible number of test cases.
√ Find out organization benchmark for test case generation and execution
productivity.
√ Find the possible types of tests to conduct.
√ Find out possible environments to test.
√ Find out the risks.

Based on the above information, the test manager could provide the information such as

√ The duration, ex. The total testing effort required for this system is 5600
hours
√ Number of Resources needed ex. The total testing resources needed for this
system to test is 20
√ Cost of the testing – only when demanded and needed

Consider that you will test all test cases in all environments while estimating
testing effort. From this idea, come out with an approximation.
Case 2: - Estimating testing effort for the iterations

The test manager will have a fair idea of the total testing effort that will be spent over
period of system development. However, the system is developed in iterations and in
each iteration the testing scope will differ. Due to this fact, the test manager is needed to
provide estimation at the beginning of each iteration. This information is needed to

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 7 of 12
Agile Testing Project Estimation – Some Thoughts

decide the length of each iteration. Appropriate iteration length need to be fixed based on
scope and testing needs.

The following are the points that will be helpful in arriving at testing estimation at the
iteration level.

√ What types of testing have to planned for the current iteration? (Smoke,
functionality, regression, UAT ...) – Note the more the types of testing, the more
effort you will need to invest.
√ What is the likely or minimum and maximum duration of each iteration?
o In a small duration, say a 2 week, iteration, all types of testing may not be
possible to fit in the timeline unless there is enough resource available.
√ How many environments are available for testing? Is it only Test and Stage
environment? Or any other environment to test? Note: The more the environments
the more the tests will be required to be conducted
√ Do we have any plan/scope to automate the test cases to reduce repetitive testing?
Note: if there is automation suite, it perhaps may reduce regression testing effort.
√ How many testers can be earmarked for exploring next iteration? These testers
will understand scope of next iteration, create as many test cases and participate
in reviews. The testers may not be able to contribute in the current release.
√ What is the attrition level of testers in this project? If there is much “in/out” of
testers then the more the effort for knowledge transfer and to bring them to a
productive level.
√ What is test team strategy to cope up with last minute business changes resulting
in more tests?
√ What is the effort in hours for test management activities? That is an industry
practice? Usually my experience is that 10% of testing effort go in test
management activities.
√ In each iteration, the minimum testing work, that a tester can expect is

o New functionality
o Defect fixes
o Change requests
This will call for an effort from each tester to

o Write test cases

o Modification of automation suite
o Reviews
o Test Execution
o Regression
o Defect fix and Change implementation

The test manager has to constantly discuss with the test team members to know
the testing effort. This could be done by way of constant interaction, discussion
with each test member.

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 8 of 12
Agile Testing Project Estimation – Some Thoughts

√ As the system is enhanced with more and more features, it starts to increase in
size. The effect is that, it will require more time for doing regression, defect fix
verification. Please follow below schematic diagram.

The test manager must consider the points below.

√ Which iteration we are asked for test estimation?

√ What are the new features in implementation?
√ What are the defects fixes?
√ What tests are planned in the iteration?

It is also recommended that, the test manager discusses with business to arrive at testing
scope before arriving at an estimate.

√ Bigger iteration does not mean test manager can ask for more number of people or
more time from the management or from client, smart thinking is needed as
below.

o Is it worth testing a functionality which did not have any defects in

production or there was no complaint? And no changes are initiated in it
since last a few iterations. Note: More risk means more test, less risk
means less test and no risk means perhaps no test.

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 9 of 12
Agile Testing Project Estimation – Some Thoughts

o Is there a dependency, if a module x is fixed? Is it dependent on other

modules? If not, is it worth doing testing for dependent modules also?
o How is the defect rate per release? Which modules are more risky for the
business? And which modules were buggier? Note: Buggier the modules
more tests need to be conducted to make them stable.
o Can we test using a basic automation to check quickly on positive test
cases?
o Can we test only new functionality and run priority 1 test cases for other
functionality?

The test manager can use above guidelines when required to give an effort estimate for
the release. Based on the test effort estimation, the iteration length could be decided.

3.0 Summary

™ Agile methodologies have more thrust on conducting unit testing at the

development stage itself.
™ XP recommends working 40 hours per week. If the team spends more hours or
week ends to make up the schedule then needs “Re estimation”
™ Pekka Abrahmson in his article “Agile Software development methods” [1]
suggests possible breakdown of software development activities effort. This
information could be used while evaluating or comparing the efforts.
• Analysis - 16%
• Design - 17%
• Code/Unit Test - 34%
• System/Integration Test - 18%
• Documentation - 8%
• Implementation/Install - 7%

™ The test manager must be always on toe to do re estimation in order to cover up

any backlog, delays in testing or delay in getting the builds for testing or fixing a
blocker defects.
™ Every day “stand up” meeting with the testing team to discuss – what was planned
to be done? What was actually done? And what is planned to do till next? Will
greatly help in controlling the delays in testing activities.
™ The test manager needs to consult each of the team members on estimation of
their own tasks in iteration. This will give a fair idea as to what could be achieved
and what can’t be?
™ Estimation of test activities particularly when there is a large test team is a
challenge due to increased communication.

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 10 of 12
Agile Testing Project Estimation – Some Thoughts

4.0 Reference

[1] Agile Software development methods – Review and Analysis – By, Pekka
Abrahmson, Outi Salo.
[2] An Introduction to Agile Methods - Steve Hayes (Khatovar Technology)
[3] Software Estimation – Steve McConnell. , Microsoft
[4] Estimating Software Costs – T. Capers Jones. McGraw Hill Edition
[5] Testingreflections.com -Submitted by Michael Bolton's Blog on Agile
[6] Pro Active Testing - Robin F. Goldsmith, JD – Go Pro Management, Inc
[7] The New Methodology – Martin Fowler, MartinFowler.com. Martin has compiled
very good material on using agile methods
[8] Extreme Programming Explained – By Kent Beck, Pearson Asia edition

5.0 About the Author

Govind has over 12 years of IT experience in software analysis, design, development,

and testing. Since last 6 years he has focused mostly on providing software testing
solutions for companies like British Telecom, Autodesk, Covad, MetLife and Euromax.
He has worked as a test team member, lead and test manager to provide testing
solutions in functional, automation and performance testing and tuning.

Govind is a M. Sc in Statistics followed by a Post Graduate degree in Statistical Quality

Control. He is also a Project Management Professional (PMP) from the PMI, USA,
Certified Quality Analyst from QAI, USA and certified ISO 9000 Quality Management
System auditor from A-Pex, U.K.

He is working with Zensar for last year and had managed to provide testing solutions
for Euromax Terminal, a Holland based terminal which is coming up at Rotterdam. The
challenge of the project was to manage a group of vendors (the Consortium) and also to
test the software (Terminal Administration and Planning) with actual equipment system
(Equipment Control System) the testing solutions was required to be provided by
creating test assets, executing the tests, reporting the test progress and formulating an
acceptance criteria to accept the integrated software for Business Acceptance testing and
implementation.

He has also worked in the Stress and Performance testing department at MetLife USA,
to ensure that the performance test challenge of meeting 250 concurrent users is met by
planning, executing, reporting and coordinating with various stakeholders to fine tune
the system.

He has worked in a project Covad, which used Agile development methodology using
Scrum.

He has published articles and papers in software testing conference conducted by QAI
and held in India.

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 11 of 12
Agile Testing Project Estimation – Some Thoughts

His current interests include defect prediction using Bayesian Statistical methods and
performance tests and tuning. He is also pursuing education in supply chain
management.

-----------------------------------------------------------------------------------------------------------
Govind Kulkarni - Test Manager, www.Zensar.com Page: 12 of 12