WHITE PAPER ON

DEFECTS CLUSTERING AND PESTICIDE PARADOX

AND HOW TO DEAL WITH IT



Overview Of Defects Clustering:
In the software testing world, Defects Clustering means that the majority of the
defects are caused by a small number of modules, i.e. the distribution of defects
are not across the application but rather centralized in limited sections of the
application. It is particularly true for large systems where the complexity, size,
change and developer mistakes can impact the quality of the system and affect
particular modules.
It is based on the Pareto principle, also known as the 80-20 rule, where it is stated
that approximately 80% of the problems are caused by 20% of the modules.
It's a phenomenon that is very commonly observed by almost all of the testers
and basically happens because an area of the code is complex or tricky. Test
designers often use this information when making the risk assessment for
planning the tests, and will focus on these known areas that may also be called as
the 'Hot spots'.
Let's take an example:
The following image shows that the majority of the defects are
concentrated/clustered around a few particular areas of the application, viz. the
Product Details, Reviews and Search Results Page for an e-commerce website and
the remaining defects in other areas.





This gives a good indication that when a defect is found in one area of the
application, chances are there are more defects in that particular area, so it is
worth investing more time to test that particular area of the application to find as
many defects as possible. Defect clustering also indicates which area of the
application needs more regression testing to ensure related features are not
broken.
The clustering may be layered in complexity. New test cases may extend the
previous tests down one layer without addressing the root cause of the defects.
Example: A tester submits a set of defects one iteration where the GUI doesn't
identify string overflows to the user. That is fixed with GUI field checks in all of the
identified places. Next iteration, a middle-tier defect is found when an long string
is entered in a field not addressed. Then the next iteration, database errors in the
log are found that are due to other fields that are not addressed. Instead, the
system architecture as a whole should have been evaluated initially to identify the
extent of the problem.


0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Filter
Search
Pricing
Details
Sorting
Priority



Source of Defects Clustering:

There are various reason that can lead to Defect clustering, listed below are a few
of the key ones:

- Defect clustering can occur because of poorly defined or communicated
requirements for a particular functionality. Ambiguity in the
requirements is one of the major factors that causes defects clustering.
If what the customer required was incorrectly comprehended by the
designers and thus, incorrectly coded by the developers; there lies a
huge risk that the part of the code would have a lot of defects.

- Defect clustering can occur because of the complexity of the
functionality. Complex functionalities when developed by developer
who didn't give enough thought to identify the various paths and
branches in the application that can be used to reach till the code in
question also results in higher defect density than should be expected.

- Defect clustering can occur due to developer's lack of experience or
expertise. This is one of the most critical causes of Defect clustering and
can be understood when we understand that not all the people are
equally skilled. Also, even if the developer is skilled, if he doesn't have
the business understanding of the requirement, he may not develop the
code as was expected.

- Defect clustering can occur when different developers have fixed several
defects on the same functionality. Even defect fixes can lead to new
defects, generally called as Introduced Defects. Defect Clustering can
also occur if the functionality was not properly/ rigorously tested before.



Introduction to Pesticide Paradox:

Pesticide Paradox states that if we ran the same tests over and over again, we
would probably find the amount of new defects identified would decrease over
time and eventually would not show any new defects as all the defects that could
be found using the initial test cases would have been fixed but in the process
more subtler bugs are introduced into the system. It basically states that the
static test cases become stale and unproductive over time.
However, there are other possible reasons for the diminishing returns as well.
One explanation is the fallacy of exhaustive testing. Simply put, except for the
simplest of software applications it is simply not possible to identify and test
every possible scenario, so running the same tests over and over would result in a
large overlap of previously identified scenarios. Another reason is that a software
programs functionality may change over time, which again would invalidate
previously executed test cases.
When we are testing a new software against the user requirements, defects will
be finding at a large numbers in certain flow or area of the code which is more
complex or critical. When the same software or code is to be tested again &
again, after some changes or modifications, we will find the defects in initial few
iterations of testing by identifying & concentrating on the 'Hot spot'.
Tester can focus the same area in order to find the more number of defects. It will
reduce the time & cost of finding defects as by focusing on certain areas more
bugs can be find in order to improve quality of the software.
But after certain number of iterations of testing, as the testing improves we see
the defect numbers dropping, the most of the bugs will be fixed & the 'Hot spot'
area will be cleaned up. Developers will be extra careful in a places where testers
found more defects & might not look in other areas. Hence by executing the same
test cases will no longer help to find the more defects. The test cases need to be


revised, and new and different tests need to be written to exercise different parts
of the software or system to potentially find more defects.

Pesticide Paradox in Daily Practices:

An interesting element of the pesticide paradox is its impact in multiple
dimensions. Consider this:
- One may start correlating defects found with test cases and find that only
new test cases produce defects. One reason for that could be training. In a
quick, session-based test session the tester may identify many defects that
they are familiar with in the test cases. Likewise, developers begin to
identify those practices that produced those defects. This can be thought of
as a form of cognitive bias, where the creation of new test cases serves to
train the tester in different bug patterns. This continual establishment of
new and different patterns is one reason that session-based testing is so
effective when combined with a risk-based test area identification
framework.


- Each defect cluster may identify a specific flaw in the code development. If
the coders didn't identify the root cause of the flaws and change the
practices, then these clusters may occur many times in different places.
Was there a pattern to those clusters ? This is can be thought of as a form
of model-based testing, where we are looking for the underlying cause of
the clusters. Superficially zeroing in on clusters without this overall view of
the cluster patterns or modeling of underlying cause of the clusters would
have limited value in improving test quality.




Defects Clustering and Pesticide Paradox Related ?

When the test designers and the testers identify a Hot Spot, they tend to pay
more attention to the area at the cost of neglecting a few areas that have been
stable for some time. Pesticide paradox, that's a contradictory concept to Defect
Clustering states that if the same tests are repeated over and over again, the
same set of test cases will no longer find any new defect, hence test cases need to
be revised.
In order to deal with these two opposing situations, we may either:
1. Write whole new set of test cases to exercise different parts of the software.
2. Prepare new test cases & add them to the existing test cases.
In the first case, we will be finding more potential defects in an area where we did
not focused earlier or the area at which developer was not extra cautious as the
tester was not raising the defects from these areas. But by neglecting earlier
identified defect cluster, we are certainly taking a risk of giving less importance to
the area which used be very critical in order to find the more defects in earlier
iterations of testing.
In the second case, we can find the new potential defects in the new area as well
as we can focus on the earlier identified defect cluster. But on the other hand the
number of test case will become so large that it will increase the testing time & in
turn increase the cost of testing. Also, in the UTP (cost per test) based method in
which most of the companies now function, this approach may not be accepted.

How do we deal with this:
To work with this, we can approach the middle path for balancing the
disadvantages of both the listed choices. We can identify & remove the test cases


that are not much important & have failed to detect any defect in certain number
of test cycles.
For example, if we have 10 tests that cover the same area and none of them have
detected bugs in number of cycles (for example, 5 test cycles), then we should
review them and reduce the number of test cases.
In this way we can keep the check on number of useless test cases without
compromising with the quality as all the test cases covering important areas of
the software will be retained. We have to follow this method regularly & keep
updating/modifying our test cases whenever there is a change/modification
occurs in the software.
Although this is a method that is effective, it's very time consuming as it involves a
lot of manual intervention. Also, this move is a little risky as a few test cases may
be overlooked due to the test designer's bias/opinions.
Another effective technique is not to rely solely on the structured and formal
techniques but also to perform ad-hoc and exploratory testing to uncover bugs.
Test cases may be written to validate the functionalities against the requirements
and exploratory testing may be used to further delve into the why's and how's of
the functionalities so that the dormant defects may also be uncovered. Again, this
technique, although cost effective, may not be time effective. It may not be
possible for the testers to explore much of the code due to the stringent timelines
of the projects.
This is where Test Optimisation comes into picture.







Test Case and Coverage Optimisation:

An effective suite of well-designed regression test cases will not only provide a
baseline assessment of the current version, they can also be used during the
release of newer versions of the product. Obviously, testing only the section of
code which has been changed is more cost-effective than retesting the entire
suite; while certain tests should be run on the entire product, intensively testing
code which has not been edited is a waste of time. The simple adage of If it aint
broke, dont fix it can be applied to code which a tester already knows to be
functioning well. The main purpose here is to find the redundant test cases that
dont provide great value to the overall testing effort and are added to simply
increase the code coverage or inflate the number of tests.

How to achieve it:

- Regression test cases do not normally need to test bound, invalid data, etc.;
normally they will be designed and focused based on the design of the test
system. It is assumed that the above were tested prior to the regression
test effort in previous release, so we only run those tests that are most
likely to fail or had failed multiple times.

- Ensure test cases are managed according to risks, with a risk index attached
to each planned test case. Then we can check the risk against the new
software release to ensure they are still critical as they were in the previous
releases. This is known as Risk Based Testing or RBT. Its very important to
understand and establish the owner of the Risk if this approach is to be
followed.



- Using random test generators to create regression suites on the fly is
becoming more common. In this practice, instead of maintaining tests, we
generate test suites as we need them by choosing several specifications
and generating a number of tests from each specification. Test Case
Optimisation tools like Hexawise that increase the test coverage and reduce
the number of tests at the same time using the technique of Pair Wise
Testing should be used.

- Test case prioritization is also very commonly used to ensure test cases to
be executed are reordered to maximizing the score function. There are
many techniques which can predict the probability of each test case finding
faults in regression testing.

- Combination approach: any or all of the above methods can be used
together.


Challenges in Test Case Optimisation:

- In case of manual optimisation, it's important for the test designer to have
an in-depth knowledge of the application that is being tested, so that any
scenario that may be business critical doesn't get skipped.

- In case the test optimisation tools are use used, there lies an overhead cost
of attaining the license of the tool and providing training to the users

- During the initial phases of the test designers using the Tools, there might
be a few slip-ups because of lack of expertise and hands on knowledge.



Benefits Of Test Case Optimisation:

- Ensures maximum coverage in the least possible number of test cases,
thereby saving time and energy of the tester and revenue of the client.

- Each time a regression test suite is run it consumes computing resources.
For very large suites this could tie up machines specifically for the task of
running regression (continuously or otherwise). Test Optimisation takes
care of that.