Scribd
Upload
78 views8 pages

Against Test Cases

The document criticizes an overreliance on "test cases" in software testing. It argues that test cases only describe a fraction of actual testing activities and that excellent testing is a richer, more open-ended intellectual activity that cannot be fully encapsulated into discrete procedural units. The document also claims that test case counts alone do not provide meaningful information about testing effectiveness. It advocates for managing testing as an activity-based process focused on exploration rather than as an artifact-based process focused solely on creating test cases.

Uploaded by

Bertie Back
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
78 views8 pages

Against Test Cases

The document criticizes an overreliance on "test cases" in software testing. It argues that test cases only describe a fraction of actual testing activities and that excellent testing is a richer, more open-ended intellectual activity that cannot be fully encapsulated into discrete procedural units. The document also claims that test case counts alone do not provide meaningful information about testing effectiveness. It advocates for managing testing as an activity-based process focused on exploration rather than as an artifact-based process focused solely on creating test cases.

Uploaded by

Bertie Back
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

5/19/2009

The Case Against Test Cases

James Bach
Satisfice, Inc.
[email protected]
www.satisfice.com

“Test Cases” describe only a


fraction of testing.
Programming cases?
Driving cases?
Traveling cases?
Parenting cases?
Learning cases?
Science cases?
Living cases?

Excellent testing is a rich and open-ended intellectual activity.


It cannot be encapsulated into discrete procedural units.

1
5/19/2009

What is a test case?

There is a lot of confusion.


There’s no specific definition in common use.
– “a computer executable script”
– “a sequence of discrete steps”
– “specific detailed input and action steps taken during the execution with
predetermined expected results”
– “inputs, execution conditions, and expected results”
– “a set of input values, execution preconditions, expected results and
execution post-conditions, developed to cover certain test condition(s).”
Different testers have different ideas about them.
Sometimes, a test itself is called a test case.

What is a test case?

An artifact.
A container. (“test cases are like briefcases”)
A model. (a pattern, idea, spec, or schematic for a test)
Part of a suite.
A mystery. (“why did he write the test that way?”)
A burden to the future. (is it broken? is it obsolete? is it
limiting my imagination or stealing my time that I could be using
to TEST BETTER?)
A measure of testing progress.
A unit of testing itself.

2
5/19/2009

What is a test case?

I use this definition:


A test case is one particular
instance or variation of a test
or test idea.

Compare to:
A test procedure is a way of performing a test.

A test activity is a line of investigation


that fulfills some part of the test strategy.
It can encompass many test cases.

A Testing Industry Pandemic:


OCD

Obsessive Case Disorder: confusing test cases with


testing, such that “creating test cases” is assumed to
encompass all the thinking of testing.
Obsessive Counting Disorder: the belief that counting
test cases tells you something meaningful about
testing.
Overspecified Cases Disease: the irrational
compulsion to treat humans like dimwitted robots by
creating recklessly specific test documentation.

3
5/19/2009

In the absence of context…


Test Case Counts Mean NOTHING!

How much testing is 40 test cases?


How much is 400?
How about 40,000 test cases?

You shouldn’t take


test case counts seriously because…
Test cases are not independent.
Test cases are not interchangeable.
Test cases vary widely in value from case to case, tester to
tester, product to product, project to project, test technique to
test technique, and over time.
Test case design is subjective, so counts are easy to inflate.
Test cases do not— and can not—capture all the testing that
occurs (example: bug investigation)
Testers often don’t follow the test cases, anyway.
Automated test cases are fundamentally different from sapiently
executed tests.
Test cases represent what’s easy to put into a test case.

4
5/19/2009

(unless you want to miss lots of bugs)

To test a very simple product meticulously,


part of a complex product meticulously,
or to maximize test integrity…

1. Start the test from a known (clean) state.


2. Prefer simple, deterministic actions.
3. Trace test steps to a specified model.
4. Follow established and consistent lab procedures.
5. Make specific predictions, observations and records.
6. Make it easy to reproduce (automation helps).

5
5/19/2009

To find unexpected problems,


elusive problems that may occur in the field,
or more problems quickly in a complex product…

1. Start from different states (not necessarily clean).


2. Prefer complex, challenging actions.
3. Generate tests from a variety of models.
4. Question your lab procedures and tools.
5. Try to see everything with open expectations.
6. Make the test hard to pass, instead of easy to reproduce.

Exploiting Variation To Find More Bugs

Micro-behaviors: Unreliable and distractible humans make each test a little bit
new each time through.
Randomness: Can protect you from unconscious bias.
Data Substitution: The same actions may have dramatically different results
when tried on a different database, or with different input.
Platform Substitution: Supposedly equivalent platforms may not be.
Timing/Concurrency Variations: The same actions may have different results
depending on the time frame in which they occur and other concurrent events.
Scenario Variation: The same functions may operate differently when
employed in a different flow or context.
State Pollution: Hidden variables of all kinds frequently exert influence in a
complex system. By varying the order, magnitude, and types of actions, we may
accelerate state pollution, and discover otherwise rare bugs.
Sensitivities and Expectations: Different testers may be sensitive to different
factors, or make different observations. The same tester may see different things at
different times or when intentionally shifting focus to different things.

6
5/19/2009

Testing is about searching an


infinite space in a finite time

To do that, we must use all


available information…

7
5/19/2009

…as it becomes available.


We must explore.

The alternative to artifact-based testing


is activity-based testing.

Manage testing as an activity consisting of manageable


activity units.
An activity is something a human tester does.
This can be formalized:
– Exploratory Testing
– Automatic coverage logging
– Session-Based Test Management
– General Functionality and Stability Test Procedure
Testing knowledge can be propagated formally and informally
through concise documentation and on-the-job training.

You might also like