Introduction to Software

Testing
(2nd edition)
Chapter 4

Putting Testing First

Paul Ammann & Jeff Offutt

http://www.cs.gmu.edu/~offutt/softwaretest/

August 2014
 Note
 This is the modified version of the original textbook slide
for SWE 202 course.

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 2

 The Increased Emphasis on Testing
 Philosophy of traditional software development methods
– Upfront analysis
– Extensive modeling
– Reveal problems as early as possible
More work must be revised

Root problem is harder to find

Cost
Delta

Time
Original Revision
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 3
 Traditional Assumptions
1. Modeling and analysis can identify potential problems
early in development
2. Savings implied by the cost-of-change curve justify the
cost of modeling and analysis over the life of the project
 These are true if requirements are always complete and
current
 But those annoying customers keep changing their minds!
– Humans are naturally good at approximating
– But pretty bad at perfecting
 These two assumptions have made software engineering
frustrating and difficult for decades
Thus, agile methods …
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 4
 Why Be Agile?
 Agile methods start by recognizing that neither
assumption is valid for many current software projects
 Response to first assumption
– Software engineers are not good at developing requirements
• We do not anticipate many changes
• Many of the changes we do anticipate are not needed
 Response to second assumption
– Requirements (and other “non-executable artifacts”) tend to go
out of date very quickly
• We seldom take time to update them
• Many current software projects change continuously

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 5

 Why Be Agile?
 Agile Philosophy: A flexible, iterative approach to software
development and project management that emphasizes
collaboration, customer feedback, and delivering small,
incremental improvements to meet changing needs.
 Agile approaches (frameworks and methodologies)
– Scrum
– Kanban
– Extreme Programming (XP)
– Lean
– Dynamic Systems Development Method (DSDM)
– Scaled Agile Framework (SAFe)
– Feature-Driven Development (FDD)
– Crystal
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 6
 Why Be Agile?
 Agile
– Extreme Programming (XP)
• Test Driven Development (TDD)
 How XP looks to the requirements and hence
documentation of software?
– Uses the models to understand the system, but don’t use them
as documentation.
– Recognizes the value of a UML diagram in communicating
information about a particular system design.
– Archiving the UML diagram is problematic. Discard the model
once its original purpose (design) has been served.

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 7

 Why Be Agile?
 The agile principle that goes directly to the heart of the
both assumptions:
– “You are not going to need it!”, or YAGNI.
 Traditional planning is not always true
– Predicting a system’s future needs is inherently difficult.
 Expected savings from the cost-of-change curve do not
materialize.
– Building features upfront based on speculation often wastes
effort.

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 8

 Why Be Agile?
 Agile approaches
– Defer many design and analysis decisions
– Focus on creating a running system that does “something” as
early as possible. (Start small)
– Evolve the software also by using testing over time. (Evolve over
time)
– Test, evolve, test, evolve, etc.

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 9

 Correctness
 Verifying correct behavior for a specific test case is more
focused and straightforward than ensuring correctness
across an entire system.
 Traditional approaches, seek correctness through
comprehensive definitions and analysis (a universal
perspective)
 Agile methods measure correctness through practical test
cases (a targeted, iterative approach).
 Agile
– Emphasizes building and validating software incrementally.
– Rlyes on tests to confirm behavior rather than aiming for
theoretical completeness upfront.

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 10

 Correctness
 In traditional methods, we try to define all correct
behavior completely, at the beginning.
– What is correctness?
– Does “correctness” mean anything in large engineering products?
– People are VERY BAD at completely defining correctness
 In agile methods, we redefine correctness to be relative to
a specific set of tests.
– If the software behaves correctly on the tests, it is “correct”
– Instead of defining all behaviors, we demonstrate some behaviors

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 11

 TDD and Test Harnesses
 Tests in TDD act as the de facto specification, replacing
traditional requirement documents.
– Tests define and verify system behavior through executable code.
 Automation is a prerequisite, enabling rapid feedback and
frequent runs.
 Each test includes a test oracle.
 Tests must be high quality, concise, and fast.
– Tests run with every code change.
 These automated tests are typically organized and
executed within a test harness.
A test harness is a framework or toolset that runs the tests,
manages their execution, and reports results.
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 12
 Continuous Integration
 Agile methods work best when the current version of the
software can be run against all tests at any time
A continuous integration server rebuilds the system,
returns, and reverifies tests whenever any update is
checked into the repository
 Mistakes are caught earlier
 Other developers are aware of changes early
 The rebuild and reverify must happen as soon as possible
– Thus, tests need to execute quickly

A continuous integration server doesn’t just run tests,

it decides if a modified system is still correct
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 13
 Acceptance Tests in Agile
– Acceptance tests in agile approaches are based on user stories.

A user story is a few sentences that captures what a user

will do with the software

Withdraw money from

checking account
Support technician sees
customer’s history on
demand
Agent sees a list of today’s
interview applicants

– In the language of the end user

– Usually small in scale with few details
– Not archived
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 14
 Acceptance Tests in Agile
Acceptance
User TDD
Test
Story Test 1
(Failing)

Change
Tests software &
Acceptance archived Refactor
Test
Continue adding
(Passing) TDD tests until TDD
acceptance test Test 2
passes Change
software & Refactoring avoids
Refactor maintenance debt
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 15
 Adding TDD to Existing Systems
 Most of today’s software is legacy
– No legacy tests
– Legacy requirements hopelessly outdated
– Designs, if they were ever written down, lost
 For most of these systems;
– The software must be changed.
– Tests must be applied.
 Two common needs for testing legacy systems:
– Sometimes we need to refactor legacy code,
– Sometimes we need to change the functionality of legacy code.

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 16

 Adding TDD to Existing Systems

 Refactoring: Modify the structure of existing code without

changing its behavior.
 When we refactor;
– We need to check that the behavior has not changed during
refactoring.
 Changing Functionality: Introduce new behavior or repair
a fault.
 When we apply these sort of changes we need tests.

How to apply TDD to legacy

software with no tests?
Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 17
 Adding Tests to Existing Systems
 Refactoring
– Create TDD tests for just the refactored code.
 Changing functionality
– Create TDD tests for the code where we intend to make
changes.
– Some of the tests fail;
• The new behavior is not yet implemented
• The fault is not yet repaired.
– The changes are made to the software.
– All the tests should pass, including those that failed earlier.

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 18

 The Testing Shortfall
 Do agile tests test the software well?
 Do the tests achieve good coverage on the code?
 Do the tests find most of the faults?
 If the software passes, should management feel confident
the software is reliable?
 TDD and Agile testing are powerful but they are not a
silver bullet.
– A combination of testing strategies, along with rigorous
processes, is needed to ensure software reliability.

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 19

 The Testing Shortfall
 Tests should cover all possible scenarios, including;
– Happy paths
– Confused-user paths
– Creative-user paths
– Malicious-user paths

Introduction to Software Testing, Edition 2 (Ch 4) © Ammann & Offutt 20

 Thank You.

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 21