Software Testing Steps

Ensuring a Successful Software Test Strategy

• Specify product requirements in a quantifiable manner long before testing
commences
• State testing objectives explicitly in measurable terms
• Understand the user of the software (through use cases) and develop a
profile for each user category
• Develop a testing plan that emphasizes rapid cycle testing to get quick
feedback to control quality levels and adjust the test strategy
• Build robust software that is designed to test itself and can diagnose certain
kinds of errors
• Use effective formal technical reviews as a filter prior to testing to reduce the
amount of testing required
• Conduct formal technical reviews to assess the test strategy and test cases
themselves
• Develop a continuous improvement approach for the testing process through
the gathering of metrics
Test Strategies for Conventional Software
 Unit Testing
• Focuses testing on the function or software module
• Concentrates on the internal processing logic and data structures
• Is simplified when a module is designed with high cohesion
– Reduces the number of test cases
– Allows errors to be more easily predicted and uncovered
• Concentrates on critical modules.

‘Test Case is a set of actions executed to verify particular functionality’

Tester compares expected result with actual results to determine wheather

the software is functioning as per the customer requirements.
Targets for Unit Test Cases
Common Computational Errors in Execution
Paths
• Misunderstood or incorrect arithmetic
precedence
• Mixed mode operations (e.g., int, float, char)
• Incorrect initialization of values
• Precision inaccuracy and round-off errors
• Incorrect symbolic representation of an
expression (int vs. float)
 Other Errors to Uncover
• Comparison of different data types
• Incorrect logical operators or precedence
• Expectation of equality when precision error
makes equality unlikely (using == with float types)
• Incorrect comparison of variables
• Improper or nonexistent loop termination
• Failure to exit when divergent iteration is
encountered
• Improperly modified loop variables
• Boundary value violations
 Problems to uncover in
Error Handling
• Error description is unintelligible or ambiguous
• Error noted does not correspond to error
encountered
• Error condition causes operating system
intervention prior to error handling
• Exception condition processing is incorrect
• Error description does not provide enough
information to assist in the location of the
cause of the error
 Drivers and Stubs for
Unit Testing
• Driver
– A simple main program that accepts test case data, passes such data to the
component being tested, and prints the returned results
• Stubs(Dummy program)
– Serve to replace modules that are subordinate to (called by) the component to
be tested
– It uses the module’s exact interface, may do minimal data manipulation,
provides verification of entry, and returns control to the module undergoing
testing
• Drivers and stubs both represent overhead
– Both must be written but don’t constitute part of the installed software
product
Example: Module A-login
Module B-Homepage
Module C-Logout
 Integration Testing
• Defined as a systematic technique for constructing the software
architecture
– At the same time integration is occurring, conduct tests to uncover errors
associated with interfaces
• Objective is to take unit tested modules and build a program structure
based on the prescribed design
• Two Approaches
– Non-incremental Integration Testing
– Incremental Integration Testing
 Non-incremental
Integration Testing
• Commonly called the “Big Bang” approach
• All components are combined in advance
• The entire program is tested as a whole
• Chaos results
• Many seemingly-unrelated errors are encountered
• Correction is difficult because isolation of causes is
complicated
• Once a set of errors are corrected, more errors
occur, and testing appears to enter an endless loop
 Incremental Integration Testing

• The program is constructed and tested in small increments

• Errors are easier to isolate and correct
• Interfaces are more likely to be tested completely
• A systematic test approach is applied

• Two kinds
– Top-down integration
– Bottom-up integration
 Top-down Integration
• Modules are integrated by moving downward through the control
hierarchy, beginning with the main module
• Subordinate modules are incorporated in either a depth-first or breadth-
first fashion
– DF: All modules on a major control path are integrated
– BF: All modules directly subordinate at each level are integrated

• Subordinates can be replaced one at a time

• Advantages
– This approach verifies major control or decision points early in the test process
• Disadvantages
– Stubs need to be created to substitute for modules that have not been built or
tested yet; this code is later discarded
– Because stubs are used to replace lower level modules, no significant data
flow can occur until much later in the integration/testing process
DF:M1,M2,M5,M8/M6
BF:LEVEL ORDER{M2,M3,M4}
 Bottom-up Integration
• Integration and testing starts with the most atomic modules(lowest level) in
the control hierarchy
• As Components are integrated from bottom-up need for stubs can be
eliminated.
• Low level components are combined into clusters(builds) which performs a
specific subfunction.
• Cluster is tested using drivers,later clusters are interfaced directly with
modules.
• Advantages
– This approach verifies low-level data processing early in the testing process
– Need for stubs is eliminated
• Disadvantages
– Driver modules need to be built to test the lower-level modules; this code is later
discarded or expanded into a full-featured version
– Drivers inherently do not contain the complete algorithms that will eventually use
the services of the lower-level modules; consequently, testing may be incomplete
or more testing may be needed later when the upper level modules are available
 Regression Testing
• Each new addition or change to baselined software may cause problems
with functions that previously worked flawlessly
• Regression testing re-executes a small subset of tests that have already
been conducted
– Ensures that changes have not propagated unintended side effects
– Helps to ensure that changes do not introduce unintended behavior or
additional errors
– May be done manually or through the use of automated capture/playback
tools
• Regression test suite contains three different classes of test cases
– A representative sample of tests that will exercise all software functions
– Additional tests that focus on software functions that are likely to be affected
by the change
– Tests that focus on the actual software components that have been changed
 Smoke Testing
• It is integration testing approach used when product software is
developed
• Designed as a pacing mechanism for time-critical projects
– Allows the software team to assess its project on a frequent basis
• Includes the following activities
– The software is compiled and linked into a build
– A series of tests is designed to expose errors that will keep the build from
properly performing its function
• The goal is to uncover “show stopper” errors that have the highest likelihood of
throwing the software project behind schedule
– The build is integrated with other builds and the entire product is smoke
tested daily
• Daily testing gives managers and practitioners a realistic assessment of the progress
of the integration testing
– The integration may be top down or bottom up
– It is a mini and quick regression test of major functionality
– It is to verify the stability of application
 Benefits of Smoke Testing
• Integration risk is minimized
– Daily testing uncovers incompatibilities and show-stoppers early in the testing
process, thereby reducing schedule impact
• The quality of the end-product is improved
– Smoke testing is likely to uncover both functional errors and architectural and
component-level design errors
• Error diagnosis and correction are simplified
– Smoke testing will probably uncover errors in the newest components that
were integrated
• Progress is easier to assess
– As integration testing progresses, more software has been integrated and
more has been demonstrated to work
– Managers get a good indication that progress is being made
 Test Strategies for
Object-Oriented Software
• With object-oriented software, you can no longer test a single operation in
isolation (conventional thinking)
• Traditional top-down or bottom-up integration testing has little meaning
• Class testing for object-oriented software is the equivalent of unit testing
for conventional software
– Focuses on operations encapsulated by the class and the state behavior of the
class
• Drivers can be used
– To test operations at the lowest level and for testing whole groups of classes
– To replace the user interface so that tests of system functionality can be
conducted prior to implementation of the actual interface
• Stubs can be used
– In situations in which collaboration between classes is required but one or
more of the collaborating classes has not yet been fully implemented
 Test Strategies for Object-Oriented Software
(continued)
• Integration testing in OO context
• Two different object-oriented testing strategies
– Thread-based testing
• Integrates the set of classes required to respond to one input or event for the
system
• Each thread (key functional capabilities of specific task)is integrated and tested
individually
• Regression testing is applied to ensure that no side effects occur
– Use-based testing
• First tests the independent classes that use very few server classes.
• Then the next layer of classes, called dependent classes that use independent
classes are tested.
• This sequence of testing layer of dependent classes continues until the entire
system is constructed
 Validation Testing
• Validation testing follows integration testing
• The distinction between conventional and object-oriented software disappears
• Focuses on user-visible actions and user-recognizable output from the system
• Demonstrates conformity with requirements
• Designed to ensure that
– All functional requirements are satisfied
– All behavioral characteristics are achieved
– All performance requirements are attained
– Documentation is correct
– Usability and other requirements are met (e.g., transportability, compatibility, error
recovery, maintainability)
• After each validation test
– The function or performance characteristic conforms to specification and is accepted
– A deviation from specification is uncovered and a deficiency list is created
• A configuration review or audit ensures that all elements of the software
configuration have been properly developed, cataloged, and have the necessary
detail for entering the support phase of the software life cycle
 Alpha and Beta Testing
• Alpha testing
– Conducted at the developer’s site by end users
– Software is used in a natural setting with developers watching intently
– Testing is conducted in a controlled environment
• Beta testing
– Conducted at end-user sites
– Developer is generally not present
– It serves as a live application of the software in an environment that cannot be
controlled by the developer
– The end-user records all problems that are encountered and reports these to
the developers at regular intervals
• After beta testing is complete, software engineers make software
modifications and prepare for release of the software product to the
entire customer base
• A variation on beta testing is customer acceptance testing
 System Testing
• It is a series of different tests whose primary
purpose is to fully exercise the computer
based system.
 Different Types
• Recovery testing
– Tests for recovery from system faults
– Forces the software to fail in a variety of ways and verifies that recovery is properly
performed
– Tests reinitialization, check pointing mechanisms, data recovery, and restart for
correctness
• Security testing
– Verifies that protection mechanisms built into a system will, in fact, protect it from
improper access.
– Tester plays the role of penetrator.
• Stress testing
– Executes a system in a manner that demands resources in abnormal quantity,
frequency, or volume
• Performance testing
– Tests the run-time performance of software within the context of an integrated system
– Often coupled with stress testing and usually requires both hardware and software
instrumentation(resource utilization)ex:processor cycles
– Can uncover situations that lead to degradation and possible system failure
 Different Types(cont.)
• Deployment testing
-Also called configuration testing
-Excercises the software in various environments/platforms.
Ex: Safehome web app tested using all web browsers
The Art of Debugging
 Debugging process
• Debugging occurs as a consequence of successful testing
• Debugging is the process that results in removal of the errors.
• Good debugging ability may be an innate human trait
• Debugging should be an orderly process.
• The debugging process begins with the execution of a test case
• Results are assessed and the difference between expected and
actual performance is encountered
• This difference is a symptom of an underlying cause that lies
hidden
• The debugging process attempts to match symptom with cause,
thereby leading to error correction
Debugging process
 Why is Debugging so Difficult?
• The symptom and the cause may be geographically remote
• The symptom may disappear (temporarily) when another
error is corrected
• The symptom may actually be caused by nonerrors (e.g.,
round-off accuracies)
• The symptom may be caused by human error that is not
easily traced
• The symptom may be intermittent such as in embedded
systems involving both hardware and software
• The symptom may be due to causes that are distributed
across a number of tasks running on different processes
 Debugging Strategies
• Objective of debugging is to find and correct the cause of a software error
• Bugs are found by a combination of systematic evaluation, intuition, and
luck
• Debugging methods and tools are not a substitute for careful evaluation
based on a complete design model and clear source code
• There are three main debugging strategies
– Brute force
– Backtracking
– Cause elimination
 Strategy #1: Brute Force
• Most commonly used and least efficient method
• Used when all else fails
• Involves the use of memory dumps,run-time traces, and
output statements
Ex:print intermediate values and check various values of
variables
• Leads many times to wasted effort and time
 Strategy #2: Backtracking
• Can be used successfully in small programs
• The method starts at the location where a symptom has been
uncovered
• The source code is then traced backward (manually) until the
location of the cause
(error) is found
• In large programs, the number of potential backward paths
may become unmanageably large
 Strategy #3: Cause Elimination
• Data related to the error occurrence are organized to isolate potential
causes
• A cause hypothesis is devised, and the aforementioned data are used to
prove or disprove the hypothesis
• A list of all possible causes is developed, and tests are conducted to
eliminate each cause
• If initial tests indicate that a particular cause hypothesis shows promise,
data are refined in an attempt to isolate the bug