Syed Mujahid Hussain Naqvi

TOPICS
A strategic approach to software testing
Unit Testing Integration Testing

Validation Testing
System Testing The ART of Debugging

Summary

Syed Mujahid Hussain Naqvi

STRATEGIC APPROACH TO SOFTWARE TESTING

Generic characteristics of software testing strategies: Testing begins at module level and works outward

towards the of integration entire computer based system. Different testing techniques are required at different points in time. Testing is conducted by the s/w developer and ITG ( Independent Test Group ) for large projects. Testing and Debugging are different and Debugging is essential in any testing strategy.
Syed Mujahid Hussain Naqvi

Verification and Validation

Verification -- Does the product meet its

specifications? Validation -- Does the product perform as desired?

Syed Mujahid Hussain Naqvi

V& V goals
Verification and validation should establish

confidence that the software is fit for purpose. This does NOT mean completely free of defects. Rather, it must be good enough for its intended use and the type of use will determine the degree of confidence that is needed.

V & V confidence
Depends on systems purpose, user expectations and

marketing environment
Software function The level of confidence depends on how critical the software is to an organisation. User expectations Users may have low expectations of certain kinds of software. Marketing environment Getting a product to market early may be more important than finding defects in the program.

Static and dynamic verification

Software inspections. Concerned with analysis of

the static system representation to discover problems (static verification)

May be supplement by tool-based document and code analysis

Software testing. Concerned with exercising and

observing product behaviour (dynamic verification)

The system is executed with test data and its operational

behaviour is observed

Software Testing Strategy

A Software Testing Strategy

Syed Mujahid Hussain Naqvi

Software Testing Strategy

Syed Mujahid Hussain Naqvi

STRATEGIC APPROACH
Issues to be addressed to develop a successful

software testing strategy:

Specify product requirements in a quantifiable

manner long before testing commences. State testing objectives explicitly. Understand the users of the software. Develop testing plan that emphasizes rapid cycle testing.

Syed Mujahid Hussain Naqvi

STRATEGIC APPROACH
Issues to be addressed to develop a successful

software testing strategy:

Build robust software that is designed to test itself. Use effective formal technical reviews as a filter to

testing. Conduct formal technical reviews to assess test strategy and test cases. Develop continuous improvement approach

Syed Mujahid Hussain Naqvi

UNIT TESTING
Unit testing -- focuses on the smallest element of

software design viz. the module.

Corresponds to class testing in the OO context.

Makes heavy use of white-box testing.

Syed Mujahid Hussain Naqvi

UNIT TESTING
Unit Test Generation Considerations:
Review Design information - develop unit test cases.
interface local data structures boundary conditions independent paths error handling paths

driver
Module to be tested stub stub
Syed Mujahid Hussain Naqvi

Test cases

Unit Test Generation

Interface considerations # of input parameters = # arguments? Parameter and argument attributes match? Parameter and argument units match? Order correct (if important)? Number and order of arguments for built-ins? References to parms not associated with entry point? Attempt to modify input-only arguments? Global variable definitions consistent? Constraints passed as arguments?

Syed Mujahid Hussain Naqvi

Unit Test Generation

External I/O considerations Files attributes correct? OPEN/CLOSE correct? Format specification matches I/O statement? Buffer size matches record size? Files opened before use? EOF handled correctly? I/O errors handled? Textual errors in output?

Syed Mujahid Hussain Naqvi

Unit Test Generation

Data structure considerations Improper or inconsistent typing? Erroneous initialization or default values? Incorrect variable names? Inconsistent data types? Underflow, overflow and addressing exceptions?

Syed Mujahid Hussain Naqvi

Unit Test Generation

Test cases must cover all execution paths Common computational errors to be checked: incorrect arithmetic mixed mode operations incorrect initialization precision inaccuracy incorrect symbolic representation of expression Other tests needed incompatible data types in comparisons incorrect logical operators or precedence comparison problems (e.g., == on floats) loop problems

Syed Mujahid Hussain Naqvi

Unit Test Generation

Error handling tests Exception-handling is incorrect? Error description is unintelligible, insufficient or incorrect? Error condition causes system interrupt before error handling completed?

Syed Mujahid Hussain Naqvi

INTEGRATION TESTING
A systematic approach for constructing program

structure while conducting tests to uncover errors associated with interfacing. Tendency for Non-Incremental integration.. Big Bang approach . Chaos !! ( usually ). Incremental integration - program is constructed and tested in small segments.
Top-Down Integration testing Bottom-Up Integration testing

Syed Mujahid Hussain Naqvi

INTEGRATION TESTING

Syed Mujahid Hussain Naqvi

INTEGRATION TESTING
Top-Down Approach Begin construction and testing with main module.
Stubs are substituted for all subordinate modules.

Subordinate stubs are replaced one at a time by actual

modules. Tests are conducted as each module is integrated. On completion of each set of tests, another stub is replaced with the real module. Regression testing may be conducted to ensure that new errors have not been introduced.

Syed Mujahid Hussain Naqvi

Top Down Approach - Use Stubs

Syed Mujahid Hussain Naqvi

Top-Down Approach :
Advantages:

INTEGRATION TESTING
Verifies major control or decision points early in the test process. With the use of depth-first integration testing, a complete function of the software can be demonstrated. -Confidence builder for developer/customer. Since stubs replace lower level modules, no significant data can flow upwards to the main module.

Disadvantages:

Syed Mujahid Hussain Naqvi

INTEGRATION TESTING
Bottom Up Approach :
This approach begins construction and testing with

modules at the lowest levels in the program structure.

Low-level modules are combined into clusters. A driver is written to coordinate test case input and output. The cluster is tested. Drivers are removed and clusters are combined moving upward in the program hierarchy.

Syed Mujahid Hussain Naqvi

Bottom Up Approach

Syed Mujahid Hussain Naqvi

Bottom Up Approach
Advantages:

INTEGRATION TESTING
Easier test case design and lack of stubs. The program as an entity is does not exist until the last module is added.

Disadvantages:

Sandwich Testing:- combined approach

Top down strategy for upper levels and Bottom up

strategy for subordinate levels.

Syed Mujahid Hussain Naqvi

 Regression Testing Re-execution of some subset of tests already conducted to ensure that the new changes do not have unintended side effects.

INTEGRATION TESTING

The Regression test suite should contain three

different classes of test cases :

A representative sample of tests that will exercise all

software functions Additional tests that focus on functions that are likely to be affected by the change. Tests that focus on software components that have changed.
Syed Mujahid Hussain Naqvi

INTEGRATION TESTING
1
Scope of testing

2
Test plan

3
Test Procedure n

4
Actual Test Results

5
Ref. & Appendix

Environment Test Schedule Unit Test / Resources phases test case and Overhead data Test builds software environment Order of Integration
Syed Mujahid Hussain Naqvi

Expected Results for build n

VALIDATION TESTING
It provides final assurance that software meets all

functional, behavioral, and performance requirements.

-- Exclusive use of Black-box testing techniques.

After each validation test case either software conforms to specs or a deviation from specs is detected and a deficiency list needs to be worked. Alpha and Beta testing
Alpha test -- At developers site by customer. Beta test -- At customers site in a live environment.
Syed Mujahid Hussain Naqvi

SYSTEM TESTING
A series of tests to verify that all system

elements have been properly integrated.

Recovery Testing: Forces software to fail in a variety of ways and verifies that recovery is properly performed. Security Testing: Attempts to verify the softwares protection mechanisms. The software designer tries to make penetration cost more than the value of information obtained by breaking in.

Syed Mujahid Hussain Naqvi

SYSTEM TESTING
Stress Testing: Executes the system in a manner that demands resources in abnormal quantity, frequency or volume. Performance Testing: To test the run time performance of a system within the context of an integrated system.

Syed Mujahid Hussain Naqvi

CLCS Test Approach

Operations Environment

User Acceptance Tests

System S/W Validation Tests

System Delivery

COTS H/W on Dock

System Test

User App S/W Validation Tests

Acceptance Test

User Eval CSCI Int Test

Integration Environment

Developers
Early Unit Integ User Eval Test

System Integration and Test Group

Validation Group Application S/W IPT

Unit Design Test

Syed Mujahid Hussain Naqvi

Development Environment

Users

THE ART OF DEBUGGING

Debugging is a consequence of successful testing --

when a test case uncovers an error, it is the debugging process that results in the removal of the error. Debugging is an ART. The external manifestation of the error and the cause of the error normally do not share an obvious relationships.

Syed Mujahid Hussain Naqvi

 Debugging Approaches Brute force : - Take memory dumps, invoke run time traces. Least efficient and quite common. Backtracking :- Once an error is uncovered, trace your way back to the cause of the error. Cause Elimination : - Isolate potential causes, devise cause hypotheses tests to isolate bug. Use of debugging tools

THE ART OF DEBUGGING

Syed Mujahid Hussain Naqvi

 Should the software developer be involved with

COMMENTS

testing ?
Developers have a vested interest in demonstrating

that their software is error-free. Developers (psychologically) feel that testing is destructive.

When are we done with testing ? You are never done with testing, the burden shifts from you to the customer.

Syed Mujahid Hussain Naqvi

 Software Testing accounts for the largest percentage of

SUMMARY

technical effort in the software process. Objective of Software testing -- To uncover errors, maintain software quality. Steps : Unit, Integration, Validation, System. Debugging is often an art and the most valuable resource is often the counsel of other software engineers.

Syed Mujahid Hussain Naqvi