Software Testing

Software Testing

Testing Definition : o Testing is the process of executing a program with intention of finding errors. IEEE Definition: 1. The process of operating a system or component under specified condition, observing or recording the results, and making an evaluation of some aspect of the system or component. 2. The process of analyzing a software item to detect the difference between the existing and required conditions (that is, bugs) and evaluate the features of the software item.

Software Test - Definition

Is a formal (Software Test Plan) process carried out by specialized testing team (independent) in which a software unit, several integrated software units or entire software package are examined by running the programs on a computer. All the associated tests are performed according to approved test procedures on approved test case.

Software Test Objectives

Direct Objectives: o To identify and reveal as many errors as possible in the tested software. o To bring the tested software, after correction of the identified errors and retesting, to an acceptable level of quality. o To perform the required tests efficiently, within budgetary and scheduling limitations. Indirect Objectives: o To compile a record of software errors for use in error prevention (by corrective & preventive actions)

Software Testing - Strategies

To test the software in its entirety, once the completed package is available; otherwise known as big bang testing. To test the software piecemeal, in modules, as they are completed (unit tests); then to test groups of tested modules integrated with newly completed modules (integration tests). This process continues until all the package modules have been tested. Once this phase is completed, the entire package is tested as a whole (system test). This testing strategy is usually termed incremental testing.usually termed incremental testing.

Big Bang Vs Incremental Testing

The main disadvantages of big bang : o Identification of errors becomes difficult o Corrections will be at the same time. o Error correction estimation of the required testing resources and testing schedule a rather fuzzy endeaver. Incremental testing advantage: o Usually performed on relatively small software modules, as unit or integration tests. (more errors detection) o Identification and correction of errors is much simpler and requires fewer resources because it is performed on a limited volume of software.

Integration Testing

Integration testing refers to the testing which software units of an application combined and tested for a communicationinterfaces between them. o Big Bang Testing o Bottom Up Testing o Top Down Testing

BIG BANG TESTING software components of an application are combinedall at once into a overall system accordingto this approach Advantage o To check the data flow from one module to another ommunication between various modules is C checked.

Bottom Up Vs Top Down

Advantage Of Bottom Up : o The relative ease of its performance Disadvantage o The lateness at which the prog. As whole can be observed(at the stage following testing of the last module)
Advantage Of Top Down: o The early stage at which it is possible to demonstrate the program as a whole, a condition that supports early identification of analysis and design errors. Disadvantage o The comparative difficulty of its performance.

Big Bang Testing

Module 1

Module 6

Module 2

System

Module 5

Module 3

Module 4

Software Testing Team

Software Testing Team Software testing can be done by software testers. Until the 1980s the term "software tester" was used generally, but later it was also seen as a separate profession. Regarding the periods and the different goals in software testing, different roles have been established: manager, test lead, test designer, tester, automation developer, and test administrator.

Classification According to Testing Concept

White Box Testing: o Examines internal calculation paths in order to identify bugs. o The white is meant to emphasize the contrast between methods o Testing that takes into account the internal mechanism of a system or component. Black Box Testing: o Identifies bugs only according to SW malfunctioning as they are revealed in its erroneous output. o Incases that outputs are found to be correct, black box testing disregarded the internal path of calculations and processing performed.

White Box Testing

White box testing concept requires verification of every program statement and comment. White box testing enables performance of o Data processing and calculations correctness tests o Software qualification tests o Maintainability tests o Reusability tests Every computational operation in the sequence of operations created by each test case ( Path ) must be examined. This type of verification allows us to decide whether the processing operations and their sequences were programmed correctly for the path in question, but not for other paths.

Advantages & Disadvantages of White Box Testing

Advantages o It permits direct checking of processing paths and algorithms. o It provides line coverage follow-up that delivers lists of lines of code that have not yet been executed. o It is capable of testing the quality of coding work. Disadvantages o It requires vast resources, much above those required for black box testing. o It cannot test the performance of software in terms of availability, reliability, stress, etc.

Black Box Testing

Black Box (Functionality) Testing o Identifies bugs only according to SW malfunctioning as they are revealed in its erroneous output. o Incases that outputs are found to be correct, black box testing disregarded the internal path of calculations and processing performed.

IEEE Definition:
o

Testing that ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions. Testing conducted to evaluate the compliance of a system or component with specified functional requirements.

Black Box Testing

Documentation tests o Common components of documentation Functional descriptions of the software system. Installation manual User manual Programmer manual o Document testing should include the following Document completeness check. Document correctness tests. Document style and editing inspection. Availability tests (reaction time) o The time needed to obtain the requested information o The time required for firmware to react. o More importance in on-line applications

Black Box Testing

Reliability tests o Events occurring over time, such as average time between failures average time for recovery after system failure average downtime per month It is better to carry it out once the system is completed. Problem o Testing process may need hundreds of hours o Comprehensive test case file must be constructed. Stress tests o Load tests Relates to the functional performance of the system under maximal operational load maximal transactions per minute There are usually realized for loads higher than those indicated in the requirements specification

Black Box Testing

Stress tests o Durability tests They are typically required for real-time firmware These test include o Firmware responses to climatic effects such as - extreme hot and cold temperatures, dust,etc, o Operation failures resulting from sudden electrical failures, voltage jumps, and sudden cutoffs in communications. Software system security tests o It aims at Preventing unauthorized access to the system or parts of it. Detection of unauthorized access and the activities performed by the penetration. Recovery of damages caused by unauthorized penetration cases.

Black Box Testing

Training usability tests o Used to estimate the resources needed to train a new employee How many hours of training are required for a new employee To achieve a defined level of acquaintance with the system To reach a defined hourly production rate. Operational usability tests o They focus on the operators productivity(quantitatively and qualitatively) Maintainability tests o Concerned by: System structure compliance to the standards and development procedures Programmers manual It is prepared according to approved documentation standards Internal documentation It is prepared according to coding procedures and conventions Covers the systems documentation requirements

Black Box Testing

Flexibility tests o The efforts required to adapt the software to Customer needs Additional changes to improve improving system functionality Testability tests o The ease of testing the software system o The addition of special features that help the testers in their work, such as Obtaining intermediate results Adding diagnostic tool for the analysis of the system and reporting any failure Portability tests o The possibility to use other operating systems, hardware or communication equipment standards to use the tested SW Interoperability test o Test the capabilities of interfacing with other HW or SW Receive inputs or send outputs

Advantages & Disadvantages of Black Box Testing

Advantages o It allows the tester to carry out almost all test classes. o For test classes that can be carried out by both white and black box testing, black box testing requires considerably fewer resources. Disadvantages o It allows for identification of coincidental errors as correct. o It lacks control of line coverage. o It lacks possibilities to test the quality of coding work.

Equivalence Class (EC) Partitioning

Equivalence Class (EC) Partitioning

Method that can help you derive test cases. You identify classes of input or output conditions. The rule is that each member in the class causes the same kind of behavior of the system. In other words, the "Equivalence Class Partitioning" method creates sets of inputs or outputs that are handled in the same way by the application. There are 2 major steps we need to do in order to use equivalence class partitioning: o Identify the equivalence classes of input or output. Take each input's or output's condition that is described in the specification and derive at least 2 classes for it: One class that satisfies the condition the valid class. Second class that doesn't satisfy the condition the invalid class. o Design test cases based on the equivalence classes.

Equivalence Class (EC) Partitioning

Example:

In a computer store, the computer item can have a quantity between -500 to +500. What are the equivalence classes?
Answer:

Valid class: -500 <= QTY <= +500 Invalid class: QTY > +500 Invalid class: QTY < -500

Testing Process

Determining the appropriate software quality standard Planning the tests Designing the tests Performing the tests (implementation)

Determining the appropriate software quality standard

Depends on the characteristics of the softwares application o Nature and magnitude of damages in case of system failure o The higher the expected level of damage, the higher standard for software quality is needed The Testing Strategy o The testing strategy A big bang or incremental testing strategy Bottom-up or top-down o Which parts to be performed using white box testing model o Which parts of the testing must be performed using automated testing

Planning the Tests

Unit tests o Deal with small units of software or modules Integration tests o Deal with several units that combine into a subsystem System tests o Refer to the entire software package/system. Commonly documented in a Software Test Plan (STP) Issues to consider before initiating a specific test plan:
o o o o o

What to test? Which sources to use for test cases? Who is to perform the tests? Where to perform the tests? When to terminate the tests?

What to test?

It is always preferred to perform a full and comprehensive

o o

This will ensure top quality software Requires the investment of vast resources Which modules should be unit tested Which integrations should be tested How to allocate available resources when performing the test (priorities )

So we must decide
o o o

Which sources to use for test cases?

Should we use o Synthetic test case o Real-life test cases Stratified Sampling Break down the random sample into sub-populations of test cases Reducing the proportion of the majority regular population tested Increasing the sampling proportion of small populations and high potential error populations Thus minimizes the number of repetitions

For each component of the testing plan we must decide wither o The use of a single or combined source of test cases, or both o How many test cases from each source are to be prepared o The characteristics of the test cases.

Who performs the tests?

Integration tests and unit tests o Generally performed by the SW dev. Team o In some instances it is the testing unit System tests o Usually performed by an independent testing team (internal or external) In large software systems o more than one testing team can be employed In small software development organizations o Another development team in the enterprise o Outsourcing of testing responsibilities

Where to perform the tests?

Usually is performed at the software developers site When the test is performed by external testing consultants o It is performed at consultants site. As a rule, the environment at the customers site differs from that at the developers site o Despite efforts to simulate that environment

When are tests terminated?

The completed implementation route The mathematical models application route The error seeding route The dual independent testing teams route

Termination after resources have petered out

Test Design

Composed of o Detailed design and procedures for each test The software test procedure document o Test case database/file The test case file document o In some cases the two document are integrated in one document called Software test description (STD)

Test Implementation

In general it consists of o A series of tests o Corrections of detected errors o Re-tests (regression tests) o Regression testing is done to Verify that the detected errors in the previous test runs have been properly corrected o No new errors have entered as a result of faulty corrections. It is advisable to re-test according to the original test procedure Usually only a portion of the original test procedure is re-tested to save testing resources This involve the risk of not detecting new errors produced when performing the correction

Classification of Software Failure Damages

A possible classification of failures of software products into five different types is as follows:

Transient. Transient failures occur only for certain input values while invoking a function of the system.

Permanent. Permanent failures occur for all input values while invoking a function of the system.
Recoverable. When recoverable failures occur, the system recovers with or without operator intervention.

Unrecoverable. In unrecoverable failures, the system may need to be restarted.

Cosmetic.These classes of failures cause only minor irritations, and do not lead to incorrect results. An example of a cosmetic failure is the case where the mouse button has to be clicked twice instead of once to invoke a given function through the graphical user interface.

Issues Affecting Software Risk Level

Module/application issues o Magnitude, Complexity o Difficulty Percentage of original software (vs. percentage of reused software) Programmer issues o Professional qualifications o Experience with the modules specific subject matter o Availability of professional support o Acquaintance with the programmer (the ability to evaluate their capabilities

Software Test Plan

A document detailing a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of what the eventual workflow will be. Documents the strategy that will be used to verify and ensure that a product or system meets its design specifications and other requirements. A test plan is usually prepared by or with significant input from Test Engineers.

Software Test Plan

Scope of the tests o The software package to be tested (name, version and revision) o The documents that provide the basis for the planned tests (name and version for each document) Testing environment o Testing sites o Required hardware and firmware configuration o Participating organizations o Manpower requirements o Preparation and training required of the test team Test details (for each test) o Test identification o Test objective o Cross-reference to the relevant design document and the requirement document o Test class o Test level (unit, integration or system tests) o Test case requirements o Special requirements (e.g., measurements of response times, security requirements) o Data to be recorded o Test schedule (for each test or test group) including time estimates for the following: o Preparation o Testing o Error correction o Regression tests

Software Test Description

Scope of the tests o The software package to be tested (name, version and revision) o The documents providing the basis for the designed tests (name and version for each document) Test environment (for each test) o Test identification (the test details are documented in the STP) o Detailed description of the operating system and hardware configuration and the required switch settings for the tests o Instructions for software loading Testing process o Instructions for input, detailing every step of the input process o Data to be recorded during the tests Test cases (for each case) o Test case identification details o Input data and system settings o Expected intermediate results (if applicable) o Expected results (numerical, message, activation of equipment, etc.) Actions to be taken in case of program failure/cessation Procedures to be applied according to the test results summary

Implementation Phase Activities

The testing implementation phase activities consist of a series of tests, corrections of detected errors and re-tests (regression tests). Testing is culminated when the re-test results satisfy the developers. The tests are carried out by running the test cases according to the test procedures. Documentation of the test procedures and the test case database/ file comprises the software test description (STD). Re-testing (also termed regression testing) is conducted to verify that the errors detected in the previous test runs have been properly corrected, and that no new errors have entered as a result of faulty corrections. It is consist of a series of tests, corrections of detected errors and retests (regression tests). Testing is culminated when the re-test results satisfy the developers.

Implementation Phase Activities

Software Test Report

Software Test Report (STR) is a record of the qualification testing performed on a Computer Software Configuration Item (CSCI), a software system or subsystem, or other software-related item. The STR enables the acquirer to assess the testing and its results.

Software Test Report

Test identification, site, schedule and participation o The tested software identification (name, version and revision) o The documents providing the basis for the tests (name and version for each document) o Test site o Initiation and concluding times for each testing session o Test team members o Other participants o Hours invested in performing the tests Test environment o Hardware and firmware configurations o Preparations and training prior to testing Test results o Test identification o Test case results (for each test case individually) Test case identification Tester identification Results: OK / failed If failed: detailed description of the results/problems

Software Test Report

Summary tables for total number of errors, their distribution and types o Summary of current tests o Comparison with previous results (for regression test summaries) Special events and testers proposals o Special events and unpredicted responses of the software during testing o Problems encountered during testing o Proposals for changes in the test environment, including test preparations o Proposals for changes or corrections in test procedures and test case files

Test Case

TEST CASE

Set of procedures written by a tester whichexecute in our system to find defect. o Positive test case o Negative test case A test case is said to be effective only when both positive and negative cases are prepared.

Automated testing

This process includes o Test planning o Test design o Test case preparation o Test performance o Test log and report preparation o Re-testing after correction of detected errors (regression tests) o Final test log and report preparation including comparison reports Types of automated tests o Code auditing (Qualification testing) o Does the code fulfill code structure/style Module size, Levels of (loop nesting, subroutine nesting) Prohibited constructs, such as GOTO. Naming conventions for variables, files, etc. o Do the internal program documentation follows the coding style procedures? Location of comments in the file Help index and presentation style

Automated testing

Types of automated tests o Functional tests Replace manual black-box correctness tests These tests can be executed with minimal effort or professional resources. o Coverage monitoring Produce reports about the line coverage o Load tests If it to be performed manually, in most cases it is impractical and impossible in others The solution is to use computerized simulations In general it is combined with availability and efficiency tests In this test the load is gradually increased to the specified maximal load and beyond o Test management It monitors performance of every item on long lists of test case files provide testers with reports timetable follow-up

Advantages & Disadvantages of Automated Testing

Advantages o Accuracy and completeness of performance o Accuracy of results log and summary reports o Comprehensiveness of information o Few manpower resources required to perform tests o Shorter duration of testing o Performance of complete regression tests o Performance of test classes beyond the scope of manual testing. Disadvantages o High investments required in package purchasing and training. o High package development investment costs. o High manpower requirements for test preparation. o Considerable testing areas left uncovered

Alpha & Beta Site Tests

Alpha site tests - are tests of a new software package that are performed at the developers site by the customer The identified errors are expected to include the errors that only a real user can reveal, and thus should be reported to the developer

Beta site tests - are much more commonly applied than are alpha site tests. It is applied on an advanced version of the software package The developer offers it free of charge SW to one or more potential users in order to test them.

Alpha & Beta Site Tests

Advantages o Identification of unexpected errors o A wider population in search of errors o Low costs Disadvantages o A lack of systematic testing o Low quality error reports o Difficult to reproduce the test environment o Much effort is required to examine reports