Software Testing

What is testing?
Testing is the process of evaluating a system or its component(s) with the intent to find that
whether it satisfies the specified requirements or not. This activity results in the actual,
expected and difference between their results. In simple words testing is executing a system
in order to identify any gaps, errors or missing requirements in contrary to the actual desire or
requirements.

According to ANSI/IEEE 1059 standard, Testing can be defined as

A process of analyzing a software item to detect the differences between existing and
required conditions (that is defects/errors/bugs) and to evaluate the features of the software
item.

Who does testing?

It depends on the process and the associated stakeholders of the project(s). In the IT industry,
large companies have a team with responsibilities to evaluate the developed software in the
context of the given requirements. Moreover, developers also conduct testing which is called
Unit Testing. In most cases, following professionals are involved in testing of a system within
their respective capacities:

 Software Tester
 Software Developer
 Project Lead/Manager
 End User

Different companies have difference designations for people who test the software on the
basis of their experience and knowledge such as Software Tester, Software Quality
Assurance Engineer, and QA Analyst etc.

It is not possible to test the software at any time during its cycle. The next two sections state
when testing should be started and when to end it during the SDLC.

When to Start Testing?

An early start to testing reduces the cost, time to rework and error free software that is
delivered to the client. However in Software Development Life Cycle (SDLC) testing can be
started from the Requirements Gathering phase and lasts till the deployment of the software.
However it also depends on the development model that is being used. For example in Water
fall model formal testing is conducted in the Testing phase, but in incremental model, testing
is performed at the end of every increment/iteration and at the end the whole application is
tested.

1
Testing is done in different forms at every phase of SDLC like during Requirement gathering
phase, the analysis and verifications of requirements are also considered testing. Reviewing
the design in the design phase with intent to improve the design is also considered as testing.
Testing performed by a developer on completion of the code is also categorized as Unit type
of testing.

When to Stop Testing?

Unlike when to start testing it is difficult to determine when to stop testing, as testing is a
never ending process and no one can say that any software is 100% tested. Following are the
aspects which should be considered to stop the testing:

 Testing Deadlines.
 Completion of test case execution.
 Completion of Functional and code coverage to a certain point.
 Bug rate falls below a certain level and no high priority bugs are identified.
 Management decision.

Verification & Validation

These two terms are very confusing for people, who use them interchangeably. Let's discuss
about them briefly.

S.N. Verification Validation

1 Are you building it right? Are you building the right thing?
Ensure that the software system meets all the Ensure that functionalities meet the
2
functionality. intended behavior.
Validation occurs after verification and
Verification takes place first and includes the
3 mainly involves the checking of the
checking for documentation, code etc.
overall product.
4 Done by developers. Done by Testers.
Have static activities as it includes the reviews, Have dynamic activities as it includes
walkthroughs, and inspections to verify that executing the software against the
software is correct or not. requirements.
It is an objective process and no subjective It is a subjective process and involves
6 decision should be needed to verify the subjective decisions on how well the
Software. Software works.

International Standards

2
Many organizations around the globe are developing and implementing different Standards to
improve the quality needs of their Software. The next section briefly describes some of the
widely used standards related to Quality Assurance and Testing. Here is a definition of some
of them:

The ISO 9126-1 software quality model identifies 6 main quality characteristics, namely:

 Functionality
 Reliability
 Usability
 Efficiency
 Maintainability
 Portability

These characteristics are broken down into subcharacteristics, a high level table is shown
below. It is at the subcharacteristic level that measurement for SPI will occur. The main
characteristics of the ISO9126-1 quality model, can be defined as follows:-

Functionality
Functionality is the essential purpose of any product or service. For certain items this is
relatively easy to define, for example a ship's anchor has the function of holding a ship at a
given location. The more functions a product has, e.g. an ATM machine, then the more
complicated it becomes to define it's functionality. For software a list of functions can be
specified, i.e. a sales order processing systems should be able to record customer
information so that it can be used to reference a sales order. A sales order system should also
provide the following functions:

 Record sales order product, price and quantity.

 Calculate total price.
 Calculate appropriate sales tax.
 Calculate date available to ship, based on inventory.
 Generate purchase orders when stock falls below a given threshold.

The list goes on and on but the main point to note is that functionality is expressed as a
totality of essential functions that the software product provides. It is also important to note
that the presence or absence of these functions in a software product can be verified as either
existing or not, in that it is a Boolean (either a yes or no answer). The other software
characteristics listed (i.e. usability) are only present to some degree, i.e. not a simple on or
off. Many people get confused between overall process functionality (in which software
plays a part) and software functionality. This is partly due to the fact that Data Flow
Diagrams (DFDs) and other modeling tools can depict process functionality (as a set of data
in\data out conversions) and software functionality. Consider a sales order process, that has
both manual and software components. A function of the sales order process could be to
record the sales order but we could implement a hard copy filing cabinet for the actual orders
and only use software for calculating the price, tax and ship date. In this way the
functionality of the software is limited to those calculation functions. SPI, or Software
Process Improvement is different from overall Process Improvement or Process Re-

3
engineering, ISO 9126-1 and other software quality models do not help measure overall
Process costs\benefits but only the software component. The relationship between software
functionality within an overall business process is outside the scope of ISO 9126 and it is
only the software functionality, or essential purpose of the software component, that is of
interest for ISO 9126.

Following functionality, there are 5 other software attributes that characterize the
usefulness of the software in a given environment.
Each of the following characteristics can only be measured (and are assumed to exist) when
the functionality of a given system is present. In this way, for example, a system can not
possess usability characteristics if the system does not function correctly (the two just don't
go together).

Reliability
Once a software system is functioning, as specified, and delivered the reliability
characteristic defines the capability of the system to maintain its service provision under
defined conditions for defined periods of time. One aspect of this characteristic is fault
tolerance that is the ability of a system to withstand component failure. For example if the
network goes down for 20 seconds then comes back the system should be able to recover and
continue functioning.

Usability
Usability only exists with regard to functionality and refers to the ease of use for a given
function. For example a function of an ATM machine is to dispense cash as requested.
Placing common amounts on the screen for selection, i.e. $20.00, $40.00, $100.00 etc, does
not impact the function of the ATM but addresses the Usability of the function. The ability to
learn how to use a system (learnability) is also a major subcharacteristic of usability.

Efficiency
This characteristic is concerned with the system resources used when providing the required
functionality. The amount of disk space, memory, network etc. provides a good indication of
this characteristic. As with a number of these characteristics, there are overlaps. For example
the usability of a system is influenced by the system's Performance, in that if a system takes
3 hours to respond the system would not be easy to use although the essential issue is a
performance or efficiency characteristic.

Maintainability
The ability to identify and fix a fault within a software component is what the maintainability
characteristic addresses. In other software quality models this characteristic is referenced as
supportability. Maintainability is impacted by code readability or complexity as well as
modularization. Anything that helps with identifying the cause of a fault and then fixing the
fault is the concern of maintainability. Also the ability to verify (or test) a system, i.e.
testability, is one of the subcharacteristics of maintainability.

Portability
This characteristic refers to how well the software can adopt to changes in its environment or
with its requirements. The subcharacteristics of this characteristic include adaptability.

4
Object oriented design and implementation practices

The full table of Characteristics and Subcharacteristics for the ISO 9126-1 Quality Model is:-
Characteristics Subcharacteristics Definitions
This is the essential Functionality characteristic and
Suitability refers to the appropriateness (to specification) of the
functions of the software.
This refers to the correctness of the functions, an ATM
Accurateness may provide a cash dispensing function but is the
amount correct?
A given software component or system does not
typically function in isolation. This subcharacteristic
Functionality Interoperability
concerns the ability of a software component to
interact with other components or systems.
Where appropriate certain industry (or government)
laws and guidelines need to be complied with, i.e.
Compliance
SOX. This subcharacteristic addresses the compliant
capability of software.
This subcharacteristic relates to unauthorized access to
Security
the software functions.

This subcharacteristic concerns frequency of failure of

Maturity
the software.
The ability of software to withstand (and recover)
Reliability Fault tolerance
from component, or environmental, failure.
Ability to bring back a failed system to full operation,
Recoverability
including data and network connections.

Determines the ease of which the systems functions

Understandability can be understood, relates to user mental models in
Human Computer Interaction methods.
Learning effort for different users, i.e. novice, expert,
Learnability
Usability casual etc.
Ability of the software to be easily operated by a given
Operability
user in a given environment.

Characterizes response times for a given thru put, i.e.

Time behavior
Efficiency transaction rate.
Characterizes resources used, i.e. memory, cpu, disk
Resource behavior
and network usage.

5
 Characterizes the ability to identify the root cause of a
Analyzability
failure within the software.
Maintainability Changeability Characterizes the amount of effort to change a system.
Characterizes the sensitivity to change of a given
Stability system that is the negative impact that may be caused
by system changes.
Characterizes the effort needed to verify (test) a
Testability
system change.

Characterizes the ability of the system to change to

Adaptability
new specifications or operating environments.
Characterizes the effort required to install the
Portability Installability
software.
Similar to compliance for functionality, but this
characteristic relates to portability. One example
Conformance
would be Open SQL conformance which relates to
portability of database used.
Characterizes the plug and play aspect of software
Replaceability components, that is how easy is it to exchange a given
software component within a specified environment.

ISO/IEC 9126
This standard deals with the following aspects to determine the quality of a software
application:

 Quality model
 External metrics
 Internal metrics
 Quality in use metrics

This standard presents some set of quality attributes for any Software such as:

 Functionality
 Reliability
 Usability
 Efficiency
 Maintainability
 Portability

The above mentioned quality attributes are further divided into sub-factors which you can
study when you will go in detail of the standard.

6
Software Testing Methods

There are different methods which can be use for Software testing. This chapter briefly
describes those methods.

Black Box Testing

The technique of testing without having any knowledge of the interior workings of the
application is Black Box testing. The tester is oblivious to the system architecture and does
not have access to the source code. Typically, when performing a black box test, a tester will
interact with the system's user interface by providing inputs and examining outputs without
knowing how and where the inputs are worked upon.

Advantages Disadvantages
 Well suited and efficient for large code
 Limited Coverage since only a
segments.
selected number of test scenarios are
 Code Access not required.
actually performed.
 Clearly separates user's perspective
 Inefficient testing, due to the fact that
from the developer's perspective
the tester only has limited knowledge
through visibly defined roles.
about an application.
 Large numbers of moderately skilled
 Blind Coverage, since the tester cannot
testers can test the application with no
target specific code segments or error
knowledge of implementation,
prone areas.
programming language or operating
 The test cases are difficult to design.
systems.

White Box Testing

White box testing is the detailed investigation of internal logic and structure of the code.
White box testing is also called glass testing or open box testing. In order to perform white
box testing on an application, the tester needs to possess knowledge of the internal working
of the code.

The tester needs to have a look inside the source code and find out which unit/chunk of the
code is behaving inappropriately.

Advantages Disadvantages
 As the tester has knowledge of the  Due to the fact that a skilled tester is
source code, it becomes very easy to needed to perform white box testing,
find out which type of data can help in the costs are increased.
testing the application effectively.  Sometimes it is impossible to look into
 It helps in optimizing the code. every nook and corner to find out
 Extra lines of code can be removed hidden errors that may create problems
which can bring in hidden defects. as many paths will go untested.
 Due to the tester's knowledge about  It is difficult to maintain white box
the code, maximum coverage is testing as the use of specialized tools
attained during test scenario writing. like code analyzers and debugging

7
 tools are required.

Grey Box Testing

Grey Box testing is a technique to test the application with limited knowledge of the internal
workings of an application. In software testing, the term the more you know the better carries
a lot of weight when testing an application.

Mastering the domain of a system always gives the tester an edge over someone with limited
domain knowledge. Unlike black box testing, where the tester only tests the application's user
interface, in grey box testing, the tester has access to design documents and the database.
Having this knowledge, the tester is able to better prepare test data and test scenarios when
making the test plan.

Advantages Disadvantages
 Offers combined benefits of black box
and white box testing wherever
 Since the access to source code is not
possible.
available, the ability to go over the
 Grey box testers don't rely on the
code and test coverage is limited.
source code; instead they rely on
 The tests can be redundant if the
interface definition and functional
software designer has already run a
specifications.
test case.
 Based on the limited information
 Testing every possible input stream is
available, a grey box tester can design
unrealistic because it would take an
excellent test scenarios especially
unreasonable amount of time;
around communication protocols and
therefore, many program paths will go
data type handling.
untested.
 The test is done from the point of view
of the user and not the designer.

Black Box vs Grey Box vs White Box

S.N. Black Box Testing Grey Box Testing White Box Testing
The Internal Workings of an Tester has full knowledge of
Somewhat knowledge of the
1 application are not required the Internal workings of the
internal workings are known
to be known application
Another term for grey box
Also known as closed box testing is translucent testing Also known as clear box
2 testing, data driven testing as the tester has limited testing, structural testing or
and functional testing knowledge of the insides of code based testing
the application
Performed by end users and Performed by end users and
Normally done by testers
3 also by testers and also by testers and
and developers
developers developers
4 Testing is based on external Testing is done on the basis Internal workings are fully
expectations - Internal of high level database known and the tester can

8
 behavior of the application is diagrams and data flow
design test data accordingly
unknown diagrams
The most exhaustive and
This is the least time Partly time consuming and
5 time consuming type of
consuming and exhaustive exhaustive
testing
Not suited to algorithm Not suited to algorithm
6 Suited for algorithm testing
testing testing
Data domains and Internal Data domains and Internal
This can only be done by
7 boundaries can be tested, if boundaries can be better
trial and error method
known tested

Levels of Software Testing

There are different levels during the process of Testing. In this chapter a brief description is
provided about these levels.

Levels of testing include the different methodologies that can be used while conducting
Software Testing. Following are the main levels of Software Testing:

 Functional Testing.
 Non-Functional Testing.

Functional Testing

This is a type of black box testing that is based on the specifications of the software that is to
be tested. The application is tested by providing input and then the results are examined that
need to conform to the functionality it was intended for. Functional Testing of the software is
conducted on a complete, integrated system to evaluate the system's compliance with its
specified requirements.

There are five steps that are involved when testing an application for functionality.

Steps Description

I The determination of the functionality that the intended application is meant to perform.

II The creation of test data based on the specifications of the application.

III The output based on the test data and the specifications of the application.

IV The writing of Test Scenarios and the execution of test cases.

V The comparison of actual and expected results based on the executed test cases.

9
An effective testing practice will see the above steps applied to the testing policies of every
organization and hence it will make sure that the organization maintains the strictest of
standards when it comes to software quality.

Unit Testing
This type of testing is performed by the developers before the setup is handed over to the
testing team to formally execute the test cases. Unit testing is performed by the respective
developers on the individual units of source code assigned areas. The developers use test data
that is separate from the test data of the quality assurance team.

The goal of unit testing is to isolate each part of the program and show that individual parts
are correct in terms of requirements and functionality.

Limitations of Unit Testing

Testing cannot catch each and every bug in an application. It is impossible to evaluate every
execution path in every software application. The same is the case with unit testing.

There is a limit to the number of scenarios and test data that the developer can use to verify
the source code. So after he has exhausted all options there is no choice but to stop unit
testing and merge the code segment with other units.

Integration Testing
The testing of combined parts of an application to determine if they function correctly
together is Integration testing. There are two methods of doing Integration Testing Bottom-up
Integration testing and Top Down Integration testing.

S.N. Integration Testing Method

Bottom-up integration
1 This testing begins with unit testing, followed by tests of progressively higher-level
combinations of units called modules or builds.

Top-Down integration
2 This testing, the highest-level modules are tested first and progressively lower-level
modules are tested after that.

In a comprehensive software development environment, bottom-up testing is usually done

first, followed by top-down testing. The process concludes with multiple tests of the complete
application, preferably in scenarios designed to mimic those it will encounter in customers'
computers, systems and network.

System Testing

10
This is the next level in the testing and tests the system as a whole. Once all the components
are integrated, the application as a whole is tested rigorously to see that it meets Quality
Standards. This type of testing is performed by a specialized testing team.

System testing is so important because of the following reasons:

 System Testing is the first step in the Software Development Life Cycle, where the
application is tested as a whole.
 The application is tested thoroughly to verify that it meets the functional and technical
specifications.
 The application is tested in an environment which is very close to the production
environment where the application will be deployed.
 System Testing enables us to test, verify and validate both the business requirements
as well as the Applications Architecture.

Regression Testing
Whenever a change in a software application is made it is quite possible that other areas
within the application have been affected by this change. To verify that a fixed bug hasn't
resulted in another functionality or business rule violation is Regression testing. The intent of
Regression testing is to ensure that a change, such as a bug fix did not result in another fault
being uncovered in the application.

Regression testing is so important because of the following reasons:

 Minimize the gaps in testing when an application with changes made has to be tested.
 Testing the new changes to verify that the change made did not affect any other area
of the application.
 Mitigates Risks when regression testing is performed on the application.
 Test coverage is increased without compromising timelines.
 Increase speed to market the product.

Acceptance Testing
This is arguably the most importance type of testing as it is conducted by the Quality
Assurance Team who will gauge whether the application meets the intended specifications
and satisfies the client.s requirements. The QA team will have a set of pre written scenarios
and Test Cases that will be used to test the application.

More ideas will be shared about the application and more tests can be performed on it to
gauge its accuracy and the reasons why the project was initiated. Acceptance tests are not
only intended to point out simple spelling mistakes, cosmetic errors or Interface gaps, but
also to point out any bugs in the application that will result in system crashers or major errors
in the application.

By performing acceptance tests on an application the testing team will deduce how the
application will perform in production. There are also legal and contractual requirements for
acceptance of the system.

11
Alpha Testing

This test is the first stage of testing and will be performed amongst the teams (developer and
QA teams). Unit testing, integration testing and system testing when combined are known as
alpha testing. During this phase, the following will be tested in the application:

 Spelling Mistakes
 Broken Links
 Cloudy Directions
 The Application will be tested on machines with the lowest specification to test
loading times and any latency problems.

Beta Testing

This test is performed after Alpha testing has been successfully performed. In beta testing a
sample of the intended audience tests the application. Beta testing is also known as pre-
release testing. Beta test versions of software are ideally distributed to a wide audience on the
Web, partly to give the program a "real-world" test and partly to provide a preview of the
next release. In this phase the audience will be testing the following:

 Users will install, run the application and send their feedback to the project team.
 Typographical errors, confusing application flow, and even crashes.
 Getting the feedback, the project team can fix the problems before releasing the
software to the actual users.
 The more issues you fix that solve real user problems, the higher the quality of your
application will be.
 Having a higher-quality application when you release to the general public will
increase customer satisfaction.

Non-Functional Testing

This section is based upon the testing of the application from its non-functional attributes.
Non-functional testing of Software involves testing the Software from the requirements
which are non functional in nature related but important a well such as performance, security,
user interface etc.

Some of the important and commonly used non-functional testing types are mentioned as
follows:

Performance Testing
It is mostly used to identify any bottlenecks or performance issues rather than finding the
bugs in software. There are different causes which contribute in lowering the performance of
software:

 Network delay.
 Client side processing.
 Database transaction processing.

12
  Load balancing between servers.
 Data rendering.

Performance testing is considered as one of the important and mandatory testing type in terms
of following aspects:

 Speed (i.e. Response Time, data rendering and accessing)

 Capacity
 Stability
 Scalability

It can be either qualitative or quantitative testing activity and can be divided into different sub
types such as Load testing and Stress testing.

Load Testing

A process of testing the behavior of the Software by applying maximum load in terms of
Software accessing and manipulating large input data. It can be done at both normal and peak
load conditions. This type of testing identifies the maximum capacity of Software and its
behavior at peak time.

Most of the time, Load testing is performed with the help of automated tools such as Load
Runner, AppLoader, IBM Rational Performance Tester, Apache JMeter, Silk Performer,
Visual Studio Load Test etc.

Virtual users (VUsers) are defined in the automated testing tool and the script is executed to
verify the Load testing for the Software. The quantity of users can be increased or decreased
concurrently or incrementally based upon the requirements.

Stress Testing

This testing type includes the testing of Software behavior under abnormal conditions.
Taking away the resources, applying load beyond the actual load limit is Stress testing.

The main intent is to test the Software by applying the load to the system and taking over the
resources used by the Software to identify the breaking point. This testing can be performed
by testing different scenarios such as:

 Shutdown or restart of Network ports randomly.

 Turning the database on or off.
 Running different processes that consume resources such as CPU, Memory, server
etc.

Usability Testing
This section includes different concepts and definitions of Usability testing from Software
point of view. It is a black box technique and is used to identify any error(s) and
improvements in the Software by observing the users through their usage and operation.

13
According to Nielsen, Usability can be defined in terms of five factors i.e. Efficiency of use,
Learn-ability, Memor-ability, Errors/safety, satisfaction. According to him the usability of the
product will be good and the system is usable if it possesses the above factors.

Nigel Bevan and Macleod considered that Usability is the quality requirement which can be
measured as the outcome of interactions with a computer system. This requirement can be
fulfilled and the end user will be satisfied if the intended goals are achieved effectively with
the use of proper resources.

Molich in 2000 stated that user friendly system should fulfill the following five goals i.e.
Easy to Learn, Easy to Remember, Efficient to Use, Satisfactory to Use and Easy to
Understand.

In addition to different definitions of usability, there are some standards and quality models
and methods which define the usability in the form of attributes and sub attributes such as
ISO-9126, ISO-9241-11, ISO-13407 and IEEE std.610.12 etc.

UI vs Usability Testing

UI testing involves the testing of Graphical User Interface of the Software. This testing
ensures that the GUI should be according to requirements in terms of color, alignment, size
and other properties.

On the other hand Usability testing ensures that a good and user friendly GUI is designed and
is easy to use for the end user. UI testing can be considered as a sub part of Usability testing.

Security Testing
Security testing involves the testing of Software in order to identify any flaws ad gaps from
security and vulnerability point of view. Following are the main aspects which Security
testing should ensure:

 Confidentiality.
 Integrity.
 Authentication.
 Availability.
 Authorization.
 Non-repudiation.
 Software is secure against known and unknown vulnerabilities.
 Software data is secure.
 Software is according to all security regulations.
 Input checking and validation.
 SQL insertion attacks.
 Injection flaws.
 Session management issues.
 Cross-site scripting attacks.
 Buffer overflows vulnerabilities.
 Directory traversal attacks.

14
Portability Testing
Portability testing includes the testing of Software with intend that it should be re-useable and
can be moved from another Software as well. Following are the strategies that can be used for
Portability testing.

 Transferred installed Software from one computer to another.

 Building executable (.exe) to run the Software on different platforms.

Portability testing can be considered as one of the sub parts of System testing, as this testing
type includes the overall testing of Software with respect to its usage over different
environments. Computer Hardware, Operating Systems and Browsers are the major focus of
Portability testing. Following are some pre-conditions for Portability testing:

 Software should be designed and coded, keeping in mind Portability Requirements.

 Unit testing has been performed on the associated components.
 Integration testing has been performed.
 Test environment has been established.

15