STUDY IN ENGLISH

STUDY IN ENGLISH

Software testing -
Introduction
Miodrag Zivkovic

www.study.singidunum.ac.rs
STUDY IN ENGLISH

Introduction
• Software testing is an extremely important part of
software development
• There is a QA (Quality Assurance) team in every SW
Dev company consisting of SW testers
• QA or SW tester is in demand and an important job
on the market
• Different certification schemes are available for
software testing, and the most popular is ISTQB
STUDY IN ENGLISH

Introduction

• What is industrial (commercial) software?

• What are the differences compared to “school”
software?
• How do we define school software?
STUDY IN ENGLISH

School software

• Mostly developed for demonstrations during

lectures and training, exams, or as a hobby etc.
• Usually it does not solve a real life problem
• Presence of errors (although not recommended) is
not critical – the author fixes them (if found)
• Documentation is not important
STUDY IN ENGLISH

Industrial software

• Developed with a clearly set goal

• It solves concrete user problems
• It is of utmost importance that the program
executes as expected
• Errors in program execution can cause user
dissatisfaction, financial loss, important data loss,
even human casualties
STUDY IN ENGLISH

Industrial software

• Key factor in large number of devices and systems

in modern society
• Software defines operation of the following:
• Network routers
• Financial systems
• Telephony
• Web
STUDY IN ENGLISH

Industrial software

• Software is also a key component in embedded

applications which control operations of different
complex systems:
• Aeroplanes
• Spacecrafts
• Air Traffic Control
STUDY IN ENGLISH

Industrial software

• It can also be found in less complex devices, such

as:
• Watches
• Mobile phones
• Cars
• Microwaves
• DVD players
STUDY IN ENGLISH

Industrial software

• It is assumed that there are over 50 processors in a

modern home
• Modern car has several hundred processors
• All processors execute software
• Optimistic users expect that the software shall
always execute flawlessly
STUDY IN ENGLISH

Famous software failures

• AT&T – long distance calls failure (1990)

• In January 1990, users of the AT&T were not able to
make long distance calls for several hours
• The problem was in a software which controlled
relay switches. It was updated prior to failure in
service
• AT&T loss for that day: 60 million $ (expenses,
penalties etc.)
STUDY IN ENGLISH

Famous software failures

• Therac 25 (1985) – computer controlled device for

radiotherapy
• 6 people overdosed. Consequently 5 of them died
• Cause – a missing software security lock which
prevents overdosing
STUDY IN ENGLISH

Famous software failures

• NASA – Mars Climate Orbiter (1998)

• On its mission to Mars, the spacecraft got lost in space
• After a thorough investigation, it was discovered that there was an error in
conversion of imperial units to metric system
• Different parts of development team used different units
• One group of engineers working on jet engines calculated thrust in pounds,
other group used metric Newton in calculations: jet engines had 4.45 times
more thrust than expected
• Result – 125 million $ craft came to close to Mars’ surface after an attempt
to stabilize the orbit too low, and its communication devices got broken
• Now it is a useless and quite expensive chunk of metal orbiting the Sun
STUDY IN ENGLISH

Famous software failures

• Ariane 5 flight 501, European Space Agency, 1996

• An aircraft worth 370 million $ fell apart 40 seconds
into the launch
• Problem – overflow in conversion float -> int
• Inertial referent system was operating 64-bit float data,
and converting them to 16-bit integer, causing
arithmetic overflow
• Around 8 billion $ was spent during the development of
Ariane 5, and in the moment of take off it was carrying
more than 500 million $ worth of satellites
STUDY IN ENGLISH

Famous software failures

STUDY IN ENGLISH

Famous software failures

• EDS Child Support system, UK (2004)
• An extremely complex IT system was introduced in British Child
Support Agency. At the same time, Department for Work and
Pensions decided to restructure the agency in its entirety
• These two parts of software were completely incompatible,
leading to irreversible errors
• The system managed to overpay 1.9 million people, underpay
another 700 000, there was 7 billion $ of uncollected child
support payments, and 36 000 cases stuck in the system
• Up to date it has costed UK tax payers several billion $
STUDY IN ENGLISH

Famous software failures

• Almost led to WW3 – nuclear war (1983)

• Cost would be almost complete humanity
• Soviet early warning system falsely indicated the US had launched
5 ballistic missiles
• Fortunately, Soviet officer on duty reasoned based on the pure
logic that if the US was really attacking, they would launch much
more than just 5 missiles and decided to report the apparent
attack as a false alarm instead of initiating Soviet response
• A bug in the Soviet software failed to filter out false missile
detections due to sunlight reflecting off cloud-tops
STUDY IN ENGLISH

Industrial software

• There is a clear need for quality softwares

• In real development, up to 50% of time (cost) is
spent on software testing
• There are clear demands for planned development
– phases, documentation, compliance to different
standards
• It is required to fulfill nonfunctional requirements
as well, such as security and performance
STUDY IN ENGLISH

Industrial software

• The size of industrial software can vary quite a lot

• It is measured by the number of lines of code
• Small projects have up to 10k lines of code
• Very large projects have several million lines of
code
STUDY IN ENGLISH

Industrial software
STUDY IN ENGLISH

Industrial software
STUDY IN ENGLISH

Industrial software
STUDY IN ENGLISH

Industrial software
STUDY IN ENGLISH

Industrial software
STUDY IN ENGLISH

Industrial software

• The importance of testing:

• American National Institute of Standards and
Technology (NIST) estimates that software bugs caused
60 billion $ loss in 2002 in the US economy only
• Around 70% equipment failure can be assigned to
software (usually the weak point)
• Software does not age – if the errors appear, it means
that they have been present in the software since the
beginning, and can arise after a long period of normal
operation
STUDY IN ENGLISH

Terminology

• Glossary of Software Engineering Terminology

ANSI/IEEE Std. 729–1983
• Error
• Human error, during requirement specification or coding the
program
• Fault (defect)
• The consequence of error – program is missing something, or
the function is there but it is not working as expected - bug
• Failure
• System is not able to perform requested function
• It appears when defect code is activated (executed)
STUDY IN ENGLISH

Terminology
• Bug?

• Moth in relay of the computer tested at Harvard,

1945, attached to the log
STUDY IN ENGLISH

Terminology

• Incident
• A symptom which user observes and consequently becomes aware of
the fact that there is a failure in the system
• Testing
• The process of executing software with test cases. There are two
different goals of testing: either to find failures, or to demonstrate
correct execution
• Failures can be discovered through testing but afterwards the
elimination of defects is needed – to debug the program, which is a
separate activity
• Test case
• It has identity, and it tests the defined behavior of the program. It has a
set of input variables and a set of expected results
STUDY IN ENGLISH

Analogy

• A doctor who needs to diagnose a patient

• The patient comes to the doctor’s office with a list of
observed symptoms based on which he is aware of the fact
that there are failures in the body
• The doctor needs to find fault (defect), the cause of the
symptom
• In order to make diagnostics easier, the doctor can order
tests which might show an anomaly – high blood pressure,
high glucose levels, cholesterol etc.
• Those anomalies correspond to errors
STUDY IN ENGLISH

Analogy

• Nevertheless, there is one key point which portrays

how visiting a doctor and software testing differ
• Failures in software are errors in design and
program code
• These errors do not occur spontaneously, but as a
result of a human error
• On the other hand, medical problems, similar to
failures in system hardware, are often a result of
physical degradation over time
STUDY IN ENGLISH

Analogy

• This difference is important because it shows the limit up to

which it is (not) possible to control and predict software errors
(the limit of human stupidity)
• In other words, there is not possible to discover all random errors
humans can make
• It is not possible to eliminate all possible errors in non trivial
software
• Additionally, software user is a human whose random behavior
can cause unexpected operation of software (for example, we
cannot exclude that the user would not put his fingers in the
electric socket )
STUDY IN ENGLISH

Software testing definition

• The process of executing the program with the

purpose of finding errors
• Generally speaking, it is an examination of a
software product or service with the goal of
objective quality estimation
• It is a part of the wide area of Quality Assurance
which controls the development process with the
aim of providing quality final product within
predictable time frame
STUDY IN ENGLISH

Example
STUDY IN ENGLISH

Example

• Defect in the program – it starts to search zeros from

index 1 instead of 0 in array
• In Java programming language, array index starts at 0
• Call to method numZero ([2, 7, 0]) returns correct result 1
• Call numZero ([0, 7, 2]) returns wrong - 0
• In both method calls, defect code is executed
• Although error exists in both calls only the second call
results in failure
STUDY IN ENGLISH

Example

• To understand the state of the error we have to look at the state

of the program
• The state of numZero method consists of values of variable x,
counter count, counter i and program counter PC
• In the first example, the state of the program in the first iteration
of loop in if statement is: ( x = [2, 7, 0], count = 0, i = 1, PC = if)
• Bare in mind that this is the state of error, because i should have
value 0 in the first iteration
• Coincidently, the count value is correct, so the error state is not
propagated to the end. Therefore, there is no software failure in
this case
STUDY IN ENGLISH

Test cases

• Test case can be observed as a triple [Input, State,

Output]
• Input describes input data
• State represents the state of the system at the moment
of entering input data
• Output is expected output of the system
• Test set is the set of all test cases
STUDY IN ENGLISH

Test case design

• Test cases are not selected randomly – it is necessary that they

are carefully designed
• Why don’t just test all combinations of input parameters
(exhaustive testing)?
• Almost every non trivial system has extremely large domain of
input data
• Exhaustive testing for every possible combination of input data is
either impossible or impractical
• Randomly selected test can be without importance if it exposes
error which is already detected with another test case
STUDY IN ENGLISH

Test case design

• The number of test cases is not a guarantee for successful testing

• Observe the next sample code:
if(x>y) max = x; else max = x;
• Two tests, with input data {(x=3, y=2); (x=2, y=3)} are fulfilling the
criteria
• On the other hand, three tests, with input data {(x=3, y=2); (x=4,
y=3); (x=5, y = 1)} are not appropriate, as they are not exposing
the error
• Every test should expose a different error (the last two tests in
the second example are redundant, as they are targeting the
same error)
STUDY IN ENGLISH

Expected test results

• Predicting test results (test oracle)

STUDY IN ENGLISH

Expected test results

• Estimating test results is a mechanism, which is entirely

independent of software under test, which is used for validating
software’s correct behavior
• In most cases it is a human or a machine
• The most frequent software specification is used to determine
what is the correct behavior of program. Problem – specifications
are also written by people, and they are often erroneous,
incomplete or ambiguous
• Automated predictors use formal specifications, and they are
correct as much as the specification is correct. Formal
specifications usually don’t exist for the stated program, and
creating them is not easy
STUDY IN ENGLISH

Testing classification

• Two basic classifications

• Approach:
• Functional (based on specification)
• Structural (based on code)

• Level:
• Unit
• Integration
• System
STUDY IN ENGLISH

Structural testing

• Structural testing is focused on program

implementation
• Also known as white box techniques
• The goal is to execute and activate all program
structures and data structures in the code
• Does not check the specification – it is not possible
to detect if some parts of specification are not
implemented in the code
STUDY IN ENGLISH

Functional testing

• Program is observed as a function which maps values from input

domain to output
• When implementation is unknown, program is a black box
• Program specification is the only information used for selecting tests
• The benefits of functional approach:
• Test cases are independent of concrete implementation – they are usable even
after implementation change
• Test cases development can be performed parallel to the implementation, thus
saving needed time
• Drawbacks:
• Possibly some test case redundancy
• Some parts of implementation are not covered with tests
STUDY IN ENGLISH

Unit testing

• Unit testing provides developers with the

opportunity to test their code
• We frequently test an isolated unit of source code,
or a part of the class
• Unit test is a small peace of code, which tests
functionality and behavior of another part of the
code
• Programmers often use specific environments for
writing unit tests (JUnit).
STUDY IN ENGLISH

Unit testing

• Main advantage of unit tests is the fact that the

execution of these tests is automated
• Programmer writes those tests once, and executes
them numerous times when it is required
• This feature eases regression testing
• Regression testing – bugs which have been fixed,
but again occurred after development of the new
program functionality
STUDY IN ENGLISH

Integration testing

• After unit testing is completed, units and modules are

integrated in a whole
• The goal of integration testing is verification of
functionality, performances and robustness between
integrated modules
• The focus is on verification of interfaces between
modules
• There are several strategies for incremental integration,
preferred in comparison to putting all components at
once – Big bang integration
STUDY IN ENGLISH

System testing

• Testing of the system as a whole when compared to

the specification
• Most of the functional requirements has been
verified in lower levels of testing, so focus can be
on nonfunctional requirements such as security,
speed, robustness etc.