Chapter 7 – Software Testing

Program testing

 Testing is intended to show that a program does what it is intended to do and to discover
program defects before it is put into use.

 When you test software, you execute a program using artificial data.

 Running test allows you to identify errors and anomalies in the program.

 Testing is part of a more general verification and validation process, which also includes
static validation techniques.

2
Program testing goals

 To demonstrate to the developer and the customer that the software meets its requirements.
 For custom software, this means that there should be at least one test for every requirement in the
requirements document.
 For generic software products, it means that there should be tests for all of the system features, plus
combinations of these features, that will be incorporated in the product release.

 To discover situations in which the behavior of the software is incorrect, undesirable, or

does not conform to its specification. These are a consequence of software defects. Defect
testing is concerned with rooting out undesirable system behavior such as:
 System crashes
 Unwanted interactions with other systems
 Incorrect computations
 Data corruption

3
Validation and defect testing

 The first goal leads to validation testing.

 You expect the system to perform correctly using a given set of test cases that reflect the system’s expected
use.

 The second goal leads to defect testing.

 The test cases are designed to expose defects. The test cases in defect testing need not reflect how the
system is normally used

 There is no definite boundary between these two approaches to testing.

4
An input-output model of program testing

5
Verification vs validation

 Verification.
 "Are we building the product right”.
 The software should conform to its specification.

 Validation.
 "Are we building the right product”.
 The software should do what the user really requires.

6
V & V confidence

 Aim of V & V is to establish confidence that the system is ‘fit for purpose’.

 This confidence depends on:

 Software purpose. The level of confidence depends on how critical the software is to an organisation.
 User expectations. Users may have low expectations of certain kinds of software. They may tolerate
failures because the benefits of use outweigh the costs of failure recovery.
 Marketing environment. Getting a product to market early may be more important than finding defects in
the program.

7
Inspections and testing

 Software inspections Concerned with analysis of the static system representation to

discover problems (static verification).

 Software testing Concerned with exercising and observing product behaviour (dynamic
verification).
 The system is executed with test data and its operational behaviour is observed.

8
Inspections and testing

9
Inspections and testing

 Inspections and testing are complementary and not opposing verification techniques.

 Both should be used during the V & V process.

 Inspections can check conformance with a specification but not conformance with the
customer’s real requirements.

 Inspections cannot check non-functional characteristics such as performance, usability, etc.

10
Inspections

 These involve people examining the source representation with the aim of discovering
anomalies and defects.

 Inspections not require execution of a system so may be used before implementation.

 They may be applied to any representation of the system (requirements, design,

configuration data, test data, etc.).

 They have been shown to be an effective technique for discovering program errors.

11
Advantages of software inspection over testing

 During testing, errors can mask (hide) other errors. Because inspection is a static process,
you don’t have to be concerned with interactions between errors.

 Incomplete versions of a system can be inspected without additional costs. If a program is

incomplete, then you need to develop specialized test harnesses to test the parts that are
available.

 As well as searching for program defects, an inspection can also consider broader quality
attributes of a program, such as compliance with standards, portability and maintainability.

12
A model of the software testing process

13
Stages of testing

 Development testing, where the system is tested during development to discover bugs and
defects.

 Release testing, where a separate testing team test a complete version of the system
before it is released to users.

 User testing, where users or potential users of a system test the system in their own
environment.

14
Development testing

 Development testing includes all testing activities that are carried out by the team that
develops the system.

 Unit testing, where individual program units or object classes are tested. Unit testing should focus on
testing the functionality of objects or methods.

 Component testing, where several individual units are integrated to create composite components.
Component testing should focus on testing component interfaces.

 System testing, where some or all of the components in a system are integrated and the system is tested
as a whole. System testing should focus on testing component interactions.

15
Unit testing

 Unit testing is the process of testing individual components in isolation.

 Units may be:

 Individual functions or methods within an object

 Object classes with several attributes and methods

16
Object class testing

 Test all operations associated with the object.

 Set and check the value of all attributes associated with the object.

 Put the object into all possible states.

17
Object class testing

 Check if “identifier” attribute has been properly set

up.

 Define test cases for the methods “reportWeather”,

“reportStatus”, …, etc.

18
Object class testing

 Test the states of the weather

station.
 Shutdown -> Running ->
Shutdown
 Configuring -> Running ->
Testing -> Transmitting ->
Running
 Running -> Collecting ->
Running -> Summarizing ->
Transmitting -> Running

19
Automated testing

 You should automate unit testing.

 An automated test has three parts:

 A setup part (the inputs and expected outputs)
 A call part (call the object or method to be tested)
 An assertion part (compare the result of the call with the expected result)

20
Unit testing using PHPUnit

21
Unit testing using PHPUnit

22
Unit testing using PHPUnit

23
Unit testing using PHPUnit

 Create a UserTest class.

24
Unit testing using PHPUnit

 Test Constructor using a testClassConstructor method.

25
Unit testing using PHPUnit

 Run tests in PHPUnit.

 You can run all the tests in a directory using the PHPUnit command.

Folder name

 You can also run a single test by providing the path to the test file.

26
Unit testing using PHPUnit

 Test Output.
 The output shows that we ran 1 test, and made 3 assertions in it. We also see how long it took to
run the test, as well as how much memory was used in running the test.

27
Unit testing using PHPUnit

 Edit the code to check if the user age is the same as 21.

28
Unit testing using PHPUnit

 Running the test again this time gives this output.

1 test, with 2 successful assertions,

and also a failed one.

29
Unit testing using PHPUnit

 Test tellName.

30
Unit testing using PHPUnit

 Test tellAge.

31
Unit testing using PHPUnit

 Test addFavoriteMovie.

32
Unit testing using PHPUnit

 Test removeFavoriteMovie.

33
Component testing

 Software components are often composite components that are made up of several
interacting objects.
 In the weather station system, the reconfiguration component includes objects that deal with each
aspect of the reconfiguration.

 You access the functionality of these objects through the defined component
interface.

 Testing composite components focus on the component interface.

34
Component testing

 There are different types of interface between program components:

 Parameter interfaces (Methods in an object have a parameter interface).

 Shared memory interfaces where a block of memory is shared between components. This type of
interface is often used in embedded systems.

 Procedural interfaces where one component encapsulates a set of procedures

 Message passing interfaces where one component requests a service from another component.

35
Component testing

 Interface errors fall into three classes:

 Interface misuse. An error in the use of its interface when calling a component (wrong type
parameters, parameters passed in the wrong order, wrong number of parameters are passed).

 Interface misunderstanding. A calling component misunderstands the specification of the interface

(a binary search method is called with an unordered array).

 Timing errors. These occur in real-time systems that use a shared memory or a message-passing
interface (the consumer can access out-of-date information).

36
System testing

 System testing during development involves integrating components to create a

version of the system and then testing the integrated system.

 It obviously overlaps with component testing but there are two important differences:

 Reusable components that have been separately developed and off-the-shelf systems may be
integrated with newly developed components.

 Components developed by different team members or groups may be integrated at this stage.

37
System testing - Use-case testing

38
Test-driven development

 Test-driven development (TDD) is an approach to program development in which you

interleave testing and code development.

 Essentially, you develop the code incrementally, along with a test for that increment.
You don’t move on to the next increment until the code that you have developed
passes its test.

 Test-driven development was introduced as part of agile methods such as Extreme

Programming. However, it can also be used in plan-driven development processes.

39
Test-driven development

40
Test-driven development

 Test-driven development (TDD) is an approach to program development in which you

interleave testing and code development.

 Essentially, you develop the code incrementally, along with a test for that increment.
You don’t move on to the next increment until the code that you have developed
passes its test.

 Test-driven development was introduced as part of agile methods such as Extreme

Programming. However, it can also be used in plan-driven development processes.

41
Benefits of test-driven development

 Code coverage
 Every code segment that you write has at least one associated test so all code written has at least
one test.
 Regression testing
 A regression test suite is developed incrementally as a program is developed.
 Simplified debugging
 When a test fails, it should be obvious where the problem lies. The newly written code needs to be
checked and modified.
 Simplified debugging
 The tests themselves are a form of documentation that describe what the code should be doing.

42
Regression testing

 Regression testing is testing the system to check that changes have not ‘broken’
previously working code.

 In a manual testing process, regression testing is expensive but, with automated

testing, it is simple and straightforward. All tests are rerun every time a change is
made to the program.

 Tests must run ‘successfully’ before the change is committed.

43
Release testing

 Release testing is the process of testing a particular release of a system that is

intended for use outside of the development team.

 There are two important distinctions between release testing and system testing.
 A separate team that has not been involved in the system development, should be responsible for
release testing.
 System testing by the development team should focus on discovering bugs in the system (defect
testing). The objective of release testing is to check that the system meets its requirements and is
good enough for external use (validation testing).

 The primary goal of the release testing process is to convince the supplier of the
system that it is good enough for use. If so, it can be released as a product or
delivered to the customer. 44
Requirements-based testing

 Requirements-based testing involves examining each requirement and developing a

test or tests for it.

 MHC-PMS requirements:
 If a patient is known to be allergic to any particular medication, then prescription of that medication
shall result in a warning message being issued to the system user.
 If a prescriber chooses to ignore an allergy warning, they shall provide a reason why this has been
ignored.

45
Requirements-based testing

 To check if these requirements have been satisfied, you may need to develop the
above related tests.
 Set up a patient record with no known allergies. Prescribe medication for allergies that are known
to exist. Check that a warning message is not issued by the system.
 Set up a patient record with a known allergy. Prescribe the medication to that the patient is allergic
to, and check that the warning is issued by the system.
 Set up a patient record in which allergies to two or more drugs are recorded. Prescribe both of
these drugs separately and check that the correct warning for each drug is issued.
 Prescribe two drugs that the patient is allergic to. Check that two warnings are correctly issued.
 Prescribe a drug that issues a warning and overrule that warning. Check that the system requires
the user to provide information explaining why the warning was overruled.

46
Scenario testing

Kate is a nurse who specializes in mental health care. One of her responsibilities is to visit
patients at home to check that their treatment is effective and that they are not suffering from
medication side -effects.
On a day for home visits, Kate logs into the MHC-PMS and uses it to print her schedule of
home visits for that day, along with summary information about the patients to be visited. She
requests that the records for these patients be downloaded to her laptop. She is prompted for
her key phrase to encrypt the records on the laptop.
One of the patients that she visits is Jim, who is being treated with medication for depression.
Jim feels that the medication is helping him but believes that it has the side -effect of keeping
him awake at night. Kate looks up Jim’s record and is prompted for her key phrase to decrypt
the record. She checks the drug prescribed and queries its side effects. Sleeplessness is a
known side effect so she notes the problem in Jim’s record and suggests that he visits the
clinic to have his medication changed. He agrees so Kate enters a prompt to call him when
she gets back to the clinic to make an appointment with a physician. She ends the
consultation and the system re-encrypts Jim’s record.
After, finishing her consultations, Kate returns to the clinic and uploads the records of
patients visited to the database. The system generates a call list for Kate of those patients
who she has to contact for follow-up information and make clinic appointments.

47
Scenario testing

 The Features tested by scenario are:

 Authentication by logging on to the system.
 Downloading and uploading of specified patient records to a laptop.
 Home visit scheduling.
 Encryption and decryption of patient records on a mobile device.
 Record retrieval and modification.
 Links with the drugs database that maintains side-effect information.
 The system for call prompting.
 Release tester should play the role of Kate and observing how the system behaves
in response to different inputs.
 Scenario testing checks that combinations of requirements do not cause problems.

48
Performance testing

 Part of release testing may involve testing the emergent properties of a system, such
as performance and reliability.

 Tests should reflect the profile of use of the system.

 Performance tests usually involve planning a series of tests where the load is
steadily increased until the system performance becomes unacceptable.

 Stress testing is a form of performance testing where the system is deliberately

overloaded to test its failure behaviour. is particularly relevant to distributed systems.
49
User testing

 User or customer testing is a stage in the testing process in which users or

customers provide input and advice on system testing.

 User testing is essential, even when comprehensive system and release testing have
been carried out.

 In practice, there are three different types of user testing.

50
Types of user testing

 Alpha testing
 Users of the software work with the development team to test the software at the developer’s site.

 Beta testing
 A release of the software is made available to users to allow them to experiment and to raise
problems that they discover with the system developers. Beta testers may be early adopters of the
system or anyone who is interested in it (if it is publicly available).

 Alpha testing
 Customers test a system to decide whether or not it is ready to be accepted from the system
developers and deployed in the customer environment. Primarily for custom systems.

51