Software Testing Foundations, 5th Edition: A Study Guide for the Certified Tester Exam

Overview

1Introduction

2Software Testing Basics

3Testing Throughout the Software Development Lifecycle

4Static Testing

5Dynamic Testing

6Test Management

7Test Tools

Appendices

AImportant Notes on the Syllabus and the Certified Tester Exam

BGlossary

CReferences

Contents

1Introduction

2Software Testing Basics

2.1Concepts and Motivations

2.1.1Defect and Fault Terminology

2.1.2Testing Terminology

2.1.3Test Artifacts and the Relationships Between Them

2.1.4Testing Effort

2.1.5Applying Testing Skills Early Ensures Success

2.1.6The Basic Principles of Testing

2.2Software Quality

2.2.1Software Quality according to ISO 25010

2.2.2Quality Management and Quality Assurance

2.3The Testing Process

2.3.1Test Planning

2.3.2Test Monitoring and Control

2.3.3Test Analysis

2.3.4Test Design

2.3.5Test Implementation

2.3.6Test Execution

2.3.7Test Completion

2.3.8Traceability

2.3.9The Influence of Context on the Test Process

2.4The Effects of Human Psychology on Testing

2.4.1How Testers and Developers Think

2.5Summary

3Testing Throughout the Software Development Lifecycle

3.1Sequential Development Models

3.1.1The Waterfall Model

3.1.2The V-Model

3.2Iterative and Incremental Development Models

3.3Software Development in Project and Product Contexts

3.4Testing Levels

3.4.1Component Testing

3.4.2Integration Testing

3.4.3System Testing

3.4.4Acceptance Testing

3.5Test Types

3.5.1Functional Tests

3.5.2Non-Functional Tests

3.5.3Requirements-Based and Structure-Based Testing

3.6Testing New Product Versions

3.6.1Testing Following Software Maintenance

3.6.2Testing Following Release Development

3.6.3Regression Testing

3.7Summary

4Static Testing

4.1What Can We Analyze and Test?

4.2Static Test Techniques

4.3The Review Process

4.3.1Review Process Activities

4.3.2Different Individual Review Techniques

4.3.3Roles and Responsibilities within the Review Process

4.4Types of Review

4.5Critical Factors, Benefits, and Limits

4.6The Differences Between Static and Dynamic Testing

4.7Summary

5Dynamic Testing

5.1Black-Box Test Techniques

5.1.1Equivalence Partitioning

5.1.2Boundary Value Analysis

5.1.3State Transition Testing

5.1.4Decision Table Testing

5.1.5Pair-Wise Testing

5.1.6Use-Case Testing

5.1.7Evaluation of Black-Box Testing

5.2White-Box Test Techniques

5.2.1Statement Testing and Coverage

5.2.2Decision Testing and Coverage

5.2.3Testing Conditions

5.2.4Evaluation of White-Box Testing

5.3Experience-Based Test Techniques

5.4Selecting the Right Technique

5.5Summary

6Test Management

6.1Test Organization

6.1.1Independent Testing

6.1.2Roles, Tasks, and Qualifications

6.2Testing Strategies

6.2.1Test Planning

6.2.2Selecting a Testing Strategy

6.2.3Concrete Strategies

6.2.4Testing and Risk

6.2.5Testing Effort and Costs

6.2.6Estimating Testing Effort

6.2.7The Cost of Testing vs. The Cost of Defects

6.3Test Planning, Control, and Monitoring

6.3.1Test Execution Planning

6.3.2Test Control

6.3.3Test Cycle Monitoring

6.3.4Test Reports

6.4Defect Management

6.4.1Evaluating Test Reports

6.4.2Creating a Defect Report

6.4.3Classifying Failures and Defects

6.4.4Defect Status Tracking

6.4.5Evaluation and Reporting

6.5Configuration Management

6.6Relevant Standards and Norms

6.7Summary

7Test Tools

7.1Types of Test Tools

7.1.1Test Management Tools

7.1.2Test Specification Tools

7.1.3Static Test Tools

7.1.4Tools for Automating Dynamic Tests

7.1.5Load and Performance Testing Tools

7.1.6Tool-Based Support for Other Kinds of Tests

7.2Benefits and Risks of Test Automation

7.3Using Test Tools Effectively

7.3.1Basic Considerations and Principles

7.3.2Tool Selection

7.3.3Pilot Project

7.3.4Success Factors During Rollout and Use

7.4Summary

Appendices

AImportant Notes on the Syllabus and the Certified Tester Exam

BGlossary

CReferences

C.1Literature

C.2Norms and Standards

C.3URLs

1Introduction

Software is everywhere! Nowadays there are virtually no devices, machines, or systems that are not partially or entirely controlled by software. Important functionality in cars—such as engine or gear control—have long been software-based, and these are now being complemented by increasingly smart software-based driver assist systems, anti-lock brake systems, parking aids, lane departure systems and, perhaps most importantly, autonomous driving systems. Software and software quality therefore not only govern how large parts of our lives function, they are also increasingly important factors in our everyday safety and wellbeing.

Equally, the smooth running of countless companies today relies largely on the reliability of the software systems that control major processes or individual activities. Software therefore determines future competitiveness. For example, the speed at which an insurance company can introduce a new product, or even just a new tariff, depends on the speed at which the corresponding IT systems can be adapted or expanded.

High dependency on reliable software

Quality has therefore become a crucial factor for the success of products and companies in the fields of both technical and commercial software.

Most companies have recognized their dependence on software, whether relying on the functionality of existing systems or the introduction of new and better ones. Companies therefore constantly invest in their own development skills and improved system quality. One way to achieve these objectives is to introduce systematic software evaluation and testing procedures. Some companies already have comprehensive and strict testing procedures in place, but many projects still suffer from a lack of basic knowledge regarding the capacity and usefulness of software testing procedures.

Grass-roots knowledge of structured evaluation and testing

This book aims to provide the basic knowledge necessary to set up structured, systematic software evaluation and testing techniques that will help you improve overall software quality.

This book does not presume previous knowledge of software quality assurance. It is designed for reference but can also be used for self-study. The text includes a single, continuous case study that provides explanations and practical solutions for each of the topics covered.

This book is aimed at all software testers in all types of companies who want to develop a solid foundation for their work. It is also for programmers and developers who have taken over (or are about to take over) existing test scenarios, and it is also aimed at project managers who are responsible for budgeting and overall procedural improvement. Additionally, it offers support for career changers in IT-related fields and people involved in application approval, implementation, and development.

Especially in IT, lifelong learning is essential, and software testing courses are offered by a broad range of companies and individuals. Universities, too, are increasingly offering testing courses, and this book is aimed at teachers and students alike.

Certification program for software testers

The ISTQB® Certified Tester program is today seen as the worldwide standard for software testing and quality assurance training. The ISTQB® (International Software Testing Qualifications Board) [URL: ISTQB] coordinates qualification activities in individual countries and ensures the global consistency and comparability of the syllabi and exam papers. National Testing Boards are responsible for publishing and maintaining local content as well as the organization and supervision of exams. They also approve courses and offer accreditation for training providers. Testing Boards therefore guarantee that courses are of a consistently high standard and that participants end up with an internationally recognized certificate. Members of the Testing Boards include training providers, testing experts from industrial and consulting firms, and university lecturers. They also include representatives from trade associations.

Three-stage training scheme

The Certified Tester training scheme is made up of units with three levels of qualification. For more details, see the ISTQB® [URL: ISTQB] website. The basics of software testing are described in the Foundation Level syllabus. You can then move on to take the Advanced Level exam, which offers a deeper understanding of evaluation and testing skills. The Expert Level certificate is aimed at experienced software testing professionals, and consists of a set of modules that cover various advanced topics (see also section 6.1.2). In addition, there are syllabi for agile software development (foundation and advanced level) as well as special topics from the testing area (for example, Security Tester, Model-Based Tester, Automotive Software Tester).

This book covers the contents of the Foundation Level syllabus. You can use the book for self-study or in conjunction with an approved course.

Chapter overview

The topics covered in this book and the basic content of the Foundation Certificate course are as follows:

Software testing basics

Chapter 2 discusses the basics of software testing. Alongside the concepts of when to test, the objectives to aim for, and the required testing thoroughness, it also addresses the basic concepts of testing processes. We also talk about the psychological difficulties that can arise when you are looking for errors in your own work.

Lifecycle testing

Chapter 3 introduces common development lifecycle models (sequential, iterative, incremental, agile) and explains the role that testing plays in each. The various test types and test levels are explained, and we investigate the difference between functional and non-functional testing. We also look at regression testing.

Static testing

Static testing (i.e., tests during which the test object is not executed) are introduced in Chapter 4. Reviews and static tests are used successfully by many organizations, and we go into detail on the various approaches you can take.

Dynamic testing

Chapter 5 addresses testing in a stricter sense and discusses black-box and white-box dynamic testing techniques. Various test techniques and methods are explained in detail for both. We wrap up this chapter by looking at when it makes sense to augment common testing techniques using experience-based or intuitive testing techniques.

Test management

Chapter 6 discusses the organizational skills and tasks that you need to consider when managing test processes. We also look at the requirements for defect and configuration management, and wind up with a look at the economics of testing.

Test tools

Testing software without the use of dedicated tools is time-consuming and extremely costly. Chapter 7 introduces various types of testing tools and discusses how to choose and implement the right tools for the job you are doing.

Case Study: VirtualShowRoom VSR-II

Most of the processes described in this book are illustrated using a case study based on the following scenario:

A car manufacturer has been running an electronic sales system called VirtualShowRoom (VSR) for over a decade. The system runs at all the company’s dealers worldwide:

Customers can configure their own vehicle (model, color, extras, and so on) on a computer, either alone or assisted by a salesperson. The system displays the available options and immediately calculates the corresponding price. This functionality is performed by the DreamCar module.

Once the customer has selected a configuration, he can then select optimal financing using the EasyFinance module, order the vehicle using the JustInTime module, and select appropriate insurance using the NoRisk module. The FactBook module manages all customer and contract data.

The manufacturer’s sales and marketing department has decided to update the system and has defined the following objectives:

VSR is a traditional client-server system. The new VSR-II system is to be web-based and needs to be accessible via a browser window on any type of device (desktop, tablet, or smartphone).

The DreamCar, EasyFinance, FactBook, JustInTime, and NoRisk modules will be ported to the new technology base and, during the process, will be expanded to varying degrees.

The new ConnectedCar module is to be integrated into the system. This module collects and manages status data for all vehicles sold, and communicates data relating to scheduled maintenance and repairs to the driver as well as to the dealership and/or service partner. It also provides the driver with various additional bookable services, such as a helpdesk and emergency services. Vehicle software can be updated and activated over the air.

Each of the five existing modules will be ported and developed by a dedicated team. An additional team will develop the new ConnectedCar module. The project employs a total of 60 developers and other specialists from internal company departments as well as a number of external software companies.

The teams will work using the Scrum principles of agile development. This agile approach requires each module to be tested during each iteration. The system is to be delivered incrementally.

In order to avoid complex repeat data comparisons between the old and new systems, VSR-II will only go live once it is able to duplicate the functionality provided by the original VSR system.

Within the scope of the project and the agile approach, most project participants will be confronted or entrusted with test tasks to varying degrees. This book provides the basic knowledge of the test techniques and processes required to perform these tasks. Figure 1-1 shows an overview of the planned VSR-II system.

Fig. 1-1VSR-II overview

Certified Tester syllabus and exam

The appendices at the end of the book include references to the syllabus and Certified Tester exam, a glossary, and a bibliography. Sections of the text that go beyond the scope of the syllabus are marked as side notes.

The book’s website

The book’s website [URL: Softwaretest Knowledge] includes sample exam questions relating to each chapter, updates and addenda to the text, and references to other books by authors whose work supports the Certified Tester training scheme.

Web-based Training System vsr.testbench.com

We have put a free implementation of VSR-II as a test object online for training purposes¹. It reproduces the VSR-II examples included in the book on a realistic, executable system, so you can test live to find the software bugs hidden in VSR-II by applying the test techniques presented in the book. It takes just a few mouse clicks to get started:

Open your browser and load vsr.testbench.com

Create your personal VSR-II training workspace

Log into your VSR-II workspace and start

Fig. 1-2VSR-II Training System Login-Screen

Also included in your registration for a VSR-II training workspace is a free basic license for the test management system TestBench CS, which includes the VSR-II test specification as a demo project and several of the VSR-II test cases presented in the book.

You can use TestBench CS not only for learning and training, but also for efficient testing of your own real software. A description of all features can be found at [URL: TestBench CS].

Many thanks to our colleagues at imbus Academy, imbus JumpStart and imbus TestBench CS Development Team for this awesome implementation of the VSR-II Case Study as a web-based training system.

2Software Testing Basics

This introductory chapter explains the basic concepts of software testing that are applied in the chapters that follow. Important concepts included in the syllabus are illustrated throughout the book using our practical VSR-II case study. The seven fundamental principles of software testing are introduced, and the bulk of the chapter is dedicated to explaining the details of the testing process and the various activities it involves. To conclude, we will discuss the psychological issues involved in testing, and how to avoid or work around them.

2.1Concepts and Motivations

Quality requirements

Industrially manufactured products are usually spot-checked to make sure they fulfill the planned requirements and perform the required task. Different products have varying quality requirements and, if the final product is flawed or faulty, the production process or the design has to be modified to remedy this.

Software is intangible

What is generally true for industrial production processes is also true for the development of software. However, checking parts of the product or the finished product can be tricky because the product itself isn’t actually tangible, making hands-on testing impossible. Visual checks are limited and can only be performed by careful scrutiny of the development documentation.

Faulty software is a serious problem

Software that is unreliable or that simply doesn’t perform the required task can be highly problematic. Bad software costs time and money and can ruin a company’s reputation. It can even endanger human life—for example, when the autopilot software in a partially autonomous vehicle reacts erroneously or too late.

Testing helps to assess software quality

It is therefore extremely important to check the quality of a software product to minimize the risk of failures or crashes. Testing monitors software quality and reduces risk by revealing faults at the development stage. Software testing is therefore an essential but also highly complex task.

Case Study: The risks of using faulty software

Every release of the VSR-II system has to be suitably tested before it is delivered and rolled out. This aims to identify and remedy faults before they can do any damage. For example, if the system executes an order in a faulty way, this can cause serious financial problems for the customer, the dealer and the manufacturer, as well as damaging the manufacturer’s image. Undiscovered faults like this increase the risk involved in running the software.

Testing involves taking a spot-check approach

Testing is often understood as spot-check execution¹ of the software in question (the test object) on a computer. The test object is fed with test data covering various test cases and is then executed. The evaluation that follows checks whether the test object fulfills its planned requirements.²

Testing involves more than just executing tests on a computer

However, testing involves much more than just performing a series of test cases. The test process involves a range of separate activities, and performing tests and checking the results are just two of these. Other testing activities include test planning, test analysis, and the design and implementation of test cases. Additional activities include writing reports on test progress and results, and risk analysis. Test activities are organized differently depending on the stage in a product’s lifecycle. Test activities and documentation are often contractually regulated between the customer and the supplier, or are based on the company’s own internal guidelines. Detailed descriptions of the individual activities involved in software testing are included in sections 2.3 and 6.3.

Static and dynamic testing

Alongside the dynamic tests that are performed on a computer (see Chapter 5), documents such as requirement specifications, user stories, and source code also need to be tested as early as possible in the development process. These are known as static tests (see Chapter 4). The sooner faults in the documentation are discovered and remedied, the better it is for the future development process, as you will no longer be working with flawed source material.

Verification and validation

Testing isn’t just about checking that a system fulfills its requirements, user stories, or other specifications; it is also about ensuring that the product fulfills the wishes and expectations of its users in a real-world environment. In other words, checking whether it is possible to use the system as intended and making sure it fulfills its planned purpose. Testing therefore also involves validation (see Principle #7 in section 2.1.6—Absence-of-errors is a fallacy).

No large system is fault-free

There is currently no such thing as a fault-free software system, and this situation is unlikely to change for systems above a given degree of complexity or those with a large number of lines of code. Many faults are caused by a failure to identify or test for exceptions during code development—things like failing to account for leap years, or not considering constraints when it comes to timing or resource allocation. It is therefore common—and sometimes unavoidable—that software systems go live, even though faults still occur for certain combinations of input data. However, other systems work perfectly day in day out in all manner of industries.

Freedom from faults cannot be achieved through testing

With the exception of very small programs, even if every test you perform returns zero defects, you cannot be sure that additional tests won’t reveal previously undiscovered faults. It is impossible to prove complete freedom from faults by testing.

2.1.1Defect and Fault Terminology

The test basis as a starting point for testing

A situation can only be classed as faulty if you define in advance what exactly is supposed to happen in that situation. In order to make such a definition, you need to know the requirements made of the (sub)system you are testing as well as other additional information. In this context, we talk about the test basis against which tests are performed and that serves as the basis for deciding whether a specific function is faulty.

What counts as a defect?

A defect is therefore defined as a failure to fulfill a predefined requirement, or a discrepancy between the actual behavior (at run time or while testing) and the expected behavior (as defined in the specifications, the requirements, or the user stories). In other words, when does the system’s behavior fail to conform to its actual requirements?

Unlike physical systems, software systems don’t fail due to age or wear. Every defect that occurs is present from the moment the software is coded, but only becomes apparent when the system is running.

Faults cause failures

System failures result from faults and only become apparent to the tester or the user during testing or at run-time. For example, when the system produces erroneous output or crashes.

We need to distinguish between the effects of a fault and its causes. A system failure is caused by a fault in the software, and the resulting condition is considered to be a defect. The word bug is also used to describe defects that result from coding errors, such as an incorrectly programmed or forgotten instruction in the code.

Defect masking

It is possible that a fault can be offset by one or more other faults in other parts of the program. Under these circumstances, the fault in question only becomes apparent when the others have been remedied. In other words, correcting a fault in one place can lead to unexpected side effects in others.

Not all faults cause system failures, and some failures occur never, once, or constantly for all users. Some failures occur a long way from where they are caused.

A fault is always the result of an error or a mistake made by a person—for example, due to a programming error at the development stage.

People make errors

Errors occur for various reasons. Some typical (root) causes are:

All humans make errors!

Time pressure is often present in software projects and is a regular source of errors.

The complexity of the task at hand, the system architecture, the system design, or the source code.

Misunderstandings between participants in the project—often in the form of differing interpretations of the requirements or other documents.

Misunderstandings relating to system interaction via internal and external interfaces. Large systems often have a huge number of interfaces.

The complexity of the technology in use, or of new technologies previously unknown to project participants that are introduced during the project.

Project participants are not sufficiently experienced or do not have appropriate training.

A human error causes a fault in part of the code, which then causes some kind of visible system failure that, ideally, is revealed during testing (see figure 2-1: Debugging, see below). Static tests (see Chapter 4) can directly detect faults in the source code.

System failures can also be caused by environmental issues such as radiation and magnetism, or by physical pollution that causes hardware and firmware failures. We will not be addressing these types of failures here.

Fig. 2-1The relationships between, errors, faults, and failures

False positive and false negative results

Not every unexpected test result equates to a failure. Often, a test will indicate a failure even though the underlying fault (or its cause) isn’t part of the test object. Such a result is known as a false positive. The opposite effect can also occur—i.e., a failure doesn’t occur even though testing should reveal its presence. This type of result is known as a false negative. You have to bear both of these situations in mind when evaluating your test results. Your result can also be a correct positive (failure revealed by testing) or a correct negative (expected behavior confirmed by testing). For more detail on these situations, see section 6.4.1.

Learning from your mistakes

If faults and the errors or mistakes that cause them are revealed by testing it is worth taking a closer look at the causes in order to learn how to avoid making the same (or similar) errors or mistakes in future. The knowledge you gain this way can help you optimize your processes and reduce or prevent the occurrence of additional faults.

Case Study: Vague requirements as a cause of software faults

Customers can use the VSR EasyFinance module to calculate various vehicle-financing options. The interest rate the system uses is stored in a table, although the purchase of vehicles involved in promotions and special offers can be subject to differing interest rates.

VSR-II is to include the following additional requirement:

REQ: If the customer agrees to and passes an online credit check, the EasyFinance module applies an interest rate from a special bonus interest rate table.

The author of this requirement unfortunately forgot to clarify that a reduction in the interest rate is not permissible for vehicles sold as part of a special offer. This resulted in this special case not being tested in the first release. In turn, this meant that customers were offered low interest rates online but were charged higher rates when billed, resulting in complaints.

2.1.2Testing Terminology

Testing is not debugging

In order to remedy a software fault it has to be located. To start with, we only know the effect of the fault, but not its location within the code. The process of finding and correcting faults is called debugging and is the responsibility of the developer. Debugging is often confused with testing, although these are two distinct and very different tasks. While debugging pinpoints software faults, testing is used to reveal the effect a fault causes (see figure 2-1).

Confirmation testing

Correcting a fault improves the quality of the product (assuming the correction doesn’t cause additional, new faults). Tests used to check that a fault has been successfully remedied are called confirmation tests. Testers are often responsible for confirmation testing, whereas developers are more likely to be responsible for component testing (and debugging). However, these roles can change in an agile development environment or for other software lifecycle models.

Unfortunately, in real-world situations fault correction often leads to the creation of new faults that are only revealed when completely new input scenarios are used. Such unpredictable side effects make testing trickier. Once a fault has been corrected you need to repeat your previous tests to make sure the targeted failure has been remedied, and you also need to write new tests that check for unwanted side effects of the correction process.

Objectives of testing

Static and dynamic tests are designed to achieve various objectives:

A qualitative evaluation of work products related to the requirements, the specifications, user stories, program design, and code

Prove that all specific requirements have been completely implemented and that the test object functions as expected for the users and other stakeholders

Provide information that enables stakeholders to make a solid estimate of the test object’s quality and thus generate confidence in the quality provided³

The level of quality-related risk can be reduced through identification and correction of software failures. The system will then contain fewer undiscovered faults.

Analysis of the program and its documentation in order to avoid unwanted faults, and to document and remedy known ones

Analyze and execute the program in order to reproduce known failures

Receive information about the test object in order to decide whether the component in question can be committed for integration with other components

Demonstrate that the test object adheres and/or conforms to the necessary contractual, legal and regulatory requirements and standards

Objectives depend on context

Test objectives can vary depending on the context. Furthermore, they can vary according to the development model you use (agile or otherwise) and the level of test you are performing—i.e., component, integration, system, or acceptance tests (see section 3.4).

When you are testing a component, your main objective should be to reveal as many failures as possible and to identify (i.e., debug) and remedy the underlying faults as soon as possible. Another primary objective can be to select tests that achieve the maximum possible level of code coverage (see section 2.3.1).

One objective of acceptance testing is to confirm that the system works and can be used as planned, and thus fulfills all of its functional and non-functional requirements. Another is to provide information that enables stakeholders to evaluate risk and make an informed decision about whether (or not) to go live.

Side Note: Scheme for naming different types of testing

The various names used for different types of tests can be confusing. To understand the naming of tests it is useful to differentiate between the following naming categories:

Test objective

The naming of a test type is based on the test objective (for example, a load test).

Test method/technique

A test is named according to the method or technique used to specify and/or perform the test (i.e., state transition testing, as described in section 5.1.3)

Test object

A test is named according to the type of object to be tested (for example, GUI test or database test)

Test level

A test is named according to the corresponding level of the development model being used (for example, a system test)

Test person

A test is named after the person or group who perform the test (for example, developer test, user test)

Test scope

A test is named according to its scope (for example, a partial regression test)

As you can see, not all of these terms define a distinct type of test. Instead, the different names highlight different aspects of a test that are important or in focus in a particular context or with regard to a particular testing objective.

2.1.3Test Artifacts and the Relationships Between Them

The previous sections have already described some types of test artifacts. The following sections provide an overview of the types of artifacts necessary to perform dynamic testing.

Test basis

The test basis is the cornerstone of the testing process. As previously noted, the test basis comprises all documents that help us to decide whether a failure has occurred during testing. In other words, the test basis defines the expected behavior of the test object. Common sense and specialist knowledge can also be seen as part of the test basis and can be used to reach a decision. In most cases a requirements document, a specification, or a user story is available, which serves as a test basis.

Test cases and test runs

The test basis is used to define test cases, and a test run takes place when the test object is fed with appropriate test data and executed on a computer. The results of the test run are checked and the team decides whether a failure has occurred—i.e., whether there is a discrepancy between the test object’s expected and actual behaviors. Usually,