Chapter 1

Software Testing - ISTQB Advance

Test Manager Exam Preparation

Testing process

1 Testing Process 2 Test Management 3 Reviews 4 Defect Management

5 Improving Process 6 Test Tools 7 People Skills

Testing Process
Contents
1.1 Introduction
1.2 Test Planning,
Monitoring and Control

1.3 Test Analysis

1.4 Test Design
1.5 Test Implementation
1.6 Test Execution
1.7 Evaluating Exit Criteria and Reporting
1.8 Test Closure Activities
Testing Process
Introduction
The ISTQB® Foundation Level syllabus describes a fundamental test process which includes the
following activities:

 Planning,
 Monitoring and control
 Analysis
 Design
 Implementation
 Execution
 Evaluating exit criteria and reporting
 Test closure activities
Testing Process
Test Planning
 For each test level, test planning starts at the initiation of the test process for that level and
continues throughout the project until the completion of closure activities for that level. It
involves the identification of the activities and resources required to meet the mission and
objectives identified in the test strategy.
 Test planning also includes identifying the methods for gathering and tracking the metrics that
will be used to guide the project, determine adherence to plan and assess achievement of the
objectives.
 The strategy (or strategies) selected for the testing project help to determine the tasks that
should occur during the planning stages.
 Risk information may also be used to determine the priorities of the various testing activities.
 The test planning stage is where the approach to testing is clearly defined by the Test Manager,
including which test levels will be employed, the goals and objectives of each level, and what
test techniques will be used at each level of testing.
 Complex relationships may exist between the test basis (e.g., specific requirements or risks), test
conditions and the tests that cover them. Many-to-many relationships often exist between these
work products.
 Relationships may also exist between work products produced by the development team and
the testing team.
 The test plan may also list the specific features of the software that are within its scope (based
on risk analysis, if appropriate), as well as explicitly identifying features that are not within its
scope.
 There may also be a requirement at this stage for the Test Manager to work with the project
architects to define the initial test environment specification, to verify availability of the
resources required, to ensure that the people who will configure the environment are
committed to do so, and to understand cost/delivery timescales and the work required to
complete and deliver the test environment.
 Finally, all external dependencies and associated service level agreements (SLAs) should be
identified and, if required, initial contact should be made. Examples of dependencies are
resource requests to outside groups, dependencies on other projects (if working within a
program), external vendors or development partners, the deployment team, and database
administrators.
Testing Process
Contents
1.1 Introduction
1.2 Test Planning, Monitoring and Control
1.3 Test Analysis
1.4 Test Design
1.5 Test Implementation
1.6 Test Execution
1.7 Evaluating Exit Criteria and Reporting
1.8 Test Closure Activities
Testing Process
Test Monitoring and Control
 In order for a Test Manager to provide efficient test control, a testing schedule and
monitoring framework needs to be established to enable tracking of test work products
and resources against the plan. This framework should include the detailed measures
and targets that are needed to relate the status of test work products and activities to
the plan and strategic objectives.
 For small and less complex projects, it may be relatively easy to relate test work products
and activities to the plan and strategic objectives, but generally more detailed objectives
need to be defined to achieve this.
 Of particular importance is the need to relate the status of test work products and
activities to the test basis in a manner that is understandable and relevant to the project
and business stakeholders.
 Defining targets and measuring progress based on test conditions and groups of test
conditions can be used as a means to achieve this by relating other testing work
products to the test basis via the test conditions.
 Sometimes, the detailed measures and targets that stakeholders require to be
monitored do not relate directly to system functionality or a specification, especially if
there is little or no formal documentation.
 Involvement of business stakeholders at an early stage in a project can help define these
measures and targets which not only can be used to help provide better control during
the project, but can also help to drive and influence the testing activities throughout the
project.
 Test control is an ongoing activity.
 It involves comparing actual progress against the plan and implementing corrective
actions when needed.
 Test control guides the testing to fulfill the mission, strategies, and objectives, including
revisiting the test planning activities as needed.
 Appropriate reactions to the control data depend on detailed planning information.
Testing Process
Contents
1.1 Introduction
1.2 Test Planning, Monitoring and Control
1.3 Test Analysis
1.4 Test Design
1.5 Test Implementation
1.6 Test Execution
1.7 Evaluating Exit Criteria and Reporting
1.8 Test Closure Activities

Testing Process
Test Analysis
 Test analysis is the activity that defines "what" is to be tested in the form of test
conditions.
 Test conditions can be identified by analysis of the test basis, test objectives, and
product risks.
 Test conditions should also be traceable forward to test designs and other test work
products as those work products are created.
 Test analysis for a given level of testing can be performed as soon as the basis for testing
is established for that level.
 Formal test techniques and other general analytical techniques (e.g., analytical risk-
based strategies and analytical requirements-based strategies) can be used to identify
test conditions.
Testing Process
Test Analysis
There are a number of factors to consider when deciding on the level of detail at which to specify test
conditions, including:

 Level of testing
 Level of detail and quality of the test basis
 System/software complexity
 Project and product risk
 The relationship between the test basis, what is to be tested and how it is to be tested
 Software development lifecycle in use
 Test management tool being utilized
 Level at which test design and other test work products are to be specified and documented
 Skills and knowledge of the test analysts
 The level of maturity of the test process and the organization itself (note that higher maturity
may require a greater level of detail, or allow a lesser level of detail)
 Availability of other project stakeholders for consultation

Specifying test conditions in a detailed fashion will tend to result in a larger number of test conditions.
Some advantages of specifying test conditions at a detailed level include:

 Facilitates more flexibility in relating other test work products (e.g., test cases) to the test basis
and test objectives, thus providing better and more detailed monitoring and control for a Test
Manager
 Contributes to defect prevention, as discussed in the Foundation Level, by occurring early in a
project for higher levels of testing, as soon as the test basis is established and potentially before
system architecture and detailed design are available
 Relates testing work products to stakeholders in terms that they can understand
 Helps influence and direct not just other testing activities, but also other development activities
 Enables test design, implementation and execution, together with the resulting work products to
be optimized by more efficient coverage of detailed measures and targets
 Provides the basis for clearer horizontal traceability within a test level
Testing Process
Test Analysis

Some disadvantages of specifying test conditions at a detailed level include:

 Potentially time-consuming
 Maintainability can become difficult in a changing environment
 Level of formality needs to be defined and implemented across the team

Specification of detailed test conditions can be particularly effective in the following situations:

 Lightweight test design documentation methods, such as checklists, are being used due to
accommodate the development lifecycle, cost and/or time constraints or other factors
 Little or no formal requirements or other development work products are available as the test
basis
 The project is large-scale, complex or high risk and requires a level of monitoring and control
that cannot be delivered by simply relating test cases to development work products

Test conditions may be specified with less detail when the test basis can be related easily and directly to
test design work products. This is more likely to be the case for the following:

 Component level testing

 Less complex projects where simple hierarchical relationships exist between what is to be tested
and how it is to be tested
 Acceptance testing where use cases can be utilized to help define tests

Testing Process
Contents
1.1 Introduction
1.2 Test Planning, Monitoring and Control
1.3 Test Analysis
1.4 Test Design
1.5 Test Implementation
1.6 Test Execution
1.7 Evaluating Exit Criteria and Reporting
1.8 Test Closure Activities
Testing Process
Test Design
 Test design is the activity that defines "how" something is to be tested. It involves the
identification of test cases by the stepwise elaboration of the identified test conditions or test
basis using test techniques identified in the test strategy and/or the test plan.
 Depending on the approaches being used for test monitoring, test control, and traceability, test
cases may be directly related (or indirectly related via the test conditions) to the test basis and
defined objectives. These objectives include strategic objectives, test objectives and other
project or stakeholder criteria for success.
 Test design for a given test level can be performed once test conditions are identified and
enough information is available to enable the production of either low or high-level test cases,
according to the employed approach to test design.
 For higher levels of testing, it is more likely that test design is a separate activity following earlier
test analysis. For lower levels of testing, it is likely that test analysis and design will be conducted
as an integrated activity.
 It is also likely that some tasks that normally occur during test implementation will be integrated
into the test design process when using an iterative approach to building the tests required for
execution; e.g., the creatfest design is t of test data. In fact, this approach can optimize the
coverage of test conditions, either creating low-level;dentification of high-level test cases in the
process.

Testing Process
Contents
1.1 Introduction
1.2 Test Planning, Monitoring and Control
1.3 Test Analysis
1.4 Test Design
1.5 Test Implementation
1.6 Test Execution
1.7 Evaluating Exit Criteria and Reporting
1.8 Test Closure Activities
Testing Process
Test Implementation

 Test implementation is the activity during which tests are organized and prioritized by
the Test Analysts. In formally-documented contexts, test implementation is the activity
in which test designs are implemented as concrete test cases, test procedures, and test
data.
 Test implementation also involves final checks to ensure the test team is ready for test
execution to take place. Checks could include ensuring delivery of the required test
environment, test data and code (possibly running some test environment and/or code
acceptance tests) and that all test cases have been written, reviewed and are ready to be
run. It may also include checking against explicit and implicit entry criteria for the test
level.
 The level of detail and associated complexity of work done during test implementation
may be influenced by the detail of the test work products (e.g., test cases and test
conditions). In some cases, particularly where tests are to be archived for long-term re-
use in regression testing, tests may provide detailed descriptions of the steps necessary
to execute a test, so as to ensure reliable, consistent execution regardless of the tester
executing the test.
 During test implementation, the order in which manual and automated tests are to be
run should be included in a test execution schedule. Test Managers should carefully
check for constraints, including risks and priorities, that might require tests to be run in a
particular order or on particular equipment. Dependencies on the test environment or
test data must be known and checked.
 There may be some disadvantages to early test implementation. With an Agile lifecycle,
for example, the code may change dramatically from iteration to iteration, rendering
much of the implementation work obsolete.
 Even without a lifecycle as change-prone as Agile, any iterative or incremental lifecycle
may result in significant changes between iterations, making scripted tests unreliable or
subject to high maintenance needs.
 There may be some advantages in early test implementation. For example, concrete
tests provide worked examples of how the software should behave, if written in
accordance with the test basis.
 Business domain experts are likely to find verification of concrete tests easier than
verification of abstract business rules, and may thereby identify further weaknesses in
software specifications.
 Testing Process
Contents
1.1 Introduction
1.2 Test Planning, Monitoring and Control
1.3 Test Analysis
1.4 Test Design
1.5 Test Implementation
1.6 Test Execution
1.7 Evaluating Exit Criteria and Reporting
1.8 Test Closure Activities

Testing Process
Test Execution
 Test execution begins once the test object is delivered and the entry criteria to test
execution are satisfied. Tests should be designed or at least defined prior to test
execution.
 Tools should be in place, particularly for test management, defect tracking and (if
applicable) test execution automation.
 Test results tracking, including metrics tracking, should be working and the tracked data
should be understood by all team members.
 Standards for test logging and defect reporting should be available and published. By
ensuring these items are in place prior to test execution, the execution can proceed
efficiently.
 Tests should be executed according to the test cases, although the Test Manager should
consider allowing some amount of latitude so that the tester can cover additional
interesting test scenarios and behaviors that are observed during testing.
 When following a test strategy that is at least in part reactive, some time should be
reserved for test sessions using experience-based and defect-based techniques.
 Of course, any failure detected during such unscripted testing must describe the
variations from the written test case that are necessary to reproduce the failure.
Automated tests will follow their defined instructions without deviation.
  The main role of a Test Manager during test execution is to monitor progress according
to the test plan and, if required, to initiate and carry out control actions to guide testing
toward a successful conclusion in terms of mission, objectives, and strategy.
 To do so, the Test Manager can use traceability from the test results back to the test
conditions, the test basis, and ultimately the test objectives, and also from the test
objectives forward to the test results.

Testing Process
Contents
1.1 Introduction
1.2 Test Planning, Monitoring and Control
1.3 Test Analysis
1.4 Test Design
1.5 Test Implementation
1.6 Test Execution
1.7 Evaluating Exit Criteria and Reporting
1.8 Test Closure Activities

Testing Process
Evaluating Exit Criteria & Reporting
  Documentation and reporting for test progress monitoring and control are
discussed in detail in Chapter 2.
 From the point of view of the test process, it is important to ensure that effective
processes are in place to provide the source information necessary for evaluating
exit criteria and reporting.
 Definition of the information requirements and methods for collection are part of
test planning, monitoring and control.
 During test analysis, test design, test implementation and test execution, the Test
Manager should ensure that members of the test team responsible for those
activities are providing the information required in an accurate and timely
manner so as to facilitate effective evaluation and reporting.
 The frequency and level of detail required for reporting are dependent on the
project and the organization. This should be negotiated during the test planning
phase and should include consultation with relevant project stakeholders.

Testing Process
Contents
1.1 Introduction
1.2 Test Planning, Monitoring and Control
1.3 Test Analysis
1.4 Test Design
1.5 Test Implementation
1.6 Test Execution
1.7 Evaluating Exit Criteria and Reporting
1.8 Test Closure Activities

Testing Process
Test Closure Activities
Once test execution is determined to be complete, the key outputs should be captured
and either passed to the relevant person or archived. Collectively, these are test closure
activities. Test closure activities fall into four main groups:
1. Test completion check
2. Test artifacts handover
3.Lessons learned
A. Was the user representation in the quality risk analysis sessions a broad enough
cross-section?
B. Were the estimates accurate?
C. What are the trends and the results of cause and effect analysis of the defects?
D. Are there potential process improvement opportunities?
E. Were there any unanticipated variances from the plan that should be
accommodated in future planning?
4. Archiving results
These tasks are important, often missed, and should be explicitly included as part of the
test plan.

Testing Process
Sample Question
1. You are the Test Manager working on a project developing a tourist information
mobile application. The project recently switched to an agile process and test-driven
development. Each development cycle lasts 15 days, with daily builds beginning at day 7.
After day 10, no new features are allowed to be added. The development team is
composed of very experienced team members, who are proud of their work, but not
tolerant of the testing team. The requirements are written down as coarse-grained user
stories like the following one:

US 03-30: Search 1. You are the Test Manage application. The project rec
nearest matching The software depends on existing w ting is done by
hotel developers, while system and user acceptance testing is
As a casual user at the testing team's responsibility. System test in earlier
an unfamiliar development cycles was often blocked due to severe
location I want to failures of newly developed features. Analysis shows that
get information on many of these failures could have been found during unit
the nearest hotel test. Analysis of issues found during production show
matching my that 30% of performance problems were due to
financial and unreliable web services delivered by 3rd party suppliers.
comfort profile Primary test objectives are to mitigate the perceived
best. performance risks and to increase the confidence that no
Priority: High; failures with high severity will occur in user stories with
Estimate: 7 (out of priority>= High. Moreover, upper management
10) demanded for closer cooperation of testers and
developers.
Which of the following test activities and/or work products will achieve the test
objectives best?
Select TWO options.
Answer Set
a. Approval of detailed design specifications by inspections done by the test team
before day 7, when the daily builds begin.
b. Identification of external web services and enforcement of service level
agreements (SLAs) with service provider done by project management and test
management.(Đ)
c. Integration test level plan defined by test manager before each development
cycle and handed over to developers on day 10.
d. Metrics suite for unit testing defined by and reported to test management at day
7.
 e. Automated performance testing of user stories with priority >= High done by
testers during system test with test execution starting on day 10.(Đ)

Testing Process
Sample Question
2.Assume that you are working for an ambitious start-up. They are creating a system that
will provide customized loyalty and rewards programs for small- and medium-sized companies
selling to customers on the Web. These companies enroll themselves on the system's web store.
This allows the companies to create customized buttons, to be placed on their websites, that let
customers enroll in the companies' loyalty and rewards program. Each subsequent purchase
earns points, and both companies and their customers can manage the program; for example,
to determine the number of points required to receive a free product or service.
Your employer's marketing staff are heavily promoting the system, offering aggressive discounts
on the first year's fees to sign up inaugural companies. The marketing materials state that the
service will be highly reliable and extremely fast for companies and their customers.
At this time, the requirements are complete, and development of the software has just begun.
The current schedule will allow companies and their customers to start enrolling in three
months.
Your employer intends to use cloud computing resources to host this service, and to have no
hardware resources other than ordinary office computers for its developers, testers, and other
engineers and managers. Industry-standard web-based application software components will be
used to build the system.
Consider the following risk item that was identified during the quality risk analysis process:
Customized enrollment buttons for a company's website are not assigned the correct
URL for that company's loyalty program.
Assume that you have used traceability to determine the logical test cases that cover this risk
item. Which of the following is a positive logical test that is complete, is correct, and covers this
risk item?
Select ONE option.
Answer Set
a. Click rapidly on company enrollment button to see what happens.
b.Click on URL for our home page; check that home page displays.

c.Click on company d.Click on company enrollment button; verify that you go to our
enrollment button; verify home page
that you go to that
company's enrollment
page.