TESTING VN

Hoàng Liên Sơn

Skype: hoangliensonmt
Email: [email protected]
http://www.istqb.vn
http://www.istqb.vn
 TESTING VN

Agile Tester
Foundation Level Extension
(version 2014)

http://www.istqb.vn
http://www.istqb.vn
What is Agile Tester Extension?

http://www.istqb.vn
http://www.istqb.vn
Why Agile Tester Extension?

A tester on an Agile project will work differently than

one working on a traditional project.

Testers must
• understand the values and principles that
underpin Agile projects, and
• how testers are an integral part of a whole-team
approach together with developers and business
representatives

http://www.istqb.vn
http://www.istqb.vn
Agile Tester Extension - Exam Structure & Fee

● Exam: 40 multiple choice questions

● Pass mark: 65% correct
● Time: 60 minutes
● Exam Fee: $200 (USD)

http://www.istqb.vn
http://www.istqb.vn
Agile Tester Extension - Learning Objectives

Certified agile testers should be able to demonstrate their skills in

the following areas:

• The Fundamentals of Agile Software Development

• The different agile approaches
• The Differences between Testing in Traditional and Agile Approaches
• Testing in Agile Projects
• Roles and skills of a tester in Agile Projects
• Agile testing techniques and methods
• Assess product quality risks within an Agile project
• Estimate testing effort based on iteration content and quality risks
• Tools in Agile Projects

http://www.istqb.vn
http://www.istqb.vn
Agile Tester Extension - Contents

http://www.istqb.vn
http://www.istqb.vn
Chapter 1: Agile Software Development

Keywords
Agile Manifesto, Agile software development,
incremental development model, test basis,
iterative development model, user story,
software lifecycle, test automation, test oracle,
test-driven development

http://www.istqb.vn
http://www.istqb.vn
1. Agile Software Development

1. Agile Software Development

1. The Fundamentals of Agile Software Development
1. Agile Software Development and the Agile Manifesto
2. Whole-Team Approach
3. Early and Frequent Feedback
2. Aspects of Agile Approaches
1. Agile Software Development Approaches
2. Collaborative User Story Creation
3. Retrospectives
4. Continuous Integration
5. Release and Iteration Planning

http://www.istqb.vn
http://www.istqb.vn
1.1. The Fundamentals of Agile Software Development

What is an Agile Tester?

The members in an Agile project communicate with

each other early and frequently, which helps with
removing defects early and developing a quality
product.

Testers must understand

● values and principles that underpin Agile projects
● how testers are an integral part of a whole-team
A tester on an Agile project will work differently than approach (together with developers and business
one working on a traditional project. representatives.)

http://www.istqb.vn
http://www.istqb.vn
1.1. The Fundamentals of Agile Software Development

http://www.istqb.vn
http://www.istqb.vn
1.1.1. Agile Software Development and the Agile Manifesto - Values

Agile Manifesto
Manifesto contains four statements of values:

Individuals and interactions over processes and tools

Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

The Agile Manifesto argues that although the concepts on the right have value,
those on the left have greater value.

http://www.istqb.vn
http://www.istqb.vn
1.1.1. Agile Software Development and the Agile Manifesto

Individuals and Interactions

Agile development is very people-centered.

Teams of people build software, and it is through

continuous communication and interaction, rather than a reliance on tools or processes,
that teams can work most effectively.

http://www.istqb.vn
http://www.istqb.vn
1.1.1. Agile Software Development and the Agile Manifesto

Working Software

From a customer perspective,

working software is much more useful and valuable than overly detailed documentation and
it provides an opportunity to give the development team rapid feedback.

(it’s especially useful in time-to-market advantage & rapidly changing business environments)

http://www.istqb.vn
http://www.istqb.vn
1.1.1. Agile Software Development and the Agile Manifesto

Customer Collaboration

Collaborating directly with the customer improves the likelihood of

understanding exactly what the customer requires.
While having contracts with customers may be important,
working in regular and close collaboration with them is likely to bring more success to the project.
http://www.istqb.vn
http://www.istqb.vn
1.1.1. Agile Software Development and the Agile Manifesto

Responding to Change

Change is inevitable in software projects.

The environment in which the business operates,

legislation, competitor activity, technology advances,
and other factors can have major influences on the
project and its objectives.

Having flexibility in work practices to embrace change

is more important than
simply adhering rigidly to a plan.

http://www.istqb.vn
http://www.istqb.vn
1.1.1. Agile Software Development and the Agile Manifesto - Principles

The core Agile Manifesto values are captured in twelve principles

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
2. Welcome changing requirements, even late in development. Agile processes harness change for the
customer's competitive advantage.
3. Deliver working software frequently, at intervals of between a few weeks to a few months, with a preference to
the shorter timescale.
4. Business people and developers must work together daily throughout the project.
5. Build projects around motivated individuals. Give them the environment and support they need, and trust
them to get the job done.
6. The most efficient and effective method of conveying information to and within a development team is
face-to-face conversation.
7. Working software is the primary measure of progress.
8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to
maintain a constant pace indefinitely.
9. Continuous attention to technical excellence and good design enhances agility.
10. Simplicity - the art of maximizing the amount of work not done - is essential.
11. The best architectures, requirements, and designs emerge from self-organizing teams.
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior
accordingly.
http://www.istqb.vn
http://www.istqb.vn
1.1.2. Whole-Team Approach

The whole-team approach is supported

through the daily stand-up meetings
involving all members of the team,
where work progress is communicated
and any impediments to progress are
highlighted

http://www.istqb.vn
http://www.istqb.vn
1.1.2. Whole-Team Approach - Properties

http://www.istqb.vn
http://www.istqb.vn
1.1.2. Whole-Team Approach - Properties

The whole-team approach means

involving everyone with the knowledge and skills necessary to ensure project success.

The team includes

representatives from the customer and other business stakeholders
who determine product features. (Product Owner)

The team should be relatively small

successful teams have been observed with as few as three people and as many as nine.

Ideally,
the whole team shares the same workspace and interaction.

The whole-team approach promotes more effective and efficient team dynamics.

http://www.istqb.vn
http://www.istqb.vn
1.1.2. Whole-Team Approach - Quality Responsibilities

The whole team is responsible for quality in Agile projects

Business
● Product Owners / Product Manager
● Subject Matter Experts

Technology
● Architects
● Database Administrators
● User Experience (UX) Designers
● Operations/Support team members

Team
● Developers
● Testers
● Business Analysts

http://www.istqb.vn
http://www.istqb.vn
1.1.2. Whole-Team Approach - QA and Stakeholders

Testers can thus transfer and extend testing The whole team is involved in any consultations
knowledge to other team members and or meetings in which product features are
influence the development of the product presented, analyzed, or estimated

http://www.istqb.vn
http://www.istqb.vn
1.1.2. Whole-Team Approach - QA, DEV and Clients

The concept of involving testers, developers, and business representatives in

all feature discussions is known as the power of three [Crispin08]

http://www.istqb.vn
http://www.istqb.vn
1.1.2. Whole-Team Approach - Benefits

The use of a whole-team approach to product development is one of the main benefits of Agile development.

Enhancing communication and collaboration

within the team
Enabling the various skill sets within the team
to be leveraged to the benefit of the project
Making quality everyone’s responsibility

Its benefits

http://www.istqb.vn
http://www.istqb.vn
1.1.3. Early and Frequent Feedback

Agile projects have short iterations(1) enabling the

project team to receive
early and continuous feedback
on product(2) quality
throughout the development lifecycle.

http://www.istqb.vn
http://www.istqb.vn
1.1.3. Early and Frequent Feedback - Sequential vs Agile Development Approaches

too late

earlier

http://www.istqb.vn
http://www.istqb.vn
1.1.3. Early and Frequent Feedback - Customer First

Early and frequent feedback helps the team focus on the features with the highest business value,
or associated risk, and these are delivered to the customer first
http://www.istqb.vn
http://www.istqb.vn
1.1.3. Early and Frequent Feedback - How to calculate customer’s feedback?

NPS = Net Promoter Score

How likely is it that you would recommend a product to a friend or colleague?

http://www.istqb.vn
http://www.istqb.vn
1.1.3. Early and Frequent Feedback - an NPS Example

NPS = 34.97% - 10.38% = 24.59%

http://www.istqb.vn
http://www.istqb.vn
1.1.3. Early and Frequent Feedback - Benefits of Early and Frequent Feedback

The benefits of early and frequent feedback include:

● Avoiding requirements misunderstandings, which may not have been detected until later in the
development cycle when they are more expensive to fix

● Clarifying customer feature requests, making them available for customer use early. This way,
the product better reflects what the customer wants

● Discovering (via continuous integration), isolating, and resolving quality problems early

● Providing information to the Agile team regarding its productivity and ability to deliver

● Promoting consistent project momentum

http://www.istqb.vn
http://www.istqb.vn
1.2. Aspects of Agile Approaches

Common practices across most Agile organizations

include:

● collaborative user story creation

● retrospectives

● continuous integration

● planning
for each iteration as well as for overall release.
There are a number of Agile approaches
in use by organizations.

http://www.istqb.vn
http://www.istqb.vn
1.2. Aspects of Agile Approaches

1.2. Aspects of Agile Approaches

1. Agile Software Development Approaches
2. Collaborative User Story Creation
3. Retrospectives
4. Continuous Integration
5. Release and Iteration Planning

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches

There are several Agile approaches, each of which implements the values and principles
of the Agile Manifesto in different ways.

In this syllabus, three representatives of Agile approaches are considered:

1. Extreme Programming (XP)
2. Scrum
3. Kanban

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Extreme Programming (XP)

XP embraces five values to guide development:

● communication
● simplicity
● feedback
● courage, and
● respect.

XP describes a set of principles as additional guidelines:

● Humanity & economics
● Mutual benefit & self-similarity
● Improvement & diversity
● Reflection & flow
● Opportunity & redundancy
● Failure & quality
● Baby steps & accepted responsibility
Extreme Programming (XP),
originally introduced by Kent Beck [Beck04]
http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Extreme Programming (XP)

XP describes thirteen primary practices:

1. sit together, 7. weekly cycle,
2. whole team, 8. quarterly cycle,
3. informative workspace 9. slack,
4. energized work, 10. ten-minute build,
5. pair programming, 11. continuous integration
6. stories 12. test first programming,
13. incremental design.

Many of the Agile software development approaches in

use today are influenced by XP and its values and
principles. For example, Agile teams following Scrum often
Extreme Programming (XP), incorporate XP practices.
originally introduced by Kent Beck [Beck04]
http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum vs XP

As opposed to XP, Scrum does NOT

1. dictate specific software development techniques (e.g., test first programming).

2. provide guidance on how testing has to be done in a Scrum project.

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - Roles

Scrum defines three roles:

1. Scrum Master
2. Product Owner
3. Development Team

http://www.istqb.vn
http://www.istqb.vn
 1.2.1. Agile Software Development Approaches - Scrum - Roles

Development Team
Develop and test the product.
The team is self-organized: Scrum Master
There is no team lead, so the Ensures that Scrum practices
team makes the decisions. The and rules are implemented and
team is also cross-functional followed, and resolves any
(Section 2.3.2 & 3.1.4). violations, resource issues, or
other impediments that could
prevent the team from
following the practices and
rules.
Product Owner
Represents the customer, and This person is
generates, maintains, and not the team lead, but a coach.
prioritizes the product backlog.
This person is not the team lead.

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - instrument & practices

Scrum is an Agile management framework which contains the following

constituent instruments and practices [Schwaber01]:
1. Product Backlog
2. Sprint Backlog
3. Sprint
4. Definition of Done
5. Product Increment
6. Time-boxing
7. Transparency
http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - instrument & practices

Product Backlog: The product owner manages a prioritized list of planned product items (called the product
backlog). The product backlog evolves from sprint to sprint (called backlog refinement - grooming).

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - instrument & practices

Sprint: Scrum divides a project into iterations (called sprints) of fixed length (usually two to four weeks).

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - instrument & practices

Product Increment: Each sprint results in a potentially releasable/shippable product (called an increment).

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - instrument & practices

Sprint Backlog: At the start of each sprint, the Scrum team selects a set of highest priority items (called the sprint
backlog) from the product backlog. Since the Scrum team, not the product owner, selects the items to be realized
within the sprint, the selection is referred to as being on the pull principle rather than the push principle.

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - instrument & practices

Definition of Done
To make sure that there is a potentially
releasable product at each sprint’s end, the
Scrum team discusses and defines
appropriate criteria for sprint completion.
The discussion deepens the team’s
understanding of the backlog items and the
product requirements.

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - instrument & practices

Timeboxing
Only those tasks, requirements, or features that the team expects to finish within the sprint are part of the sprint
backlog. If the development team cannot finish a task within a sprint, the associated product features are removed
from the sprint and the task is moved back into the product backlog. Timeboxing applies not only to tasks, but in other
situations (e.g., enforcing meeting start and end times).

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Scrum - instrument & practices

Transparency

The development team reports and updates sprint status on a daily basis at a meeting called the daily
scrum. This makes the content and progress of the current sprint, including test results, visible to the
team, management, and all interested parties. For example, the development team can show sprint
status on a whiteboard.
http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Kanban

http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Kanban

Kanban features some

similarities to Scrum. In both
frameworks, visualizing the
active tasks
(e.g., on a public whiteboard)
provides transparency of
content and progress of tasks.

Tasks not yet scheduled are

waiting in a backlog and moved
onto the Kanban board as soon
as there is new space
(production capacity) available.
http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Kanban

Kanban

Kanban [Anderson13] is a management approach that is sometimes used in Agile projects. The
general objective is to visualize and optimize the flow of work within a value-added chain. Kanban
utilizes three instruments [Linz14]:

Kanban Board: The value chain to be managed is visualized by a Kanban board. Each column
shows a station, which is a set of related activities, e.g., development or testing. The items to be
produced or tasks to be processed are symbolized by tickets moving from left to right across the
board through the stations.

Work-in-Progress Limit: The amount of parallel active tasks is strictly limited. This is controlled by
the maximum number of tickets allowed for a station and/or globally for the board. Whenever a
station has free capacity, the worker pulls a ticket from the predecessor station.

Lead Time: Kanban is used to optimize the continuous flow of tasks by minimizing the (average)
lead time for the complete value stream.
http://www.istqb.vn
http://www.istqb.vn
1.2.1. Agile Software Development Approaches - Kanban

● Iterations or sprints are optional in Kanban.

● The Kanban process allows releasing its deliverables item by item, rather than
as part of a release.
● Time-boxing as a synchronizing mechanism, therefore, is optional, unlike in
Scrum, which synchronizes all tasks within a sprint.

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - Why project failure?

Poor specifications are often a major reason for project failure.

Specification problems can result from the users’ lack of

insight into their
● true needs
● absence of a global vision for the system
● redundant or contradictory features
● miscommunications

http://www.istqb.vn
http://www.istqb.vn project failure
1.2.2. Collaborative User Story Creation - Sequential vs Agile development

In sequential development In Agile development

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - User Stories

In Agile development,
user stories are written to capture requirements from the perspectives of
developers, testers, and business representatives

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - User Story Examples

acceptance criteria

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - User Story Examples

Acceptance Criteria (AC) = Definition of Done (DoD)

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - User Story Examples

a non-functional user story

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - Testers and User Stories

asking
business missing details or
proposing non-functional
ways to test representatives
requirements.

confirming the
acceptance WHAT
criteria

HOW

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - using Techniques

The collaborative authorship of the user story can use techniques such as
brainstorming and mind mapping.

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - INVEST Technique

The tester may use the INVEST technique

● Independent
● Negotiable
● Valuable

● Estimable
● Small
● Testable

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation

According to the 3C concept [Jeffries00], a user story Confirmation

is the conjunction of three elements:

Card

tion
Conversa

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - Card

Card

Regardless of the approach taken to

document user stories,
documentation should be concise,
sufficient, and necessary.

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - Conversation

tion
Conversa

Testers bring valuable input to the

● exchange of thoughts
● opinions
● experiences

The conversation explains how the software will be used.

http://www.istqb.vn
http://www.istqb.vn
1.2.2. Collaborative User Story Creation - Confirmation

Confirmation During confirmation, various participants

play the role of a tester.
These can include developers as well as
specialists focused on:
● performance
● security
● interoperability
● other quality characteristics.

Both positive and negative tests

The acceptance criteria are used to confirm that
should be used to cover the criteria.
the story is done.

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives - What is it?

In Agile development, a retrospective is a

meeting held at the end of each iteration
to discuss:
● what was successful
● what could be improved
● how to incorporate the improvements,
and
● retain the successes in future
iterations

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives - Topics?

Retrospectives cover topics such as

• the process
• people
• organizations
• relationships
• and tools

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives - Continuous Improvement

Retrospectives can result in test-related

improvement decisions focused on
● test effectiveness
● test productivity
● test case quality, and
● team satisfaction.

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives - Address the Testability

They may also address the testability of

● the applications
● user stories
● features, or
● system interfaces.

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives - Root Causes

Root cause analysis of

defects can drive testing
and development
improvements.

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives - Timing and Organization

The timing and organization of the retrospective

depends on the particular Agile method followed.

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives - Timing and Organization

Testers Non-testers

All team members, testers and non-testers, can provide input on

both testing and non-testing activities.

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives - Attributes of a Successful

Retrospectives must occur within a professional environment characterized by

mutual trust.

The attributes of a successful retrospective are the same as those for any other
review as is discussed in the Foundation Level syllabus [ISTQB_FL_SYL],
Section 3.2.

http://www.istqb.vn
http://www.istqb.vn
1.2.3. Retrospectives

• A retrospective is a meeting held at the end of each iteration to discuss

what was successful, what could be improved, and how to incorporate the
improvements and retain the successes in future iterations
• Retrospectives can result in test-related improvement decisions focused
on test effectiveness, test productivity, test case quality, and team
satisfaction. They may also address the testability of the applications, user
stories, features, or system interfaces
• Business representatives and the team attend each retrospective as
participants while the facilitator organizes and runs the meeting
• Testers should play an important role in the retrospectives. All team
members, testers and non-testers, can provide input on both testing and
non-testing activities
http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration

Delivery of a product increment requires reliable, working, integrated software

at the end of every sprint.

http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - Defects are Detected more Quickly

reliable, working, integrated software

Continuous integration addresses this challenge

by merging all changes made to the software
and integrating all changed components
regularly, at least once a day.

Since developers integrate their work constantly,

build constantly, and test constantly,
Configuration management, compilation,
=> defects in code are detected more quickly. software build, deployment, and testing
are wrapped into a
single, automated, repeatable process.
http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - Continuous Integration Process

Posting the
A continuous integration process consists of the status of
following automated activities: all these activities
to a publicly
visible location or
Executing the e-mailing status
integration tests to the team
Executing Installing and
the unit tests, the build
checking
reporting results Report
into a
code coverage test environment Integration (dashboard)
Compiling and and
linking the code, reporting test results Test
Executing static generating the Deploy
code analysis and executable files
reporting results
Unit Test
Compile
Static Code
Analysis
http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - A Dashboard Example

http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - Daily Build Running
Automated
regression tests cover
as much functionality
as possible

daily

• new features
• implemented changes
• confirmation testing
of defect fixes
These test results are visible to all team members

The Agile testers are

freed to concentrate their
manual testing on
An automated build and test process takes place on a daily basis and
detects integration errors early and quickly.
http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - Quality Control

In addition to automated tests, organizations using continuous integration

typically use build tools to implement continuous quality control.
In addition to running unit and integration tests such tools can run
● additional static and dynamic tests
● measure and profile performance
● extract and format documentation from the source code, and
● facilitate manual quality assurance processes

http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - A Review Tool Example

http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - Build and Deployment

This reduces the errors and delays associated

with relying on specialized staff or programmers
to install releases in these environments.

http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - Benefits

Continuous integration can provide the following benefits:

● Allows earlier detection and easier root cause analysis of integration problems and conflicting changes
● Gives the development team regular feedback on whether the code is working
● Keeps the version of the software being tested within a day of the version being developed
● Reduces regression risk associated with developer code refactoring due to rapid re-testing of the code base
after each small set of changes
● Provides confidence that each day’s development work is based on a solid foundation
● Makes progress toward the completion of the product increment visible, encouraging developers and testers
● Eliminates the schedule risks associated with big-bang integration
● Provides constant availability of executable software throughout the sprint for testing, demonstration, or
education purposes
● Reduces repetitive manual testing activities
● Provides quick feedback on decisions made to improve quality and tests
http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - Risks and Challenges

However, continuous integration is not without its risks and challenges:

● Continuous integration tools have to be introduced and maintained
● The continuous integration process must be defined and established
● Test automation requires additional resources and can be complex to establish
● Thorough test coverage is essential to achieve automated testing advantages
● Teams sometimes over-rely on unit tests and perform too little system and
acceptance testing

http://www.istqb.vn
http://www.istqb.vn
1.2.4. Continuous Integration - Need a Testing Tool

Continuous integration requires the use of tools:

● including tools for testing
● tools for automating the build process
● tools for version control

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Release Planning

For Agile lifecycles, two kinds of planning occur:

● release planning, and
● iteration planning.

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Release Planning

Release and iteration planning should address test planning as well as planning for development
activities. Particular test-related issues to address include:
● The scope of testing, the extent of testing for those areas in scope, the test goals, and the
reasons for these decisions.
● The team members who will carry out the test activities.
● The test environment and test data needed, when they are needed, and whether any additions
or changes to the test environment and/or data will occur prior to or during the project.
● The timing, sequencing, dependencies, and prerequisites for the functional and non-functional
test activities (e.g., how frequently to run regression tests, which features depend on other
features or test data, etc.), including how the test activities relate to and depend on
development activities.
● The project and quality risks to be addressed (see Section 3.2.1).

In addition, the larger team estimation effort should include consideration of the time and effort
needed to complete the required testing activities.

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - overall

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning – Major Activities in Test Planning

There are some test planning activities that mentioned in the ISTQB CTFL:
● Determining the scope and risks and identifying the objectives of testing
● Defining the overall approach of testing, including the definition of the test levels and entry and exit criteria
● Integrating and coordinating the testing activities into the software life cycle activities
● Making decisions about what to test, what roles will perform the test activities, how the test activities should be
done, and how the test results will be evaluated
● Scheduling test analysis and design activities
● Scheduling test implementation, execution and evaluation
● Assigning resources for the different activities defined
● Defining the amount, level of detail, structure and templates for the test documentation
● Selecting metrics for monitoring and controlling test preparation and execution, defect resolution and risk issues
● Setting the level of detail for test procedures in order to provide enough information to support reproducible test
preparation and execution
http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Release Planning

Release plans are high-level.

Release planning provides the basis for a test
approach and test plan spanning all iterations.

In release planning, business representatives

establish and prioritize the user stories for
the release, in collaboration with the team
(see Section 1.2.2).
Based on these user stories, project and
quality risks are identified and a high-level
effort estimation is performed
(see Section 3.2).

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Release Planning

Testers are involved in release planning and especially add value in the following
activities:
● Defining testable user stories, including acceptance criteria
● Participating in project and quality risk analyses
● Estimating testing effort associated with the user stories
● Defining the necessary test levels
● Planning the testing for the release

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Iteration Planning

After release planning is done,

iteration planning for
the first iteration starts.

Iteration planning looks ahead to the end of a single iteration and is concerned with the iteration
backlog.
http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Iteration Planning

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Iteration Planning

In iteration planning, the team selects user stories from the

prioritized release backlog, elaborates the user stories,
performs a risk analysis for the user stories, and estimates
the work needed for each user story.

If a user story is too vague and attempts to clarify it have

failed, the team can refuse to accept it and use the next
user story based on priority.

The business representatives must answer the team’s

questions about each story so the team can understand what
they should implement and how to test each story.

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Iteration Planning

Testers are involved in iteration planning and especially add value in the following activities:
● Participating in the detailed risk analysis of user stories
● Determining the testability of the user stories
● Creating acceptance tests for the user stories
● Breaking down user stories into tasks (particularly testing tasks)
● Estimating testing effort for all testing tasks
● Identifying functional and non-functional aspects of the system to be tested
● Supporting and participating in test automation at multiple levels of testing

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Iteration Planning

The number of stories selected is based on established team

velocity and the estimated size of the selected user stories.
After the contents of the iteration are finalized, the user stories
are broken into tasks, which will be carried out by the
appropriate team members.

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Iteration Planning
Velocity Chart
Optimistic

Average
Pessimistic

• Velocity is a metric help us measure how much work the team is able to do, based on the number of user stories
completed in past iterations
• Provides a way to communicate what we have accomplished, what we will likely be able to accomplish, and when
we expect the project or release to be done
• Velocity is measured in whatever units the team users for its work. It often is story points
http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Changing

These changes may be triggered by internal or

external factors. In addition, iteration plans may
change during an iteration.

For example, a particular user story that was

considered relatively simple during estimation might
prove more complex than expected.

Release plans may change as the project proceeds,

including changes to individual user stories in the
product backlog.

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Changing Factors

http://www.istqb.vn
http://www.istqb.vn
1.2.5. Release and Iteration Planning - Changes vs challenges

These changes can be challenging for testers.

Testers must understand the big picture of the release for

test planning purposes, and they must have an adequate
test basis and test oracle in each iteration for test
development purposes as discussed in the Foundation Level
syllabus, Section 1.4.

The required information must be available to the tester

early, and yet change must be embraced according to
Agile principles. This dilemma requires careful decisions
about test strategies and test documentation.

http://www.istqb.vn
http://www.istqb.vn
Chapter 2: Fundamental Agile Testing Principles, Practices, and Processes

Keywords
build verification test, configuration item,
configuration management

http://www.istqb.vn
http://www.istqb.vn
Chapter 2: Fundamental Agile Testing Principles, Practices, and Processes

2.1 The Differences between Testing in Traditional and Agile Approaches

2.1.1 Testing and Development Activities
2.1.2 Project Work Products
2.1.3 Test Levels
2.1.4 Testing and Configuration Management
2.1.5 Organizational Options for Independent Testing

2.2 Status of Testing in Agile Projects

2.2.1 Communicating Test Status, Progress, and Product Quality
2.2.2 Managing Regression Risk with Evolving Manual and Automated Test Cases

2.3 Role and Skills of a Tester in an Agile Team

2.3.1 Agile Tester Skills
2.3.2 The Role of a Tester in an Agile Team

http://www.istqb.vn
http://www.istqb.vn
2.1. The Differences between Testing in Traditional and Agile Approaches

2.1 The Differences between Testing in Traditional and Agile Approaches

2.1.1 Testing and Development Activities
2.1.2 Project Work Products
2.1.3 Test Levels
2.1.4 Testing and Configuration Management
2.1.5 Organizational Options for Independent Testing

http://www.istqb.vn
http://www.istqb.vn
2.1. The Differences between Testing in Traditional and Agile Approaches

Overview

• Test activities are related to development activities => testing varies in different lifecycles
• Differences between testing in traditional lifecycle models (V-model vs RUP) and Agile lifecycles
• The Agile models differ in terms of the way testing and development activities are integrated

Testers should remember that organizations vary considerably in their implementation of lifecycles:
• Deviation from the ideals of Agile lifecycles (Section 1.1)
• The ability to adapt to the context of a given project is a key success factor for testers.

http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities

Testing tends to happen towards the end of the project life cycle.

Development Bugfix Traditional Processes

Testing

Time

Release

http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities

Testing activities occur throughout the iteration, not as a final activity.

Agile

Development / Bugfix

Testing

Time Release

These iterations are highly dynamic, with development, integration, and testing activities taking place
throughout each iteration, and with considerable parallelism and overlap.

http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities

Testers, developers, and business stakeholders all have a role in testing, as with traditional lifecycles.

• Developers perform unit tests as they develop features from the user stories.
• Testers then test those features.
• Business stakeholders (P.O) also test the stories during implementation.
Business stakeholders might:

v use written test cases, but they also might simply experiment with

bz
.s
v use the feature in order to provide fast feedback to the development team

tak
s
ter

eh
tes
Power

old
of three

ers
developers
http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities

iteration 1 iteration 2 iteration 3 iteration n

Development Development / Bugfix Development / Bugfix Development / Bugfix

Testing … Testing

Timeline

The best practice is that no feature is Addressing defects remaining from the previous
considered done until it has been iteration at the beginning of the next iteration,
integrated and tested with the system. as part of the backlog for that iteration
(referred to as “fix bugs first”).

http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities

In Agile, Risk-based testing is used as one of the test strategies,

a high-level risk analysis occurs during release planning,
with testers often driving that analysis.

A specific quality risks associated with each iteration

are identified and assessed in iteration planning.

http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities

In some Agile practices (e.g., Extreme Programming), pairing is used.

Pairing can be:
• testers working together in twos to test a feature
• a tester working collaboratively with a developer to develop and test a feature
• difficult when the test team is distributed, but processes and tools can help
enable distributed pairing

Testers may also: testing

• serve as testing and quality coaches within the team, coach
• sharing testing knowledge and supporting quality assurance work
within the team.

This promotes a sense of collective ownership of quality of the product.

http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities

Test automation at all levels of testing occurs in many Agile teams, and this
can mean that testers spend time:
• creating,
• executing,
• monitoring, and
• maintaining automated tests and results.

Because of the heavy use of test automation, a higher percentage of the manual testing on Agile projects
tends to be done using experience-based and defect-based techniques such as
• software attacks,
• exploratory testing, and
• error guessing.

tendency for Agile teams

• developers will focus on creating unit tests, to favor testers with a strong
• testers should focus on creating automated integration, system,
technical and test
and system integration tests.
automation background.
http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities

One core Agile principle is that change may occur throughout the project.
Therefore, lightweight work product documentation is favored in Agile projects.

Changes to existing features have testing implications,

especially regression testing implications.

It’s important that the rate of change not exceed the

project team’s ability to deal with the risks
associated with those changes.

http://www.istqb.vn
http://www.istqb.vn
2.1.1. Testing and Development Activities - Summary

One of the main differences between traditional lifecycles and Agile lifecycles is
the idea of very short iterations,
each iteration resulting in working software
that delivers features of value to business stakeholders.
• Testing activities occur throughout the iteration, not as a final activity
• Testers, developers, and business stakeholders all have a role in testing
• In some cases, hardening or stabilization iterations occur periodically to resolve any lingering defects
and other forms of technical debt
• The best practice:
Ø no feature is considered done until it has been integrated and tested with the system
Ø to address defects remaining from the previous iteration at the beginning of the next iteration
• A high-level risk analysis occurs during release planning, with testers often driving that analysis
• Pairing can involve testers working together in twos or with developers to test a feature.
• Testers may also serve as testing and quality coaches within the team, sharing testing knowledge
• Test automation at all levels of testing occurs in many Agile teams
• The lightweight work product documentation is favored in Agile projects.
http://www.istqb.vn
http://www.istqb.vn
2.1. The Differences between Testing in Traditional and Agile Approaches

2.1 The Differences between Testing in Traditional and Agile Approaches

2.1.1 Testing and Development Activities
2.1.2 Project Work Products
2.1.3 Test Levels
2.1.4 Testing and Configuration Management
2.1.5 Organizational Options for Independent Testing

http://www.istqb.vn
http://www.istqb.vn
2.1.2. Project Work Products - Overview

In a typical Agile project,

• It is a common practice to avoid producing vast amounts of documentation.

• Instead, focus is more on having working software,
together with automated tests that demonstrate conformance to requirements.

=> This encouragement to reduce documentation applies

only to documentation that does not deliver value to the customer.

In a successful Agile project,

• A balance is struck between increasing efficiency by reducing documentation and
providing sufficient documentation to support business, testing, development, and
maintenance activities.
• The team must make a decision during release planning about which work products
are required and what level of work product documentation is needed.

http://www.istqb.vn
http://www.istqb.vn
2.1.2. Project Work Products – Work Product Categories

Project work products of immediate interest to Agile testers typically fall into three categories:

1. Business-oriented work products that

• describe what is needed (e.g., requirements specifications) and
• how to use it (e.g., user documentation)

2. Development work products that

• describe how the system is built (e.g., database entity-relationship diagrams),
• that actually implement the system (e.g., code), or
• that evaluate individual pieces of code (e.g., automated unit tests)

3. Test work products that

• describe how the system is tested (e.g., test strategies and plans),
• that actually test the system (e.g., manual and automated tests), or
• that present test results (e.g., test dashboards – section 2.2.1)

http://www.istqb.vn
http://www.istqb.vn
2.1.2. Project Work Products – Business-oriented Work Products

Typical business-oriented work products on Agile projects include user stories & acceptance criteria.

A user story should define a feature small enough to be completed in a single iteration.
User stories are:
• the Agile form of requirements specifications, and
• should explain how the system should behave with respect to a single, coherent feature or function.

=> Larger collections of related features, or a collection of sub-features that make up a single
complex feature, may be referred to as “epics”.

• An epic may be developed over a series of sprints.

• Each epic and its user stories should have associated acceptance criteria.

http://www.istqb.vn
http://www.istqb.vn
2.1.2. Project Work Products – Developer Work Products

Typical developer work products on Agile projects include code.

Agile developers also often create automated unit tests.

These tests might be created after or before the development of code.

This approach is referred to as test first or test-driven development.

In reality, the tests are more a form of executable low-level design specifications rather than tests.

http://www.istqb.vn
http://www.istqb.vn
2.1.2. Project Work Products – Tester Work Products

Typical tester work products on Agile projects include automated tests,

as well as documents such as:
• test plans,
• quality risk catalogs,
• manual tests,
• defect reports, and
• test results logs.

The documents are captured in as lightweight a fashion as possible,

which is often also true of these documents in traditional lifecycles.
Testers will also produce test metrics from defect reports and test results logs,
and again there is an emphasis on a lightweight approach.

http://www.istqb.vn
http://www.istqb.vn
2.1.2. Project Work Products – Formal work products

In some Agile implementations,

especially regulated, safety critical, distributed, or highly complex projects and products,
further formalization of these work products is required.

For example, some teams transform user stories and acceptance criteria into more formal requirements
specifications.

Vertical and horizontal traceability reports may be prepared to satisfy auditors, regulations, and other
requirements.

http://www.istqb.vn
http://www.istqb.vn
2.1. The Differences between Testing in Traditional and Agile Approaches

2.1 The Differences between Testing in Traditional and Agile Approaches

2.1.1 Testing and Development Activities
2.1.2 Project Work Products
2.1.3 Test Levels
2.1.4 Testing and Configuration Management
2.1.5 Organizational Options for Independent Testing

http://www.istqb.vn
http://www.istqb.vn
2.1.3. Test Levels – Sequential vs Agile Life Cycles

Test levels are test activities that are logically related,

often by the maturity or completeness of the item under test.

In sequential lifecycle models,

the test levels are often defined such that the exit criteria of one level are part of the entry criteria for
the next level.

In some iterative models (ex, RUP), this rule does not apply. Test levels overlap.
Requirement specification, design specification, and development activities may overlap with test levels.

In some Agile lifecycles, overlap occurs because changes to requirements, design, and code can
happen at any point in an iteration.

http://www.istqb.vn
http://www.istqb.vn
2.1.3. Test Levels – Unit vs Acceptance Testing

While Scrum,
in theory, does not allow changes to the user stories after iteration planning,
in practice such changes sometimes occur.

During an iteration, any given user story will typically progress sequentially through
the following test activities:
1. Unit testing, typically done by the developer
2. Feature acceptance testing, which is sometimes broken into two activities:
• Feature verification testing, which is often automated, may be done by developers or testers, and
involves testing against the user story’s acceptance criteria
• Feature validation testing, which is usually manual and can involve developers, testers, and
business stakeholders working collaboratively to
ü determine whether the feature is fit for use,
ü improve visibility of the progress made, and
ü receive real feedback from the business stakeholders
http://www.istqb.vn
http://www.istqb.vn
2.1.3. Test Levels – Regression vs System Testing

There is often a parallel process of regression testing

This involves re-running the automated unit tests & feature verification tests from
the current & previous iterations,
(usually via a continuous integration framework.)

In some Agile projects, there may be a system test

level, which starts once the first user story is ready for
such testing.
This can involve executing functional tests, as well as
non-functional tests for:
• performance,
• reliability,
• usability, and
• other relevant test types.

http://www.istqb.vn
http://www.istqb.vn
2.1.3. Test Levels – Acceptance Testing

Agile teams can employ various forms of acceptance testing:

• Internal alpha tests and
• external beta tests may occur,
either at the close of each iteration, after the completion of each iteration, or after a series of iterations.

• User acceptance tests,

• operational acceptance tests,
• regulatory acceptance tests, and
• contract acceptance tests also may occur,
either at the close of each iteration, after the completion of each iteration, or after a series of iterations.

http://www.istqb.vn
http://www.istqb.vn
2.1.3. Test Levels – Test Automation Pyramid

Traditional/ Agile
Outsourcing Projects
Projects

http://www.istqb.vn
http://www.istqb.vn
2.1. The Differences between Testing in Traditional and Agile Approaches

2.1 The Differences between Testing in Traditional and Agile Approaches

2.1.1 Testing and Development Activities
2.1.2 Project Work Products
2.1.3 Test Levels
2.1.4 Testing and Configuration Management
2.1.5 Organizational Options for Independent Testing

http://www.istqb.vn
http://www.istqb.vn
2.1.4. Testing and Configuration Management

Agile projects often involve heavy use of automated tools to Developers use tools for
• develop, • static analysis,
• test, and • unit testing, and
• manage • code coverage.
software development.

Developers continuously check the code and unit tests into a configuration management system,
using automated build and test frameworks.
These frameworks allow the continuous integration of new software with the system, with the static
analysis and unit tests run repeatedly as new software is checked in.

• These automated tests can also include functional tests at the integration and system levels.
• In some cases, due to the duration of the functional tests, the functional tests are separated from the
unit tests and run less frequently.
For example, unit tests may be run each time new software is checked in,
while the longer functional tests are run only every few days.

http://www.istqb.vn
http://www.istqb.vn
2.1.4. Testing and Configuration Management

v One goal of the automated tests is to confirm that the build is functioning and installable.
v If any automated test fails, the team should fix the underlying defect in time for the next code check-in.
v This requires an investment in real-time test reporting to provide good visibility into test results.

This approach helps reduce expensive and inefficient cycles of “build-install-fail-rebuild-reinstall”

that can occur in many traditional projects, since changes that break the build or cause software to fail
to install are detected quickly.

Automated testing and build tools help to manage the regression risk associated
with the frequent change that often occurs in Agile projects.
However, over-reliance on automated unit testing alone to manage these risks
can be a problem, as unit testing often has limited defect detection effectiveness.
Automated tests at the integration and system levels are also required.

http://www.istqb.vn
http://www.istqb.vn
2.1. The Differences between Testing in Traditional and Agile Approaches

2.1 The Differences between Testing in Traditional and Agile Approaches

2.1.1 Testing and Development Activities
2.1.2 Project Work Products
2.1.3 Test Levels
2.1.4 Testing and Configuration Management
2.1.5 Organizational Options for Independent Testing

http://www.istqb.vn
http://www.istqb.vn
2.1.5. Organizational Options for Independent Testing

As discussed in the Foundation Level syllabus,

independent testers are often more effective at finding defects.

In some Agile teams,

developers create many of the tests in the form of automated tests.
=> one or more testers may be embedded within the team, performing many of the testing tasks.
However, given those testers’ position within the team,
=> there is a risk of loss of independence and objective evaluation.

Other Agile teams retain fully independent, separate test teams, and assign testers on-demand
during the final days of each sprint.
This can preserve independence, and these testers can provide an objective, unbiased evaluation
of the software.
However, time pressures, lack of understanding of the new features in the product,
and relationship issues with business stakeholders and developers often lead to problems
with this approach.

http://www.istqb.vn
http://www.istqb.vn
2.1.5. Organizational Options for Independent Testing

A third option is to have an independent,

separate test team where testers are assigned to Agile teams on a long-term basis, at the
beginning of the project, allowing them to maintain their independence while gaining a good
understanding of the product and strong relationships with other team members.

In addition, the independent test team can have specialized testers outside of the Agile teams to
work on long-term and/or iteration-independent activities, such as
• developing automated test tools,
• carrying out non-functional testing,
• creating and supporting test environments and data, and
• carrying out test levels that might not fit well within a sprint
(e.g., system integration testing).

http://www.istqb.vn
http://www.istqb.vn
2.2. Status of Testing in Agile Projects

2.2 Status of Testing in Agile Projects

2.2.1 Communicating Test Status, Progress, and Product Quality
2.2.2 Managing Regression Risk with Evolving Manual and Automated Test Cases

http://www.istqb.vn
http://www.istqb.vn
2.2. Status of Testing in Agile Projects - Overview

Change takes place rapidly in Agile projects.

This change means that
• test status,
• test progress, and
• product quality constantly evolve,
and testers must devise ways to get that information to the team so that they can make decisions to
stay on track for successful completion of each iteration.

In addition, change can affect existing features from previous iterations.

Therefore, manual and automated tests must be updated to deal effectively with regression risk.

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality

Agile teams progress by having working software at the end of each iteration.

To determine when the team will have working software, they need to monitor the progress of all
work items in the iteration and release.

Testers in Agile teams utilize various methods to record test progress and status, including:
• test automation results,
• progression of test tasks and stories on the Agile task board,
• burndown charts showing the team’s progress.

These can then be communicated to the rest of the team

using media such as:
• wiki dashboards,
• dashboard-style emails,
• verbally during stand-up meetings.

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality

Agile teams may use tools that automatically generate status reports based on test results and
task progress, which in turn update wiki-style dashboards and emails.

This method of communication also gathers metrics from the testing process,
which can be used in process improvement.

Communicating test status in such an automated

manner
also frees testers’ time to focus on
designing and executing more test cases.

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality – Burndown Chart

Teams may use burndown charts to track progress across the entire release and within each iteration.

amount of work left to be done

A burndown chart represents

the amount of work left to be done
against time allocated to
the release or iteration.

time allocated to the release or iteration.

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality – Burndown Chart

To provide an instant, detailed visual

representation of the whole team’s current
status, including the status of testing,
teams may use Agile task boards.

The story cards, development tasks, test

tasks, and other tasks are captured on the
task board, often using color-coordinated
cards to determine the task type.

During the iteration, progress is managed

via the movement of these tasks across
the task board into columns such as to do,
work in progress, verify, and done.

a physical task board

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality – Burndown Chart

Agile teams may use tools (ex. JIRA, Trello) to maintain their story cards and Agile task boards,
which can automate dashboards and status updates.
Testing tasks on the task board relate to the acceptance criteria defined for the user stories.
As test automation scripts, manual tests, and exploratory tests for a test task achieve a passing status,
the task moves into the done column of the task board.

story
task

sub-task bug a electronic task board

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality – Burndown Chart

Who update the board?

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality – Burndown Chart

The whole team reviews the status of the task board regularly,
often during the daily stand-up meetings, to ensure tasks are
moving across the board at an acceptable rate.

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality – Burndown Chart

If any tasks (including testing tasks) are not moving or are moving too slowly,
the team reviews and addresses any issues that may be blocking the progress of those tasks.

http://www.istqb.vn
http://www.istqb.vn
2.2.1. Communicating Test Status, Progress, and Product Quality – Burndown Chart

The daily stand-up meeting includes all members of the Agile team
including testers. At this meeting, they communicate their current status.
The agenda for each member is:
v What have you completed since the last meeting?
v What do you plan to complete by the next meeting?
v What is getting in your way?

Any issues that may block test progress are communicated during the daily stand-up meetings, so the
whole team is aware of the issues and can resolve them accordingly.
To improve the overall product quality, many Agile teams perform customer satisfaction surveys to
receive feedback on whether the product meets customer expectations.

Teams may use other metrics similar to those captured in traditional development methodologies, such as:
test pass/fail rates, defect discovery rates, confirmation & regression test results, defect density,
defects found & fixed, requirements coverage, risk coverage, code coverage, and code churn
to improve the product quality. is defined as lines added, modified or deleted
http://www.istqb.vn
http://www.istqb.vn to a file from one version to another.
2.2.2. Managing Regression Risk with Evolving Manual and Automated Test Cases

In an Agile project,
as each iteration completes => the product grows => the scope of testing increases

Along with testing the code changes made in the current

iteration, testers also need to verify no regression has been
introduced on features that were developed and tested in
previous iterations.

The risk of introducing regression in Agile development is

high due to extensive code churn. Since responding to
change is a key Agile principle, changes can also be made to
previously delivered features to meet business needs.

Code churn is defined as lines added, modified or deleted

http://www.istqb.vn
http://www.istqb.vn to a file from one version to another.
2.2.2. Managing Regression Risk with Evolving Manual and Automated Test Cases

Test Assets be Maintained in a Configuration Management Tool

In order to maintain velocity without incurring a large amount of technical debt,

it is critical that teams invest in test automation at all test levels as early as possible.

It is also critical that all test assets such as automated tests,

manual test cases, test data, and other testing artifacts are
kept up-to-date with each iteration.

It is highly recommended that all test assets be maintained

in a configuration management tool in order to
enable version control,
• to ensure ease of access by all team members, and
• to support making changes as required
due to changing functionality while still preserving the historic information
of the test assets.
Technical debt is a concept in software development that reflects the
implied cost of additional rework caused by choosing an easy solution now
http://www.istqb.vn
http://www.istqb.vn instead of using a better approach that would take longer.
2.2.2. Managing Regression Risk with Evolving Manual and Automated Test Cases

Update test cases

Testers need to allocate time in each iteration to review manual and
automated test cases from previous and current iterations to select test
cases that may be candidates for the regression test suite, and to retire
test cases that are no longer relevant.

Tests written in earlier iterations to verify specific features may have little
value in later iterations due to feature changes or new features which
alter the way those earlier features behave.

While reviewing test cases, testers should consider suitability for automation.
The team needs to automate as many tests as possible from previous and current iterations.
• This allows automated regression tests to reduce regression risk with less effort than manual
regression testing would require.
• This reduced regression test effort frees the testers to more thoroughly test new features and
functions in the current iteration.
http://www.istqb.vn
http://www.istqb.vn
2.2.2. Managing Regression Risk with Evolving Manual and Automated Test Cases

Update test cases

It is critical that testers have the ability to quickly identify and

update test cases from previous iterations and/or releases that
are affected by the changes made in the current iteration.

Defining how the team designs, writes, and stores test cases
should occur during release planning.

Good practices for test design and implementation need to be

adopted early and applied consistently.

The shorter timeframes for testing and the constant change in

each iteration will increase the impact of poor test design and
implementation practices.

http://www.istqb.vn
http://www.istqb.vn
2.2.2. Managing Regression Risk with Evolving Manual and Automated Test Cases

Use of test automation, at all test levels, allows Agile teams to provide rapid feedback on product quality.

Automated unit tests

Automated unit tests are run before source code is checked into the mainline (ex. UAT or Master) of the
configuration management system to ensure the code changes do not break the software build.

To reduce build breaks, which can slow down the progress of the whole team, code should not be checked
in unless all automated unit tests pass. Automated unit test results provide immediate feedback on
code and build quality, but not on product quality.

Well-written automated tests provide a living document of system functionality.

By checking the automated tests and their corresponding test results into the
configuration management system, aligned with the versioning of the product builds,
Agile teams can review the functionality tested and the test results for any given build
at any given point in time.

http://www.istqb.vn
http://www.istqb.vn
2.2.2. Managing Regression Risk with Evolving Manual and Automated Test Cases

Automated acceptance tests

Automated acceptance tests are run regularly as part of the continuous integration full system build.
These tests are run against a complete system build at least daily, but are generally not run with each
code check-in as they take longer to run than automated unit tests and could slow down code check-ins.
The test results from automated acceptance tests provide feedback on product quality with respect to
regression since the last build, but they do not provide status of overall product quality.

Automated tests can be run continuously against the system.

An initial subset of automated tests to cover critical system functionality and
integration points should be created immediately after a new build is
deployed into the test environment.
=> These tests are commonly known as build verification tests.

Results from the build verification tests will provide instant feedback on
the software after deployment, so teams don’t waste time testing
an unstable build.
http://www.istqb.vn
http://www.istqb.vn
2.2.2. Managing Regression Risk with Evolving Manual and Automated Test Cases

Automated tests contained in the regression test set are generally run as part of the daily main build
in the continuous integration environment, and again when a new build is deployed into
the test environment.
As soon as an automated regression test fails,
the team stops and investigates the reasons for the failing test.

• The test may have failed due to legitimate functional changes in the current iteration, in which case
the test and/or user story may need to be updated to reflect the new acceptance criteria. Alternatively,
the test may need to be retired if another test has been built to cover the changes.
• However, if the test failed due to a defect, it is a good practice for the team to fix the defect prior to
progressing with new features.

http://www.istqb.vn
http://www.istqb.vn
2.2.2. Managing Regression Risk with Evolving Manual and Automated Test Cases

In addition to test automation, the following testing tasks may also be automated:
• Test data generation
• Loading test data into systems
• Deployment of builds into the test environments
• Restoration of a test environment (e.g., the database or website data files) to a baseline
• Comparison of data outputs

Automation of these tasks reduces the overhead and allows the team
to spend time
developing and testing new features.

http://www.istqb.vn
http://www.istqb.vn
2.3. Role and Skills of a Tester in an Agile Team

2.3 Role and Skills of a Tester in an Agile Team

2.3.1 Agile Tester Skills
2.3.2 The Role of a Tester in an Agile Team

http://www.istqb.vn
http://www.istqb.vn
2.3. Role and Skills of a Tester in an Agile Team

In an Agile team,
testers must closely collaborate with all other team members and
with business stakeholders.

This has a number of implications in terms of the skills a tester must

have and
the activities they perform within an Agile team.

http://www.istqb.vn
http://www.istqb.vn
2.3.1. Agile Test Skills – Additional Skills

Agile testers should have all the skills mentioned in the Foundation Level syllabus.
In addition to these skills, a tester in an Agile team should be competent in:
• test automation,
• test-driven development,
• acceptance test-driven development,
• white-box,
• black-box, and
• experience-based testing.

http://www.istqb.vn
http://www.istqb.vn
2.3.1. Agile Test Skills – Test Methodologies

As Agile methodologies depend heavily on collaboration, communication, and interaction between

the team members as well as stakeholders outside the team, testers in an Agile team should have
good interpersonal skills. Testers in Agile teams should:
• Be positive and solution-oriented with team members and stakeholders
• Display critical, quality-oriented, skeptical thinking about the product
• Actively acquire information from stakeholders (rather than relying entirely on written
specifications)
• Accurately evaluate and report test results, test progress, and product quality
• Work effectively to define testable user stories, especially acceptance criteria, with customer
representatives and stakeholders
• Collaborate within the team, working in pairs with programmers and other team members
• Respond to change quickly, including changing, adding, or improving test cases
• Plan and organize their own work

Continuous skills growth, including interpersonal skills growth, is essential for all
testers, including those on Agile teams http://www.istqb.vn
http://www.istqb.vn
2.3.2. The Role of a Tester in an Agile Team – Roles in a team

http://www.istqb.vn
http://www.istqb.vn
2.3.2. The Role of a Tester in an Agile Team

The role of a tester in an Agile team includes activities that generate and provide feedback not only on
test status, test progress, and product quality, but also on process quality. In addition to the activities
described elsewhere in this syllabus, these activities include:
• Understanding, implementing, and updating the test strategy
• Measuring and reporting test coverage across all applicable coverage dimensions
• Ensuring proper use of testing tools
• Configuring, using, and managing test environments and test data
• Reporting defects and working with the team to resolve them
• Coaching other team members in relevant aspects of testing
• Ensuring the appropriate testing tasks are scheduled during release and iteration planning
• Actively collaborating with developers and business stakeholders to clarify requirements,
especially in terms of testability, consistency, and completeness
• Participating proactively in team retrospectives, suggesting and implementing improvements

http://www.istqb.vn
http://www.istqb.vn
2.3.2. The Role of a Tester in an Agile Team

Within an Agile team, each team member is responsible for product quality and plays a role in
performing test-related tasks.

Agile organizations may encounter some test-related organizational risks:

• Testers work so closely to developers that they lose the appropriate tester mindset
• Testers become tolerant of or silent about inefficient, ineffective, or low-quality practices within
the team
• Testers cannot keep pace with the incoming changes in time-constrained iterations

To mitigate these risks, organizations

may consider variations for preserving
independence discussed in Section 2.1.5.

http://www.istqb.vn
http://www.istqb.vn
Chapter 3: Agile Testing Methods, Techniques, and Tools

Keywords
acceptance criteria, exploratory testing, performance
testing, product risk, quality risk, regression testing,
test approach, test charter, test estimation,
test execution automation, test strategy,
test-driven development (TDD), unit test framework

http://www.istqb.vn
http://www.istqb.vn
3.1. Agile Testing Methods

3.1 Agile Testing Methods

3.1.1 Test-Driven Development, Acceptance Test-Driven Development, & Behavior-Driven Development
3.1.2 The Test Pyramid
3.1.3 Testing Quadrants, Test Levels, and Testing Type
3.1.4 The Role of a Tester

http://www.istqb.vn
http://www.istqb.vn
3.1. Agile Testing Methods – Good Testing Practices

Good testing practices for every development project (agile or not) to produce quality products:
• writing tests in advance to express proper behavior,
• focusing on early defect prevention, detection, and removal, and
• ensuring that the right test types are run at the right time and as part of the right test level

Agile practitioners aim to introduce these practices early.

Testers in Agile projects play a key role in guiding the use of
these testing practices throughout the lifecycle.

http://www.istqb.vn
http://www.istqb.vn
3.1.1. TDD, ATDD and BDD

Each of Test-driven development (TDD), acceptance test-driven development (ATDD), and

behavior-driven development (BDD) is an example of a fundamental principle of testing.

The benefit of early testing and QA activities,

since the tests are defined before the code is written.

TDD: Test-Driven Development

ATDD: Acceptance Test-Driven Development
BDD: Behavior-Driven Development
http://www.istqb.vn
http://www.istqb.vn
3.1.1. TDD – Test-Driven Development

The process for test-driven development (TDD) is:

1. Add a test that captures the programmer’s concept
of the desired functioning of a small piece of code
2. Run the test, which should fail since the code
doesn’t exist
3. Write the code and run the test in a tight loop until
the test passes
4. Refactor the code after the test is passed, re-
running the test to ensure it continues to pass
against the refactored code
5. Repeat this process for the next small piece of
code, running the previous tests as well as the
added tests

http://www.istqb.vn
http://www.istqb.vn
3.1.1. TDD – Test-Driven Development

Notes:
• The tests written are primarily unit level and are code-focused, though tests may also be written
at the integration or system levels.
• Test-driven development gained its popularity through Extreme Programming [Beck02], but is
also used in other Agile methodologies and sometimes in sequential lifecycles.
• It helps developers focus on clearly-defined expected results.
• The tests are automated and are used in continuous integration (CI).

http://www.istqb.vn
http://www.istqb.vn
3.1.1. ATDD – Acceptance Test-Driven Development

Acceptance Test-driven Development (ATDD) defines

acceptance criteria and tests during the creation of
user stories (see Section 1.2.2).

ATDD is a collaborative approach that allows every

stakeholder to understand:
• how the software component has to behave &
• what the developers, testers, and business
representatives need to ensure this behavior.

The process of ATDD is explained in Section 3.3.2.

http://www.istqb.vn
http://www.istqb.vn
3.1.1. ATDD – Acceptance Test-Driven Development

Acceptance test-driven development creates reusable tests for regression testing. Specific tools support
creation and execution of such tests, often within the continuous integration (CI) process.

These tools allow tests to be executed at the system or acceptance level.

Acceptance Test-driven Development allows

quick resolution of defects and validation of feature behavior

It helps determine if the acceptance criteria are met for the feature.

http://www.istqb.vn
http://www.istqb.vn
3.1.1. BDD – Behavior-Driven Development

Behavior-driven development allows a developer to focus on

testing the code based on the expected behavior of the software.

Because the tests are based on the exhibited behavior of the software,
the tests are generally easier for other team members and stakeholders to understand

http://www.istqb.vn
http://www.istqb.vn
3.1.1. BDD – Behavior-Driven Development

Specific behavior-driven development frameworks can be used to define

acceptance criteria based on the given/when/then format:

Given some initial context,

When an event occurs,
Then ensure some outcomes.

• From these requirements, the behavior-driven development framework generates code that can
be used by developers to create test cases.
• Behavior-driven development helps the developer collaborate with other stakeholders, including
testers, to define accurate unit tests focused on business needs.

http://www.istqb.vn
http://www.istqb.vn
3.1.2. The Test Pyramid

A software system may be tested at different levels.

Typical test levels are, from the base of the pyramid to the
top, unit, integration, system, and acceptance.
the number of
tests decreases
Usually unit and integration level tests are
automated and are created using API-based tools.
At the system and acceptance levels,
the automated tests are created using GUI-based tools.
large number
The test pyramid concept is based on the of tests
testing principle of early QA and testing
(i.e., eliminating defects The test pyramid emphasizes having a large number
of tests at the lower levels and,
as early as possible in the lifecycle).
as development moves to the upper levels,
the number of tests decreases.

http://www.istqb.vn
http://www.istqb.vn
3.1.3. Testing Quadrants, Test Levels, and Testing Types

Testing quadrants, defined by Brian Marick, align the test levels with the appropriate test types
in the Agile methodology.

The testing quadrants model, and its variants, helps to

ensure that all important test types and test levels are
included in the development lifecycle.

This model also provides a way to differentiate and

describe the types of tests to all stakeholders,
including:
• developers,
• testers, and
• business representatives.

Marick’s Original

http://www.istqb.vn
http://www.istqb.vn
3.1.3. Testing Quadrants, Test Levels, and Testing Types

In the testing quadrants, tests can be business (user) or technology (developer) facing.
• Some tests support the work done by the Agile team and confirm software behavior.
• Other tests can verify the product.
• Tests can be fully manual, fully automated, a combination of manual and automated, or manual but
supported by tools.

http://www.istqb.vn
http://www.istqb.vn
3.1.3. Testing Quadrants, Test Levels, and Testing Types

Quadrant Q1 is unit level, technology facing, Quadrant Q2 is system level, business facing, and
and supports the developers. confirms product behavior.
This quadrant contains unit tests. These tests This quadrant contains functional tests, examples:
should be automated and included in the story tests, user experience prototypes & simulations.
continuous integration process. • These tests check the acceptance criteria and can
be manual or automated.
• They are often created during the user story
development and thus improve the quality of the
stories.
• They are useful when creating automated regression
test suites.

http://www.istqb.vn
http://www.istqb.vn
3.1.3. Testing Quadrants, Test Levels, and Testing Types

Quadrant Q3 is system or user acceptance level, Quadrant Q4 is system or operational acceptance

business facing, and contains tests that critique the level, technology facing, and contains tests that
product, using realistic scenarios and data. This critique the product. This quadrant contains
quadrant contains ten performance, load, stress, and scalability tests,
are of
• exploratory testing,
s e tests d are security tests, maintainability, memory
• scenarios, The a l an management, compatibility and interoperability,
a n u n t ed.
• process flows, m o rie data migration, infrastructure, and recovery
r -
• usability testing,
use testing. These tests are often automated.
• user acceptance testing,
• alpha testing, and
• beta testing.

During any given iteration, tests from any or all quadrants may be required.
The testing quadrants apply to dynamic testing rather than static testing.

http://www.istqb.vn
http://www.istqb.vn
3.1.4. The Role of a Tester - overview

Throughout this syllabus, general reference has been made to Agile methods and techniques,
and the role of a tester within various Agile lifecycles.

This subsection looks specifically at the role of a tester in a project

following a Scrum lifecycle.

http://www.istqb.vn
http://www.istqb.vn
3.1.4. The Role of a Tester - Teamwork

Teamwork is a fundamental principle in Agile development. Agile emphasizes the whole-team approach.
The following are organizational and behavioral best practices in Scrum teams:
ü Cross-functional: Each team member brings a different set of skills to the team. The team works together on test
strategy, test planning, test specification, test execution, test evaluation, and test results reporting.
ü Self-organizing: The team may consist only of developers, but ideally there would be one or more testers (see 2.1.5).
ü Co-located: Testers sit together with the developers and the product owner.
ü Collaborative: Testers collaborate with their team members, other teams, the stakeholders, the product owner, and
the Scrum Master.
ü Empowered: Technical decisions regarding design and testing are made by the team as a whole (developers, testers,
and Scrum Master), in collaboration with the product owner and other teams if needed.
ü Committed: The tester is committed to question and evaluate the product’s behavior and characteristics with respect
to the expectations and needs of the customers and users.
ü Transparent: Development and testing progress is visible on the Agile task board (see Section 2.2.1).
ü Credible: The tester must ensure the credibility of the strategy for testing, its implementation, and execution, otherwise
the stakeholders will not trust the test results. This is often done by providing information to the stakeholders about the
testing process.
ü Open to feedback: Feedback is an important aspect of being successful in any project, especially in Agile projects.
Retrospectives allow teams to learn from successes and from failures.
ü Resilient: Testing must be able to respond to change, like all other activities in Agile projects.
http://www.istqb.vn
http://www.istqb.vn
3.1.4. The Role of a Tester – Sprint Zero

Sprint Zero is the first iteration of the project where many preparation activities take place (see Section
1.2.5). The tester collaborates with the team on the following activities during this iteration:
Ø Identify the scope of the project (i.e., the product backlog)
Ø Create an initial system architecture and high-level prototypes
Ø Plan, acquire, and install needed tools (e.g., for test management, defect management, test automation,
and continuous integration)
Ø Create an initial test strategy for all test levels, addressing (among other topics) test scope, technical
risks, test types (see Section 3.1.3), and coverage goals
Ø Perform an initial quality risk analysis (see Section 3.2.1)
Ø Define test metrics to measure the test process, the progress of testing in the project, and product quality
Ø Specify the definition of “done”
Ø Create the task board (see Section 2.2.1)
Ø Define when to continue or stop testing before delivering the system to the customer

Sprint zero sets the direction for

what testing needs to achieve and how testing needs to achieve it
throughout the sprints.
http://www.istqb.vn
http://www.istqb.vn
3.1.4. The Role of a Tester – Integration & Test Planning

Integration
In Agile projects, the objective is to deliver customer value on a continuous basis (preferably in every sprint).
To enable this, the integration strategy should consider both design and testing.
To enable a continuous testing strategy for the delivered functionality and characteristics,
it is important to identify all dependencies between underlying functions and features.

Test Planning
Since testing is fully integrated into the Agile team, test planning should start during the release planning
session and be updated during each sprint. (Test planning for the release and each sprint should address
the issues discussed in Section 1.2.5.)
Sprint planning results in a set of tasks to put on the task board, where each task should have a length of
one or two days of work.
In addition, any testing issues should be tracked to keep a steady flow of testing.

http://www.istqb.vn
http://www.istqb.vn
3.1.4. The Role of a Tester – Agile Testing Practices

Many Agile Testing Practices may be useful for testers in a scrum team, some of which include:
Ø Pairing: Two team members (e.g., a tester and a developer, two testers, or a tester and a product
owner) sit together at one workstation to perform a testing or other sprint task.
Ø Incremental test design: Test cases and charters are gradually built from user stories and other test
bases, starting with simple tests and moving toward more complex ones.
Ø Mind mapping: Mind mapping is a useful tool when testing. For example, testers can use mind
mapping to identify which test sessions to perform, to show test strategies, and to describe test data.

These practices are in addition to other practices discussed in

this syllabus and in
Chapter 4 of the Foundation Level syllabus.

http://www.istqb.vn
http://www.istqb.vn
3.2. Assessing Quality Risks and Estimating Test Effort

3.2 Assessing Quality Risks and Estimating Test Effort

3.2.1 Assessing Quality Risks in Agile Projects
3.2.2 Estimating Testing Effort Based on Content and Risk

http://www.istqb.vn
http://www.istqb.vn
3.2. Assessing Quality Risks and Estimating Test Effort - Overview

What is the typical objective of testing in all projects?

A typical objective of testing in all projects, Agile or traditional, is to reduce the risk
of product quality problems to an acceptable level prior to release.

Testers in Agile projects can use the same types of techniques used in traditional
projects to
ü identify quality risks (or product risks),
ü assess the associated level of risk,
ü estimate the effort required to reduce those risks sufficiently, and
ü then mitigate those risks through
Ø test design,
Ø implementation, and
Ø execution.

However, given the short iterations and rate of change in Agile projects,
some adaptations of those techniques are required.

http://www.istqb.vn
http://www.istqb.vn
3.2.1. Assessing Quality Risks in Agile Projects

One of the many challenges in testing is the proper selection, allocation, and prioritization of test conditions.
This includes determining the appropriate amount of effort to allocate in order to
cover each condition with tests, and sequencing the resulting tests in a way that optimizes the
effectiveness and efficiency of the testing work to be done.

Risk identification, analysis, and risk mitigation strategies can be used by the testers in Agile teams to
help determine an acceptable number of test cases to execute,
although many interacting constraints and variables may require compromises.

Risk is the possibility of a negative or undesirable outcome or event. The level of risk is found by
assessing the likelihood of occurrence of the risk and the impact of the risk.

v When the primary effect of the potential problem is on product quality,

potential problems are referred to as quality risks or product risks.
v When the primary effect of the potential problem is on project success,
potential problems are referred to as project risks or planning risks.

http://www.istqb.vn
http://www.istqb.vn
3.2.1. Assessing Quality Risks in Agile Projects

In Agile projects, quality risk analysis takes place at two places.

v Release planning: business representatives who know the features in the release provide a
high-level overview of the risks, and the whole team, including the tester(s), may assist in the
risk identification and assessment.
v Iteration planning: the whole team identifies and assesses the quality risks.

Examples of quality risks for a system include:

q Incorrect calculations in reports (a functional risk related to accuracy)
q Slow response to user input (a non-functional risk related to efficiency and response time)
q Difficulty in understanding screens and fields (a non-functional risk related to usability and
understandability)

http://www.istqb.vn
http://www.istqb.vn
3.2.1. Assessing Quality Risks in Agile Projects

As mentioned earlier, an iteration starts with iteration planning, which culminates in estimated tasks on a
task board. These tasks can be prioritized in part based on the level of quality risk associated with them.
Ø Tasks associated with higher risks should start earlier and involve more testing effort.
Ø Tasks associated with lower risks should start later and involve less testing effort.

An example of how the quality risk analysis process in an Agile project may be carried out during
iteration planning is outlined in the following steps:
1. Gather the Agile team members together, including the tester(s)
2. List all the backlog items for the current iteration (e.g., on a task board)
3. Identify the quality risks associated with each item, considering all relevant quality characteristics
4. Assess each identified risk, which includes two activities: categorizing the risk and determining its
level of risk based on the impact and the likelihood of defects
5. Determine the extent of testing proportional to the level of risk
6. Select the appropriate test technique(s) to mitigate each risk, based on the risk, the level of risk,
and the relevant quality characteristic
http://www.istqb.vn
http://www.istqb.vn
3.2.1. Assessing Quality Risks in Agile Projects

The tester then designs, implements, and executes tests to mitigate the risks. This includes the totality of
features, behaviors, quality characteristics, and attributes that affect customer, user, and stakeholder
satisfaction.
Throughout the project, the team should remain aware of additional information that may change the
set of risks and/or the level of risk associated with known quality risks.
Periodic adjustment of the quality risk analysis, which results in adjustments to the tests, should occur.
Adjustments include identifying new risks, re-assessing the level of existing risks, and evaluating the
effectiveness of risk mitigation activities.
Quality risks can also be mitigated before test execution starts.

periodic

http://www.istqb.vn
http://www.istqb.vn
What is it?

http://www.istqb.vn
http://www.istqb.vn
The Fibonacci Sequence

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89,…

http://www.istqb.vn
http://www.istqb.vn
3.2.2. Estimating Testing Effort Based on Content and Risk – Planning Poker

Planning poker
1. The product owner or customer reads a user story to the estimators
2. Each estimator has a deck of cards with values similar to the Fibonacci sequence or shirt sizes
3. The estimators discuss the feature, and ask questions of the product owner as needed
4. Each estimator privately selects one card to represent his or her estimate
5. All cards are then revealed at the same time.
(If all estimators selected the same value, that becomes the estimate)
6. If not, the estimators discuss the differences in estimates
7. the poker round is repeated until agreement is reached
either by consensus or by applying rules (e.g., median, highest score)

http://www.istqb.vn
http://www.istqb.vn
3.2.2. Estimating Testing Effort Based on Content and Risk – Planning Poker

During release planning, the Agile team estimates the effort required to complete the release.
The estimate addresses the testing effort as well.

A common estimation technique used in Agile projects is planning poker, a consensus-based technique.
1. The product owner or customer reads a user story to the estimators.
2. Each estimator has a deck of cards with values similar to the Fibonacci sequence (1, 2, 3, 5, 8, 8+)
or any other progression of choice (e.g., shirt sizes: XS, S, M, L, XL, XXL).

The values represent the number of story points, effort days, or other units in which the team estimates.
The Fibonacci sequence is recommended because the numbers in the sequence reflect that
uncertainty grows proportionally with the size of the story.

A high estimate usually means that the story is

• not well understood or
• should be broken down into multiple smaller stories

http://www.istqb.vn
http://www.istqb.vn
3.2.2. Estimating Testing Effort Based on Content and Risk – Planning Poker

…
3. The estimators discuss the feature, and ask questions of the product owner as needed. Aspects such
as development and testing effort, complexity of the story, & scope of testing play a role in the estimation.
Therefore, it is advisable to include the risk level of a backlog item, in addition to the priority specified by
the product owner, before the planning poker session is initiated.
4. When the feature has been fully discussed, each estimator privately selects one card to represent his
or her estimate.
5. All cards are then revealed at the same time.
=> If all estimators selected the same value, that becomes the estimate.
6. If not, the estimators discuss the differences in estimates, after which
7. the poker round is repeated until agreement is reached,
(either by consensus or by applying rules (e.g., use the median or highest score)
to limit the number of poker rounds.)

These discussions ensure a reliable estimate of the effort needed to complete

product backlog items requested by the product owner and
help improve collective knowledge of what has to be done.
http://www.istqb.vn
http://www.istqb.vn
3.3. Techniques in Agile Projects

3.3 Techniques in Agile Projects

3.3.1 Acceptance Criteria, Adequate Coverage, and Other Information for Testing
3.3.2 Applying Acceptance Test-Driven Development
3.3.3 Functional and Non-Functional Black Box Test Design
3.3.4 Exploratory Testing and Agile Testing

http://www.istqb.vn
http://www.istqb.vn
3.3. Techniques in Agile Projects

Many of the test techniques and testing levels that apply to traditional projects
can also be applied to Agile projects.

However, for Agile projects, there are some specific considerations and
variances in test techniques, terminologies, and documentation
that should be considered.

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

Agile projects outline initial requirements as user stories in a prioritized backlog at the start of the
project. Initial requirements are short and usually follow a predefined format (see Section 1.2.2).
Non-functional requirements, such as usability and performance, are also important and can be
specified as unique user stories or connected to other functional user stories.

Non-functional requirements may follow a predefined format or standard, such as [ISO25000], or an

industry specific standard.

The user stories serve as an important test basis. Other possible test bases include:
Ø Experience from previous projects
Ø Existing functions, features, and quality characteristics of the system
Ø Code, architecture, and design
Ø User profiles (context, system configurations, and user behavior)
Ø Information on defects from existing and previous projects
Ø A categorization of defects in a defect taxonomy
Ø Applicable standards (e.g., [DO-178B] for avionics software)
Ø Quality risks (see Section 3.2.1)

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

During each iteration, developers create code which implements the functions and features described in the
user stories, with the relevant quality characteristics, and this code is verified and validated via acceptance
testing. To be testable, acceptance criteria should address the following topics where relevant:

Ø Functional behavior: The externally observable behavior with user actions as input operating under certain
configurations.
Ø Quality characteristics: How the system performs the specified behavior. The characteristics may also be referred to
as quality attributes or non-functional requirements. Common quality characteristics are performance, reliability, usability, etc.
Ø Scenarios (use cases): A sequence of actions between an external actor (often a user) and the system, in order to
accomplish a specific goal or business task.
Ø Business rules: Activities that can only be performed in the system under certain conditions defined by outside
procedures and constraints (e.g., the procedures used by an insurance company to handle insurance claims).
Ø External interfaces: Descriptions of the connections between the system to be developed and the outside world.
External interfaces can be divided into different types (user interface - UI, interface to other systems – ex. API, etc.)
Ø Constraints: Any design and implementation constraint that will restrict the options for the developer. Devices with
embedded software must often respect physical constraints such as size, weight, and interface connections.
Ø Data definitions: The customer may describe the format, data type, allowed values, and default values for a data item
in the composition of a complex business data structure (e.g., the ZIP code in a U.S, email address).
http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

In addition to the user stories and their associated acceptance criteria, other information is
relevant for the tester, including:
Ø How the system is supposed to work and be used
Ø The system interfaces that can be used/accessed to test the system
Ø Whether current tool support is sufficient
Ø Whether the tester has enough knowledge and skill to perform the necessary tests

Testers will often discover the need for additional information (e.g., code coverage) throughout
the iterations and should work collaboratively with the rest of the Agile team members to obtain that
information.
Relevant information plays a part in determining whether a particular activity can be considered
done. This concept of the definition of done is critical in Agile projects and applies in a number of
different ways as discussed in the following sub-subsections.

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

Test Levels
Each test level has its own definition of done. The following list gives examples that may be relevant
for the different test levels.

Unit testing
Ø 100% decision coverage where possible, with careful reviews of any infeasible paths
Ø Static analysis performed on all code
Ø No unresolved major defects (ranked based on priority and severity)
Ø No known unacceptable technical debt remaining in the design and the code [Jones11]
Ø All code, unit tests, and unit test results reviewed
Ø All unit tests automated
Ø Important characteristics are within agreed limits (e.g., performance)

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

Integration testing
Ø All functional requirements tested, including both positive and negative tests, with the number of tests based on
size, complexity, and risks
Ø All interfaces between units tested
Ø All quality risks covered according to the agreed extent of testing
Ø No unresolved major defects (prioritized according to risk and importance)
Ø All defects found are reported
Ø All regression tests automated, where possible, with all automated tests stored in a common repository

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

System testing
Ø End-to-end tests of user stories, features, and functions
Ø All user personas covered (permissions, user types, user roles, anonymous, etc.)
Ø The most important quality characteristics of the system covered (e.g., performance, robustness, reliability)
Ø Testing done in a production-like environment(s), including all hardware and software for all supported configurations,
to the extent possible
Ø All quality risks covered according to the agreed extent of testing
Ø All regression tests automated, where possible, with all automated tests stored in a common repository
Ø All defects found are reported and possibly fixed
Ø No unresolved major defects (prioritized according to risk and importance)

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

User Story
The definition of done for user stories may be determined by the following criteria:
v The user stories selected for the iteration are complete, understood by the team, and have
detailed, testable acceptance criteria
v All the elements of the user story are specified and reviewed, including the user story
acceptance tests, have been completed
v Tasks necessary to implement and test the selected user stories have been identified and
estimated by the team

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

Feature
The definition of done for features, which may span multiple user stories or epics, may include:

Ø All constituent user stories, with acceptance criteria, are defined and approved by the customer
Ø The design is complete, with no known technical debt
Ø The code is complete, with no known technical debt or unfinished refactoring
Ø Unit tests have been performed and have achieved the defined level of coverage
Ø Integration tests and system tests for the feature have been performed according to the defined coverage criteria
Ø No major defects remain to be corrected
Ø Feature documentation is complete, which may include release notes, user manuals, and on-line help functions

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

Iteration
The definition of done for the iteration may include the following:
v All features for the iteration are ready and individually tested according to the feature level
criteria
v Any non-critical defects that cannot be fixed within the constraints of the iteration added to the
product backlog and prioritized
v Integration of all features for the iteration completed and tested
v Documentation written, reviewed, and approved

At this point, the software is potentially releasable because the iteration has been successfully completed,
but not all iterations result in a release.

http://www.istqb.vn
http://www.istqb.vn
3.3.1. Acceptance Criteria, Adequate Coverage, and Other Information of Testing

Release
The definition of done for a release, which may span multiple iterations, may include the following areas:
v Coverage: All relevant test basis elements for all contents of the release have been covered by testing. The
adequacy of the coverage is determined by what is new or changed, its complexity and size, and the associated risks
of failure.
v Quality: The defect intensity (e.g., how many defects are found per day or per transaction), the defect density (e.g.,
the number of defects found compared to the number of user stories, effort, and/or quality attributes), estimated
number of remaining defects are within acceptable limits, the consequences of unresolved and remaining defects (e.g.,
the severity and priority) are understood and acceptable, the residual level of risk associated with each identified quality
risk is understood and acceptable.
v Time: If the pre-determined delivery date has been reached, the business considerations associated with releasing
and not releasing need to be considered.
v Cost: The estimated lifecycle cost should be used to calculate the return on investment for the delivered system (i.e.,
the calculated development and maintenance cost should be considerably lower than the expected total sales of the
product). The main part of the lifecycle cost often comes from maintenance after the product has been released, due to
the number of defects escaping to production.

http://www.istqb.vn
http://www.istqb.vn
3.3.2. Applying Acceptance Test-Driven Development

Acceptance test-driven development is a test-first approach.

“Test cases are created prior to implementing the user story.”

The test cases are created by the Agile team, including the developer, the tester, and the business
representatives and may be manual or automated.

The first step is a specification workshop The next step is to create the tests.
where the user story is analyzed, discussed, This can be done by the team together
and written by developers, testers, and business or by the tester individually.
representatives. In any case, an independent person such as a
Any incompleteness, ambiguities, or errors in the user business representative validates the tests.
story are fixed during this process. The tests are examples that describe the
specific characteristics of the user story.

These examples will help the team implement the user story correctly.
Since examples and tests are the same, these terms are often used interchangeably.
The work starts with basic examples and open questions.
http://www.istqb.vn
http://www.istqb.vn
3.3.2. Applying Acceptance Test-Driven Development

Typically, the first tests are the positive tests,

• confirming the correct behavior without exception or error conditions,
• comprising the sequence of activities executed if everything goes as expected.
After the positive path tests are done, the team should write negative path tests and cover
non-functional attributes as well (e.g., performance, usability).

Tests are expressed in a way that every stakeholder is able to understand, containing sentences in
natural language involving:
• the necessary preconditions, if any,
• the inputs, and
• the related outputs.

The examples must cover all the characteristics of the user story and should not add to the story.
This means that an example should not exist which describes an aspect of the user story not
documented in the story itself.
In addition, no two examples should describe the same characteristics of the user story.
http://www.istqb.vn
http://www.istqb.vn
3.3.3. Functional and Non-Functional Black Box Test Design

In Agile testing, many tests are created by testers concurrently with the developers’
programming activities.
Just as the developers are programming based on the user stories and acceptance
criteria, so are the testers creating tests based on user stories and their acceptance criteria.
(Some tests, such as exploratory tests and some other experience-based tests, are
created later, during test execution, as explained in Section 3.3.4.)

Testers can apply traditional black box test design techniques such as equivalence
partitioning, boundary value analysis, decision tables, and state transition testing to
create these tests.
For example, boundary value analysis could be used to select test values when
a customer is limited in the number of items they may select for purchase.

http://www.istqb.vn
http://www.istqb.vn
3.3.3. Functional and Non-Functional Black Box Test Design

In many situations, non-functional requirements can be documented as user stories.

Black box test design techniques (such as boundary value analysis) can also be used to
create tests for non-functional quality characteristics.

The user story might contain performance or reliability requirements.

For example,
1. a given execution cannot exceed a time limit or
2. a number of operations may fail less than a certain number of times.

For more information about the use of black box test design techniques, see the
Foundation Level (CTFL) syllabus and the Advanced Level Test Analyst (CTAL TA) syllabus.

http://www.istqb.vn
http://www.istqb.vn
3.3.4. Exploratory Testing and Agile Testing – Test Strategies

Exploratory testing is important in Agile projects due to the limited time available for test analysis
and the limited details of the user stories.

In order to achieve the best results, exploratory testing should be combined with other experience-
based techniques as part of a reactive testing strategy, blended with other testing strategies such as:
• analytical risk-based testing,
• analytical requirements-based testing,
• model-based testing, and
• regression-averse testing.

Test strategies and test strategy blending is discussed in the Foundation Level (CTFL) syllabus.

In exploratory testing,
test design and test execution occur at the same time, guided by a prepared test charter.

A test charter provides the test conditions to cover during a time-boxed testing session.
During exploratory testing, the results of the most recent tests guide the next test.

http://www.istqb.vn
http://www.istqb.vn
3.3.4. Exploratory Testing and Agile Testing – Test Charter

Target: Where are you exploring?

It could be a feature, a requirement, or a module.

Resources: What resources will you bring with you?

Resources can be anything:
• a tool, a data set,
• a technique, a configuration, or
• perhaps an interdependent feature.

Information:
• What kind of information are you hoping to find?
• Are you characterizing the security, performance, reliability, capability,
a simple three-part template usability, or some other aspect of the system?
• Are you looking for consistency of design or violations of a standard?

http://www.istqb.vn
http://www.istqb.vn
3.3.4. Exploratory Testing and Agile Testing – Test Charter

A test charter may include the following information:

Ø Actor: intended user of the system
Ø Purpose: the theme of the charter including what particular objective the actor wants to achieve, i.e.,
the test conditions
Ø Setup: what needs to be in place in order to start the test execution
Ø Priority: relative importance of this charter, based on the priority of the associated user story or the
risk level
Ø Reference: specifications (e.g., user story), risks, or other information sources
Ø Data: whatever data is needed to carry out the charter
Ø Activities: a list of ideas of what the actor may want to do with the system (e.g., “Log on to the
system as a super user”) and what would be interesting to test (both positive and negative tests)
Ø Oracle notes: how to evaluate the product to determine correct results (e.g., to capture what
happens on the screen and compare to what is written in the user’s manual)
Ø Variations: alternative actions and evaluations to complement the ideas described
under activities
http://www.istqb.vn
http://www.istqb.vn
3.3.4. Exploratory Testing and Agile Testing – Session-based Testing

To manage exploratory testing, a method called session-based test management can be used.
A session is defined as an uninterrupted period of testing which could last from 60 to 120 minutes.
Test sessions include the following:
ü Survey session (to learn how it works)
ü Analysis session (evaluation of the functionality or characteristics)
ü Deep coverage (corner/edge cases, scenarios, interactions)

The quality of the tests depends on the testers’ ability to ask relevant questions about what to test.
Examples include the following:
ü What is the most important to find out about the system?
ü In what way may the system fail?
ü What happens if.....?
ü What should happen when.....?
ü Are customer needs, requirements, and expectations fulfilled?
ü Is the system possible to install (and remove if necessary) in all supported upgrade paths?
http://www.istqb.vn
http://www.istqb.vn
3.3.4. Exploratory Testing and Agile Testing

During test execution, the tester uses creativity, intuition, cognition, and skill to find
possible problems with the product.
The tester also needs to have good knowledge and understanding of the software
under test, the business domain, how the software is used, and how to determine
when the system fails.
A set of heuristics can be applied when testing. A heuristic can guide the tester in how to
perform the testing and to evaluate the results.

Examples include:
Ø Boundaries
Ø CRUD (Create, Read, Update, Delete)
Ø Configuration variations
Ø Interruptions (e.g., log off, shut down, or reboot)
http://www.istqb.vn
http://www.istqb.vn
3.3.4. Exploratory Testing and Agile Testing

http://www.istqb.vn
http://www.istqb.vn
3.3.4. Exploratory Testing and Agile Testing

It is important for the tester to document the process as much as possible. Otherwise, it would be difficult
to go back and see how a problem in the system was discovered.
The following list provides examples of information that may be useful to document:
Ø Test coverage: what input data have been used, how much has been covered, and how much
remains to be tested
Ø Evaluation notes: observations during testing, do the system and feature under test seem to be
stable, were any defects found, what is planned as the next step according to the current observations,
and any other list of ideas
Ø Risk/strategy list: which risks have been covered and which ones remain among the most important
ones, will the initial strategy be followed, does it need any changes
Ø Issues, questions, and anomalies: any unexpected behavior, any questions regarding the efficiency
of the approach, any concerns about the ideas/test attempts, test environment, test data,
misunderstanding of the function, test script or the system under test
Ø Actual behavior: recording of actual behavior of the system that needs to be saved
(e.g., video, screen captures, output data files)

http://www.istqb.vn
http://www.istqb.vn
3.3.4. Exploratory Testing and Agile Testing

The information logged should be captured and/or summarized into some

form of status management tools
(e.g., test management tools, task management tools, the task board),
in a way that makes it easy for
stakeholders to understand the current status for
all testing that was performed.

http://www.istqb.vn
http://www.istqb.vn
3.4. Tools in Agile Projects

3.4 Tools in Agile Projects

3.4.1 Task Management and Tracking Tools
3.4.2 Communication and Information Sharing Tools
3.4.3 Software Build and Distribution Tools
3.4.4 Configuration Management Tools
3.4.5 Test Design, Implementation, and Execution Tools
3.4.6 Cloud Computing and Virtualization Tools

http://www.istqb.vn
http://www.istqb.vn
3.4. Tools in Agile Projects - Overview

Tools described in the Foundation Level (CTFL) syllabus are relevant and used by testers on Agile teams.
Not all tools are used the same way and some tools have more relevance for Agile projects than they have
in traditional projects.

For example, although the test management tools, requirements management tools, and incident
management tools can be used by Agile teams, some Agile teams opt for an all-inclusive tool
(e.g., application lifecycle management or task management – JIRA, TFS, Trello, etc.)

that provides features relevant to Agile development, such as

task boards, burndown charts, and user stories.

Configuration management tools are important to testers in Agile teams due to the high number of
automated tests at all levels and the need to store and manage the associated automated test artifacts.

In addition to the tools described in the Foundation Level syllabus, testers on Agile projects may also utilize
the tools described in the following subsections. These tools are used by the whole team to ensure team
collaboration and information sharing, which are key to Agile practices.

http://www.istqb.vn
http://www.istqb.vn
3.4.1. Task Management and Tracking Tools

In some cases, Agile teams use physical story/task boards (e.g., whiteboard, corkboard) to manage
and track user stories, tests, and other tasks throughout each sprint. Other teams will use application
lifecycle management and task management software, including electronic task boards. These tools
serve the following purposes:
v Record stories and their relevant development and test tasks, to ensure that nothing gets lost during
a sprint
v Capture team members’ estimates on their tasks and automatically calculate the effort required to
implement a story, to support efficient iteration planning sessions
v Associate development tasks and test tasks with the same story, to provide a complete picture of
the team’s effort required to implement the story
v Aggregate developer and tester updates to the task status as they complete their work,
automatically providing a current calculated snapshot of the status of each story, the iteration, and
the overall release
v Provide a visual representation (via metrics, charts, and dashboards) of the current state of each
user story, the iteration, and the release, allowing all stakeholders to quickly check status
v Integrate with configuration management tools, which can allow automated recording of code
check-ins and builds against tasks, and, in some cases, automated status updates for tasks

http://www.istqb.vn
http://www.istqb.vn
3.4.1. Task Management and Tracking Tools

an electronic task board (JIRA)

http://www.istqb.vn
http://www.istqb.vn
3.4.2. Communication and Information Sharing Tools

In addition to e-mail, documents, and verbal communication, Agile teams often use three additional types
of tools to support communication and information sharing:

=> wikis, instant messaging, and desktop sharing.

These tools should be used to complement and extend, not replace, face-to-face communication in
Agile teams.

As discussed earlier in this syllabus, daily build and deployment of software is a key practice in Agile
teams. This requires the use of continuous integration tools and build distribution tools. The uses,
benefits, and risks of these tools was described earlier in Section 1.2.4.

http://www.istqb.vn
http://www.istqb.vn
3.4.2. Communication and Information Sharing Tools

Wikis allow teams to build and share an online knowledge base on various aspects of the project, including
the following:
Ø Product feature diagrams, feature discussions, prototype diagrams, photos of whiteboard discussions,
and other information
Ø Tools and/or techniques for developing and testing found to be useful by other members of the team
Ø Metrics, charts, and dashboards on product status, which is especially useful when the wiki is integrated
with other tools such as the build server and task management system, since the tool can update
product status automatically
Ø Conversations between team members, similar to instant messaging and email,
but in a way that is shared with everyone else on the team
Instant messaging, audio teleconferencing, and video chat tools provide the following benefits:
Ø Allow real time direct communication between team members, especially distributed teams
Ø Involve distributed teams in standup meetings
Ø Reduce telephone bills by use of voice-over-IP technology, removing cost constraints that could reduce
team member communication in distributed settings
Desktop sharing and capturing tools provide the following benefits:
Ø In distributed teams, product demonstrations, code reviews, and even pairing can occur
Ø Capturing product demonstrations at the end of each iteration, which can be posted to the team’s wiki
http://www.istqb.vn
http://www.istqb.vn
3.4.3. Software Build and Distribution Tools

http://www.istqb.vn
http://www.istqb.vn
3.4.4. Configuration Management Tools

On Agile teams, configuration management tools may be used not only to store source code and
automated tests, but manual tests and other test work products are often stored in the same
repository as the product source code.
This provides traceability between which versions of the software were tested with which particular
versions of the tests, and allows for rapid change without losing historical information.

The main types of version control systems include centralized source control systems and
distributed version control systems.
The team size, structure, location, and requirements to integrate with other tools will determine which
version control system is right for a particular Agile project.

http://www.istqb.vn
http://www.istqb.vn
3.4.5. Test Design, Implementation, and Execution Tools

Some tools are useful to Agile testers at specific points in the software testing process. While most of
these tools are not new or specific to Agile, they provide important capabilities given the rapid change
of Agile projects.
v Test design tools: Use of tools such as mind maps have become more popular to quickly design
and define tests for a new feature.
v Test case management tools: The type of test case management tools used in Agile may be part
of the whole team’s application lifecycle management or task management tool.
v Test data preparation and generation tools: Tools that generate data to populate an application’s
database are very beneficial when a lot of data and combinations of data are necessary to test
the application.

These tools can also help re-define the database structure as the product undergoes changes during
an Agile project and refactor the scripts to generate the data. This allows quick updating of test data
as changes occur.
Some test data preparation tools use production data sources as a raw material and use scripts to
remove or anonymize sensitive data. Other test data preparation tools can help with validating large
data inputs or outputs. http://www.istqb.vn
http://www.istqb.vn
3.4.5. Test Design, Implementation, and Execution Tools

v Test data load tools: After data has been generated for testing, it needs to be loaded into the
application. Manual data entry is often time consuming and error prone, but data load tools are
available to make the process reliable and efficient. In fact, many of the data generator tools
include an integrated data load component. In other cases, bulk-loading using the database
management systems is also possible.
v Automated test execution tools: There are test execution tools which are more aligned to Agile
testing. Specific tools are available via both commercial and open source avenues to support test
first approaches, such as behavior-driven development, test-driven development, and acceptance
test-driven development. These tools allow testers and business staff to express the expected
system behavior in tables or natural language using keywords.
v Exploratory test tools: Tools that capture and log activities performed on an application during an
exploratory test session are beneficial to the tester and developer, as they record the actions
taken. This is useful when a defect is found, as the actions taken before the failure occurred have
been captured and can be used to report the defect to the developers. Logging steps performed
in an exploratory test session may prove to be beneficial if the test is ultimately included in the
automated regression test suite.

http://www.istqb.vn
http://www.istqb.vn
3.4.6. Cloud Computing and Virtualization Tools

Virtualization allows a single physical resource (server) to operate as many separate, smaller resources.

When virtual machines or cloud instances are used, teams have a greater number of servers available
to them for development and testing. This can help to avoid delays associated with waiting for physical
servers.

Provisioning a new server or restoring a server is more efficient with snapshot capabilities built into
most virtualization tools.

Some test management tools now utilize virtualization technologies to snapshot servers at the point
when a fault is detected, allowing testers to share the snapshot with the developers investigating the
fault.

http://www.istqb.vn
http://www.istqb.vn
Practice Makes Perfect!
http://www.istqb.vn
References

References
• ISTQB.ORG: Foundation Level Extension Syllabus - Agile Tester
• Download link: http://www.istqb.vn/node/14
• Agile-related terms
Ø http://guide.Agilealliance.org
Ø http://whatis.techtargetcom/glossary
Ø http://www.scrumalliance.org

Standards
• [DO-178B] RTCA/FAA DO-178B: Software Considerations in Airborne Systems
and Equipment Certification, 1992.

• [ISO25000] ISO/IEC 25000:2005: Software Engineering - Software Product

Quality Requirements and Evaluation (SQuaRE), 2005.

http://www.istqb.vn
http://www.istqb.vn