Unlock quality at scale with

optimized test case design

 Content

1. Introduction:
Still using an outdated test design approach?

2. Traditional tests vs. Test Case Designer tests

3. How to test smarter not harder with model-based

test design

4. How Test Case Designer prioritizes tests

5. How to use Test Case Designer to create optimized

test scenarios

6. How to make smarter decisions about coverage

7. Integrating your optimized tests within Xray Enterprise

8. Conclusion:
Unlock quality at scale with optimized test case design

2 Unlock quality at scale with optimized test case design

 Introduction
Still using an outdated test
design approach?

Outdated test case design

Struggles with test selection lead to

Too Many
extended deadlines, more defect
Tests
vulnerabilities, and unknown coverage

Defects in Higher cost of fixing and (less tangible)

Production damage through customer satisfaction

Lack of Lost revenue from slower speed to

Agility market and competitive disadvantage

Difficult Test Larger percentage of manual

Automation execution and excessive costs

QA teams can be plagued with too many tests

and can struggle to select the right tests for their
efforts, leading to extended deadlines, more defects
slipping through, and lack of confidence in quality.
Furthermore, defects slipping through with these
outdated methods often lead to higher costs to fix
them and greater decline in customer satisfaction.

3 Unlock quality at scale with optimized test case design

 Traditional test case design can be fragile and lacks
agility which may lead to revenue loss from slower
speed to market competitive advantage due to lower
quality releases. It’s also common to struggle with
automating tests - leading to excessive costs and
difficult-to-maintain test suites.

With Xray’s Test Case Designer, users take a

model-based approach to designing more powerful
tests. Xray’s algorithm uses combinatorial/n-wise
methods to generate optimized scenarios based
on the user’s parameter models.

Now, business and technical teams can collaborate to

create a shared understanding of testing scopes and
approach test creation with 100% confidence in the
efficiency and thoroughness of their tests.

4 Unlock quality at scale with optimized test case design

 Traditional tests vs.
Test Case Designer tests
 Traditionally hand-selected tests naturally contain
wasteful repetition of testing ideas within them, leading
to critical gaps in testing coverage with certain critical
system interactions and paths left untouched.

On the other hand, Test Case Designer tests are as

systematically varied as possible, ensuring 100% critical
testing coverage is achieved in as mathematically few
tests as possible.

Traditional Tests Test Case Designer

1 Test 4

1-2 Test 1 - 2 2 Test 3

3 Test 3 - 4 3 Test 1

4 4 Test 2

6 Unlock quality at scale with optimized test case design

 On a high-level Xray’s Test Case Designer helps you:

• Save time in creating tests and covering critical

interactions

• Save time in scripting tests for manual

or automated execution

• Communicate and agree upon testing scopes

across business and technical teams

• Gain confidence in critical coverage achieved

• Make smarter, more data-driven execution

decisions

7 Unlock quality at scale with optimized test case design

 However, the Test Case Designer’s value proposition
extends far beyond just the test design phase - its
effects and value are realized in the execution and
automation phases.

• Teams become more confident and more precise

when making execution decisions.

• Teams find defects earlier in the process, and

doing so in fewer tests to run and maintain over
time, leads to time and cost savings

• Test Case Designer scenarios can begin to

contribute to automation efforts as well through
the various outputs - ensuring teams are selecting
the right scenarios to run for automation, instead
of doing that on their own.

• Greater efficiency, plus more thorough test sets

now equal more defects found per testing hour
and faster time to market.

Let’s see how this works inside Xray’s Test Case

Designer.

8 Unlock quality at scale with optimized test case design

 How to test
smarter not harder with
model-based test design
All testers know that to “test everything,” is almost
always impossible - and it’s certainly not worth
anyone’s time. Typically, within a given system under
test, there could be thousands, millions, billions, or
even trillions of possible scenarios that we could test.

On the other side of the spectrum, we are not going

to simply test everything once. This leads to significant
gaps in coverage and leaves plenty of critical system
interactions left untouched.

Instead, we are going to meet in the middle on this

spectrum - focusing on covering all of the critical
system interactions that typically lead to triggering
defects.

“Test Everything, Once” “Test Everything”

Neither is a realistic option - in most situations.

We need a strategy somewhere in the middle.

“Most failures are induced by single factor faults or by the

interaction of two factors, with progressively fewer failures
induced by interactions between three or more factors.”

Rick Kuhn | Computer Scientist at NIST

Unlock quality at scale with optimized test case design 10

 Model-based testing is a software testing technique
that creates test cases based on a model of how we
expect our system to work. Model-based test design
differs from traditional test case design; instead of
receiving requirements and immediately writing your
test scripts one by one, you first analyze your
requirements and, through that analysis, build models
of critical testing parameters that need to be covered
in your tests.

The building blocks of every Test Case Designer test

are parameters and parameter values. Parameters
are the critical factors that must be tested, these
can include input sequences, actions, functions,
diagram/workflow nodes and, in some cases, system
outcomes.

When deciding whether or not a factor within your

requirements/systems are critical to include as
parameters inside your test model, you should ask
yourself two questions:

1 Is this factor varying in different ways?

2 By varying, is it impacting how the system works in

different ways?

If you can answer “yes” to both of these questions, you

should include that parameter with its values within
your test model.

11 Unlock quality at scale with optimized test case design

 Below is an example of an abstraction of a test model
using a mind map inside the Test Case Designer.

Inside the Test Case Designer, these parameters and

values will be 100% covered in your generated,
optimized scenarios.

12 Unlock quality at scale with optimized test case design

How Test Case Designer
prioritizes tests
 By now you’ve learned that by using the Test Case
Designer feature in Xray Enterprise, you take a
model-first approach to designing more thorough and
efficient tests by setting up parameters and
parameters values. But how does the tool generate
optimized scenarios once we define those models?
And what do we mean by “optimized”?

“Combinatorial Test Design” is a prioritization strategy

that helps testers select the most efficient and effective
tests for their system by focusing on the coverage of
interactions between inputs/configurations.

The Test Case Designer uses a sophisticated test

generation algorithm powered by Intelligent
Augmentation to eliminate gaps in coverage
systematically and taking advantage of combinatorial
test case design.

Xray’s Test Case Designer also supports risk-based

testing at the parameter level - its “Mixed strength”
option allows you to focus more thorough coverage
on high-priority aspects.

14 Unlock quality at scale with optimized test case design

 Let’s take a look at the benefits that come from using
these combinatorial approaches to designing tests.

According to a study on Combinatorial Testing

conducted across 10 pilot projects at real global
companies. The results were astounding.

Test Execution Efficiency Increased

2.4x more bugs were found per tester hour

Testing Thoroughness Increased

13% more defects were found overall

Test Design Time Decreased

36% less time spent on documenting tests

Source: “Combinatorial Testing” IEEE Computer, August 2009.

Dr Rick Kuhn, Dr. Raghu Kacker, Dr. Jeff Lei, Justin Hunter.

15 Unlock quality at scale with optimized test case design

 How to use Test Case
Designer to create
optimized test scenarios
 Through academic research and practical application
at thousands of firms worldwide, it has been
statistically proven that most defects can be found
by at most two factors (parameter values) being tested
against one another. In other words, most defects are
caused by testing every single factor and, furthermore,
testing every single factor with every other factor at
least once - hence the terms “covering all pairs” or
“pairwise” testing.

Since that is how the majority of defects are

uncovered, the Test Case Designer algorithm says,
“Well, at a very minimum, then let’s start by covering
100% of those pairs between the defined parameter
values.” Not only will you cover 100% of those pairwise
(or “2-way”) interactions, but the algorithm will do so in
as mathematically few tests as possible. It is impossible
for any human to do this, at east at the speed at which
an algorithm can.

With the Test Case Designer, now you will objectively

create better tests than traditional test case creation
methods, you will be covering more and doing so in
fewer tests to execute and maintain inside Xray over
time.

At any point, while designing your parameter models,

you can click the “Scenarios” button inside the tool
to generate optimized tests within seconds. When
generating scenarios, the algorithm takes a pairwise
analysis approach.

17 Unlock quality at scale with optimized test case design

 Selecting “3-way interactions” will increase the
thoroughness of your set - instead of covering all pairs,
your scenarios will now cover 100% of the possible
combinations of 3 parameter values.
You increase your tests’ thoroughness by getting more
specific with your combinations.
Increasing the thoroughness will almost always
increase your test case count.

Remember, based on Academic Research, the majority

of defects are found by testing every 1-way or 2-way
interaction, and 3-way / 4-way / etc. defects are
significantly more rare, so be careful about increasing
the thoroughness and have a reason to do so - you
don’t want to waste your time and resources on
fruitless execution and maintenance.

18 Unlock quality at scale with optimized test case design

 Within Test Case Designer’s scripting capabilities, the
Automate screen allows you to write template Gherkin.
These Behavior-Driven Development style scripts will
be automatically applied to every applicable optimized
scenario upon Synchronizing/Exporting. Doing so
allows you to transition your optimized tests from the
Test Case Designer and into Xray Enterprise for test
management and execution purposes. Data-driven
template scripting for manual execution is also
supported.

19 Unlock quality at scale with optimized test case design

 How to make smarter
decisions about coverage
 With test optimization inside Xray’s Test Case
Designer, you will save significant time and effort - it
is faster to create tests, easier to maintain tests, and
easier to address defects earlier in the process due to
the increased thoroughness.

Let’s see how you can optimize your test execution

efforts. As seen in the below Test Case Designer
Coverage Graph, no matter which “coverage strength”
you ultimately choose, the algorithm “front-loads”
the optimized tests with coverage. Your first tests to
execute achieve significantly higher critical coverage
than your final tests.

At the end of your optimized test set, there are not

as many new interactions to still cover since so many
were covered upfront in your early tests. Using TCD’s
detailed coverage reports, you can make smart, data-
driven decisions about how much testing is enough.

If you are under a tight deadline, using the Coverage

Graph or Coverage Matrix, you may find that
executing the final tests to reach 100% coverage may
not be worth your time, due to the front-loading of
coverage that already happened in your earlier tests.
TCD provides you with flexible options and tools to
ensure you make any decision with 100% confidence.

21 Unlock quality at scale with optimized test case design

 Above, you see the Coverage Graph, one of the
detailed coverage reports provided inside TCD.
Here, you can view the exact % coverage achieved
after running any number of your optimized test
cases (e.g., “After executing the first 37 tests, we are
achieving 80.2% coverage.). Using these tools; we
can make significantly smarter decisions about our
execution efforts, answering questions such as, “how
much testing is enough for this project?”

22 Unlock quality at scale with optimized test case design

Integrating your optimized
tests within Xray Enterprise
 A few output methods are used to transition your
optimized scripts outside of the Test Case Designer.
Synchronization uses your TCD-optimized Gherkin
(or manual) scripts and the REST API to create new
issues inside Xray Enterprise or update the existing
ones. Exporting generates Feature Files, Excel files,
CSV documents, and skeletal code for automation
in various languages/frameworks available - Java,
Javascript, C#, Python, Ruby, and more.

The main difference between these two methods

is that Synchronizing places scripts inside Xray
Enterprise. In contrast, any Exporting options are
used for reporting/execution purposes outside of Xray
Enterprise.

Synchornizing your Test Case Designer tests inside

Xray Enterprise gives you full control and visibility into
your test suite so you’re ready to execute in no time.

24 Unlock quality at scale with optimized test case design

Conclusion
Unlock quality at scale with
optimized test case design

In the fast-paced world of technology, adapting to

change is key to staying competitive. You may have
been manually writing test cases and thought that this
was the only way - but that’s no longer true.

Modern QA teams know that it’s crucial to use tools to

support manual work and optimize their potential in
order to work faster and consistently improve.

With Xray Enterprise, and its Test Case Designer, it’s

easy to write test issues based on your optimized
scenarios and meet 100% of your system’s critical
interactions.

Stop the guesswork to test case design, and use the

power of Xray Enterprise to create powerful tests
that leave you confident to release the highest quality
software.

Curious to learn more about Test Case Designer?

Start a free trial of Xray Enterprise.

Try now

Unlock quality at scale with optimized test case design 25

Links

www.getxray.app/xray-enterprise

/showcase/xrayapp/

/xrayapp/

/xrayapp/

Unlock quality at scale with optimized test case design 26

 getxray.app

Unlock quality at scale with optimized test case design 27