A Systematic Mapping Study On Test Generation From Input/Output Transition Systems

Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
A systematic mapping study on test

generation from Input/Output Transition
Systems
Sofia Costa Paiva
Adenilso Simao
Instituto de Ci
encias Matem
aticas e de Computac
ao
S
ao Paulo University, Brazil
August 27, 2015
1 / 19
Introduction
Introduction
Model-Based Testing
Problem Statement

Goal and Research Questions
Search Strategy and Extraction Process

General Information about Studies
IOTS Model Characteristics
Test Selection Criteria
Technology
Test Execution
Map of selected studies
Threats to validity
Conclusion
Conclusion
2 / 19
Introduction
Conclusion
Context
Software testing:
Demonstrate the actual and expected behaviors of a system
are at least similar
Manual and laborious process
Model-Based Testing (MBT)

Strategy to minimize time and resource limitations
Automate the generation of high-quality test suites from
behavioural models.
A model of desired implementation behaviour: the starting
point of MBT.
3 / 19
Introduction
Conclusion
Techniques used in MBT

Finite State Machines FSM
Test generation methods
from FSM guarantee
fault coverage using a
fault model
Input/Output Transition
Systems - IOTS
Existing works merely
indicates that test suite
must be generated in a
nondeterministic way
4 / 19
Introduction
Conclusion
Goal
Several recent studies have addressed issues related to test
generation from IOTSs
IOTS is a suitable formalism to model complex systems, such
as communication protocols and distributed systems
Systematic mapping study

Important step to identify relevant studies and shortcomings
Rigorous process of assessment and interpretation of all
available studies
Reported systematic mapping studies regarding MBT not
including IOTS formalism.
5 / 19
Introduction
Conclusion
Goal and Research Questions

Goal
Identify methods for test case generation from IOTSs through
a systematic mapping.
Research questions:
RQ1. What are the main characteristics of IOTS models
applied in the studies?
RQ2. What are the test selection criteria used in the studies?
RQ3. What are the technologies used in the test generation
from IOTSs?
RQ4. What are the features of the test execution
implemented in the studies?
6 / 19
Introduction
Conclusion
Search strategy : using control studies

Keywords: test generation and IOTS
Engines: IEEE Xplore, Springer Link, Scopus, Web of
Knowledge, ACM Digital Library, ScienceDirect, Compendex
and Oxford Journals
Figure : Mapping Process
7 / 19
Introduction
Conclusion
Taxonomy of MBT approaches
8 / 19
Introduction
Conclusion
Taxonomy of MBT approaches
9 / 19
Introduction
Conclusion

General information about studies
Figure : Distribution of studies over

the years
Figure : Distribution of
studies into forums
10 / 19
Introduction
Conclusion
General information about studies

Computational support
TGV (6 studies)
TorX (1 study)
UPPAAL (1 study)
Tipe of evidence:
68% provide only examples
13% report a case study
8% report experimental results
10% only discuss the application of the approach
11 / 19
Introduction
Conclusion

RQ1. IOTS Model Characteristics
Untimed (92%) and
Timed (8%)
Deterministic (88%) and
non-deterministic (12%)
Discrete (98%), hybrid
(1%) and continuous
(1%)
12 / 19
Introduction
Conclusion
RQ2. Test Selection Criteria

Structural model coverage (14 studies)
Requirements coverage: test purpose (26 studies)
Requirements coverage: properties (5 studies)
Test case specification (7 studies)
Test case specification: ioco theory (31 studies)
Fault-based (12 studies)
13 / 19
Introduction
Conclusion

RQ3. Technology
Random/non-deterministic (26 studies)
Search-based algorithms (26 studies)
Model-checking (8 studies)
Simbolic execution (10 studies)
Constraint solving (6 studies)
Inference rules (5 studies)
Data abstraction (1 study)
14 / 19
Introduction
Conclusion

RQ4. Test Execution
Online (34 studies)
Offline (25 studies)
Communication:
Synchronous (26 studies)
Asynchronous (10 studies)
Distributed (12 studies)
Online and synchronous test generation have been widely used

adopted by the classical method of Tretmans.
Online testing use randomize choice techniques

provided only a probabilistic guarantee of coverage for
(unrealistic) a long time execution.
15 / 19
Introduction
Conclusion

Map of selected studies regarding Test Generation criteria and
Technology
16 / 19
Introduction
Conclusion
Threats to validity
Selection bias was treated by defining our inclusion criteria for

gathering the largest possible amount of papers that would fit
the test generation from the IOTSs domain
A specialist validated the results so that inaccuracy in data
extraction and misclassification could be alleviated.
17 / 19
Introduction
Conclusion
Conclusion
Lack of a standard of IOTS model characteristics may restrict
the adoption of each method.
Widely used coverage criteria, such as state and transition
coverage, have been applied to IOTSs in only few recent
studies.
Most of test generation technologies are non-deterministic
There exist no standard fault models, leading to most test
generation algorithms being nondeterministic.
Completeness is guaranteed in theory, by repeating the

process an unbounded number of times.
Recent studies on IOTS-testing have adopted concepts of
FSM-testing
However, fault models (which guarantee the completeness of
test suites) has played a relatively small role in IOTS testing.
Future work:
Replication of study and include more evidence
Quality evaluation
18 / 19
Introduction
Conclusion
A systematic mapping study on test

generation from Input/Output Transition
Systems
Sofia Costa Paiva
[email protected]
Instituto de Ci
encias Matem
aticas e de Computac
ao
S
ao Paulo University, Brazil
August 27, 2015
19 / 19

A Systematic Mapping Study On Test Generation From Input/Output Transition Systems

Uploaded by

Copyright:

Available Formats

A Systematic Mapping Study On Test Generation From Input/Output Transition Systems

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Systematic Mapping Study On Test Generation From Input/Output Transition Systems

Uploaded by

Copyright:

Available Formats

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results