Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
A systematic mapping study on test
generation from Input/Output Transition
Systems
Sofia Costa Paiva
Adenilso Simao
Instituto de Ci
encias Matem
aticas e de Computac
ao
S
ao Paulo University, Brazil
August 27, 2015
1 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Introduction
Model-Based Testing
Problem Statement
Mapping Planning and Conduction
Goal and Research Questions
Search Strategy and Extraction Process
Analysis and Discussion of Results
General Information about Studies
IOTS Model Characteristics
Test Selection Criteria
Technology
Test Execution
Map of selected studies
Threats to validity
Conclusion
Conclusion
2 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Context
Software testing:
Demonstrate the actual and expected behaviors of a system
are at least similar
Manual and laborious process
Model-Based Testing (MBT)
Strategy to minimize time and resource limitations
Automate the generation of high-quality test suites from
behavioural models.
A model of desired implementation behaviour: the starting
point of MBT.
3 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Techniques used in MBT
Finite State Machines FSM
Test generation methods
from FSM guarantee
fault coverage using a
fault model
Input/Output Transition
Systems - IOTS
Existing works merely
indicates that test suite
must be generated in a
nondeterministic way
4 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Goal
Several recent studies have addressed issues related to test
generation from IOTSs
IOTS is a suitable formalism to model complex systems, such
as communication protocols and distributed systems
Systematic mapping study
Important step to identify relevant studies and shortcomings
Rigorous process of assessment and interpretation of all
available studies
Reported systematic mapping studies regarding MBT not
including IOTS formalism.
5 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Goal and Research Questions
Goal
Identify methods for test case generation from IOTSs through
a systematic mapping.
Research questions:
RQ1. What are the main characteristics of IOTS models
applied in the studies?
RQ2. What are the test selection criteria used in the studies?
RQ3. What are the technologies used in the test generation
from IOTSs?
RQ4. What are the features of the test execution
implemented in the studies?
6 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Mapping Planning and Conduction
Search strategy : using control studies
Keywords: test generation and IOTS
Engines: IEEE Xplore, Springer Link, Scopus, Web of
Knowledge, ACM Digital Library, ScienceDirect, Compendex
and Oxford Journals
Figure : Mapping Process
7 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Taxonomy of MBT approaches
8 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Taxonomy of MBT approaches
9 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Analysis and Discussion of Results
General information about studies
Figure : Distribution of studies over
the years
Figure : Distribution of
studies into forums
10 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Analysis and Discussion of Results
General information about studies
Computational support
TGV (6 studies)
TorX (1 study)
UPPAAL (1 study)
Tipe of evidence:
68% provide only examples
13% report a case study
8% report experimental results
10% only discuss the application of the approach
11 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Analysis and Discussion of Results
RQ1. IOTS Model Characteristics
Untimed (92%) and
Timed (8%)
Deterministic (88%) and
non-deterministic (12%)
Discrete (98%), hybrid
(1%) and continuous
(1%)
12 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Analysis and Discussion of Results
RQ2. Test Selection Criteria
Structural model coverage (14 studies)
Requirements coverage: test purpose (26 studies)
Requirements coverage: properties (5 studies)
Test case specification (7 studies)
Test case specification: ioco theory (31 studies)
Fault-based (12 studies)
13 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Analysis and Discussion of Results
RQ3. Technology
Random/non-deterministic (26 studies)
Search-based algorithms (26 studies)
Model-checking (8 studies)
Simbolic execution (10 studies)
Constraint solving (6 studies)
Inference rules (5 studies)
Data abstraction (1 study)
14 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Analysis and Discussion of Results
RQ4. Test Execution
Online (34 studies)
Offline (25 studies)
Communication:
Synchronous (26 studies)
Asynchronous (10 studies)
Distributed (12 studies)
Online and synchronous test generation have been widely used
adopted by the classical method of Tretmans.
Online testing use randomize choice techniques
provided only a probabilistic guarantee of coverage for
(unrealistic) a long time execution.
15 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Analysis and Discussion of Results
Map of selected studies regarding Test Generation criteria and
Technology
16 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Threats to validity
Selection bias was treated by defining our inclusion criteria for
gathering the largest possible amount of papers that would fit
the test generation from the IOTSs domain
A specialist validated the results so that inaccuracy in data
extraction and misclassification could be alleviated.
17 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
Conclusion
Lack of a standard of IOTS model characteristics may restrict
the adoption of each method.
Widely used coverage criteria, such as state and transition
coverage, have been applied to IOTSs in only few recent
studies.
Most of test generation technologies are non-deterministic
There exist no standard fault models, leading to most test
generation algorithms being nondeterministic.
Completeness is guaranteed in theory, by repeating the
process an unbounded number of times.
Recent studies on IOTS-testing have adopted concepts of
FSM-testing
However, fault models (which guarantee the completeness of
test suites) has played a relatively small role in IOTS testing.
Future work:
Replication of study and include more evidence
Quality evaluation
18 / 19
Introduction
Mapping Planning and Conduction
Analysis and Discussion of Results
Conclusion
A systematic mapping study on test
generation from Input/Output Transition
Systems
Sofia Costa Paiva
[email protected]Instituto de Ci
encias Matem
aticas e de Computac
ao
S
ao Paulo University, Brazil
August 27, 2015
19 / 19
