Scribd
Upload
18 views19 pages

A Systematic Mapping Study On Test Generation From Input/Output Transition Systems

This document presents the results of a systematic mapping study that identifies methods for generating tests from Input/Output Transition Systems (IOTS). The study analyzed 95 papers and identified characteristics of IOTS models, test selection criteria, technologies used for test generation, and approaches to test execution. The results showed that most studies used untimed, deterministic IOTS models and focused on test selection based on structural coverage or requirements coverage. Common technologies were random/non-deterministic techniques, search-based algorithms, and model checking. Most approaches performed online testing in a synchronous manner. The conclusion calls for standardization of IOTS models and fault models to improve completeness and adoption of test generation methods.

Uploaded by

sofialarissac
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
18 views19 pages

A Systematic Mapping Study On Test Generation From Input/Output Transition Systems

This document presents the results of a systematic mapping study that identifies methods for generating tests from Input/Output Transition Systems (IOTS). The study analyzed 95 papers and identified characteristics of IOTS models, test selection criteria, technologies used for test generation, and approaches to test execution. The results showed that most studies used untimed, deterministic IOTS models and focused on test selection based on structural coverage or requirements coverage. Common technologies were random/non-deterministic techniques, search-based algorithms, and model checking. Most approaches performed online testing in a synchronous manner. The conclusion calls for standardization of IOTS models and fault models to improve completeness and adoption of test generation methods.

Uploaded by

sofialarissac
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

A systematic mapping study on test


generation from Input/Output Transition
Systems
Sofia Costa Paiva
Adenilso Simao
Instituto de Ci
encias Matem
aticas e de Computac
ao
S
ao Paulo University, Brazil

August 27, 2015

1 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Introduction
Model-Based Testing
Problem Statement

Mapping Planning and Conduction


Goal and Research Questions
Search Strategy and Extraction Process

Analysis and Discussion of Results


General Information about Studies
IOTS Model Characteristics
Test Selection Criteria
Technology
Test Execution
Map of selected studies
Threats to validity

Conclusion

Conclusion

2 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Context
Software testing:
Demonstrate the actual and expected behaviors of a system
are at least similar
Manual and laborious process

Model-Based Testing (MBT)


Strategy to minimize time and resource limitations
Automate the generation of high-quality test suites from
behavioural models.
A model of desired implementation behaviour: the starting
point of MBT.

3 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Techniques used in MBT


Finite State Machines FSM
Test generation methods
from FSM guarantee
fault coverage using a
fault model

Input/Output Transition
Systems - IOTS
Existing works merely
indicates that test suite
must be generated in a
nondeterministic way

4 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Goal
Several recent studies have addressed issues related to test
generation from IOTSs
IOTS is a suitable formalism to model complex systems, such
as communication protocols and distributed systems

Systematic mapping study


Important step to identify relevant studies and shortcomings
Rigorous process of assessment and interpretation of all
available studies
Reported systematic mapping studies regarding MBT not
including IOTS formalism.

5 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Goal and Research Questions


Goal
Identify methods for test case generation from IOTSs through
a systematic mapping.
Research questions:
RQ1. What are the main characteristics of IOTS models
applied in the studies?
RQ2. What are the test selection criteria used in the studies?
RQ3. What are the technologies used in the test generation
from IOTSs?
RQ4. What are the features of the test execution
implemented in the studies?

6 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Mapping Planning and Conduction

Search strategy : using control studies


Keywords: test generation and IOTS
Engines: IEEE Xplore, Springer Link, Scopus, Web of
Knowledge, ACM Digital Library, ScienceDirect, Compendex
and Oxford Journals

Figure : Mapping Process

7 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Taxonomy of MBT approaches

8 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Taxonomy of MBT approaches

9 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results


General information about studies

Figure : Distribution of studies over


the years

Figure : Distribution of
studies into forums

10 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

General information about studies


Computational support
TGV (6 studies)
TorX (1 study)
UPPAAL (1 study)

Tipe of evidence:
68% provide only examples
13% report a case study
8% report experimental results
10% only discuss the application of the approach

11 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results


RQ1. IOTS Model Characteristics
Untimed (92%) and
Timed (8%)
Deterministic (88%) and
non-deterministic (12%)
Discrete (98%), hybrid
(1%) and continuous
(1%)

12 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

RQ2. Test Selection Criteria


Structural model coverage (14 studies)
Requirements coverage: test purpose (26 studies)
Requirements coverage: properties (5 studies)
Test case specification (7 studies)
Test case specification: ioco theory (31 studies)
Fault-based (12 studies)

13 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results


RQ3. Technology
Random/non-deterministic (26 studies)
Search-based algorithms (26 studies)
Model-checking (8 studies)
Simbolic execution (10 studies)
Constraint solving (6 studies)
Inference rules (5 studies)
Data abstraction (1 study)

14 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results


RQ4. Test Execution
Online (34 studies)
Offline (25 studies)
Communication:
Synchronous (26 studies)
Asynchronous (10 studies)
Distributed (12 studies)

Online and synchronous test generation have been widely used


adopted by the classical method of Tretmans.

Online testing use randomize choice techniques


provided only a probabilistic guarantee of coverage for
(unrealistic) a long time execution.

15 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results


Map of selected studies regarding Test Generation criteria and
Technology

16 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Threats to validity

Selection bias was treated by defining our inclusion criteria for


gathering the largest possible amount of papers that would fit
the test generation from the IOTSs domain
A specialist validated the results so that inaccuracy in data
extraction and misclassification could be alleviated.

17 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Conclusion
Lack of a standard of IOTS model characteristics may restrict
the adoption of each method.
Widely used coverage criteria, such as state and transition
coverage, have been applied to IOTSs in only few recent
studies.
Most of test generation technologies are non-deterministic
There exist no standard fault models, leading to most test
generation algorithms being nondeterministic.

Completeness is guaranteed in theory, by repeating the


process an unbounded number of times.
Recent studies on IOTS-testing have adopted concepts of
FSM-testing
However, fault models (which guarantee the completeness of
test suites) has played a relatively small role in IOTS testing.

Future work:
Replication of study and include more evidence
Quality evaluation
18 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

A systematic mapping study on test


generation from Input/Output Transition
Systems
Sofia Costa Paiva
[email protected]
Instituto de Ci
encias Matem
aticas e de Computac
ao
S
ao Paulo University, Brazil

August 27, 2015

19 / 19

You might also like