TENCON 2023 - 2023 IEEE Region 10 Conference (TENCON) ThuMoSB.

4
31 Oct - 3 Nov 2023. Chiang Mai, Thailand

Artificial Intelligence in Software Testing: A

Systematic Review
Mahmudul Islam Farhan Khan
Department of Computer Science and Engineering Department of Computer Science and Engineering
Independent University, Bangladesh Independent University, Bangladesh
[email protected] farhankhan [email protected]

Sabrina Alam Mahady Hasan

Department of Computer Science and Engineering Department of Computer Science and Engineering
Independent University, Bangladesh Independent University, Bangladesh
[email protected] [email protected]

Abstract—Software testing is a crucial component of soft- the various methods, techniques, and tools utilized in this
ware development. With the increasing complexity of software domain and evaluate their efficiency. The motivation for this
systems, traditional manual testing methods are becoming less systematic literature review stems from the potential benefits
feasible. Artificial Intelligence (AI) has emerged as a promising
approach to software testing in recent years. This review paper that AI can offer in the field of software testing. AI has the
aims to provide an in-depth understanding of the current state potential to automate the testing process and optimize testing
of software testing using AI. The review will examine the various strategies, making software testing more efficient, effective,
approaches, techniques, and tools used in this area and assess and accessible. Moreover, AI can address the shortage of
their effectiveness. The selected articles for this study have been skilled testers and help keep pace with the rapid development
extracted from different research databases using the advanced
search string strategy. Initially, 40 articles have been extracted cycles of agile development methodologies. There are several
from different research libraries. After gradual filtering finally, challenges in software testing that can be solved using AI.
20 articles have been selected for the study. After studying all Some of these issues include manually generating test cases,
the selected papers, we find that various testing tasks can be test optimization, test results analysis, etc.
automated successfully using AI (Machine Learning and Deep
The following research questions have been investigated
Learning) such as Test Case Generation, Defect Prediction, Test
Case Prioritization Metamorphic Testing, Android Testing, Test in this research study.
Case Validation, and White Box Testing. This study also finds RQ1: Does manual testing have drawbacks?
that the integration of AI in software testing is making software RQ2: Can integration of AI (ML or DL techniques) in
testing activities easier along with better performance. This software testing help to overcome the drawbacks of manual
literature review paper provides a thorough analysis of the
impact AI can have on the software testing process. testing?
Index Terms—Software Testing, Artificial Intelligence, Test RQ3: What software testing tasks can be automated by
Automation, Systematic Literature Review AI (ML or DL)?
RQ4: What techniques do researchers use to assess AI
I. I NTRODUCTION (ML or DL) when used in software testing?
Software testing has a crucial role in software engineering In this research study, 40 articles have been screened from
as it is essential for ensuring the quality, performance, different research libraries but through a gradual filtering
security, and reliability of software systems. By conducting process, only 20 articles were found suitable for the study.
testing, developers can identify and rectify any bugs, or de- We have structured the paper in the following way. Related
fects in the software, improving its overall functionality and works have been discussed in section 2 while the background
making sure that the software satisfies customer needs and of software testing and AI have been presented in section
expectations. AI is a vast area , so in this paper we mainly 3. Systematic review and the results have been presented
investigate the subarea of AI which are Machine Learning in section 4 and 5 consecutively. In the end, conclusion is
(ML) and Deep Learning (DL) techniques in software testing. presented in section 6.
The field of software testing currently faces a number of
challenges. As software systems grow increasingly complex, II. R ELATED W ORKS
it becomes more challenging to manually test all possible
scenarios. Also, traditional test automation approaches are They [1] proposed a deep learning model to rank test
time-consuming and complex to implement. Apart from that, cases. In this work, they consider historical records of test
keeping pace with agile development is also a challenge case executions and based on that deep learning model
as it requires rapid testing. AI has the potential to address rank test cases. They [2] conducted an empirical study on
these challenges by offering optimized and effective testing continuous integration testing. They found the strategy of
strategies. The aim of this study is to gain a thorough under- reward function of Reinforcement learning improves the
standing of the current state of the field of software testing existing test case prioritization practices. They [3] developed
automation through the use of AI. This review will examine a deep reinforcement learning technique for performing black

979-8-3503-0219-6/23/$31.00 ©2023 IEEE 523

box testing on android apps. Their developed technique out- the code. For performing bug prediction, ML models need
performs existing techniques in terms of fault identification. to be trained on historical data from past software projects
They [4] proposed a deep learning-based approach for priori- to identify patterns. Once the model is trained, then it can
tizing test cases from the interaction of humans with software predict the likelihood of bugs occurring in new code [15].
applications. They showed that test case prioritization can They [16] used supervised ML algorithms to predict software
perform successfully from human interactions using their faults based on historical data.
proposed model. They [5] presented an approach to generate Test Case Generation using ML: In software develop-
input for the graphical user interface of software applications ment, Test case generation from the requirement specifica-
by only capturing screenshots of applications. tions document is one of the biggest challenges in software
They [6] proposed a machine learning-based approach to testing. Software test cases can be generated using ML.
predict metamorphic relations of scientific software using ML model needs to be trained on a set of data where a
graph kernels. They concluded that features extracted from set of software features are considered as input and the
graphs help to achieve a good result. They [7] presented an corresponding test cases as output. Finally, the model uses
approach to automate test oracle mechanism using machine training data to generate new test cases [17].
learning. Their proposed approach captures historical usage Test Case Prioritization using ML: Test case prior-
data and based on that generates an oracle. They [8] detected itization can be performed using machine learning. ML
metamorphic relations using graph kernels and support vector algorithms determine the most critical test cases to execute
machines (SVM). They [9] analyzed software defect pre- based on the likelihood of failure and the potential effect on
diction using machine learning algorithms. They found that the system. For prioritizing test cases, a machine learning
linear classifier performs well compared to other algorithms. model needs to be trained on a set of labeled data, where
They [10] proposed an improved CNN model to predict a set of software features are considered as input and the
software defects and their proposed model outperformed corresponding priority level of each test case as output.
existing models. Finally, the model uses this training data to prioritize new
test cases based on their predicted priority level [1].
III. S OFTWARE T ESTING & A RTIFICIAL I NTELLIGENCE
Software Testing is a process to evaluate the software and IV. S YSTEMATIC R EVIEW
identify defects [11]. It is crucial for software to work or A review is a systematic study that helps to identify the
perform as per requirements but it is natural having bugs existing work, research question improvement scope, and
or defects in software. The bugs can be generated during existing empirical studies [18]. In this study, 20 research
development, bug fixing, feature addition, code refactoring, papers have been reviewed from the past 7 years which
and even during software maintenance [12]. Therefore, it is have been collected from 6 different databases such as
obvious for the development team to test the software under ScienceDirect, IEEE, SCITEPRESS, ACM, Wiley Online
different scenarios before releasing it to the client. There are Library and MDPI.
different strategies and techniques for software testing. Based
on the nature of the software it would be decided which A. Eligibility Criteria
software testing technique should be used [13]. Software Eligibility criteria for selecting articles for a systematic
testing techniques are very tedious and automation comes literature review include relevance to the research questions,
here to ease the process. How AI can automate software publication time frame, language, publisher, and study design
testing and why it is getting more acceptance than any other [19]. For this systematic review, after filtering we have
technique will be discussed in this section. AI is a broad selected 20 articles out of 40 articles from the last 7 years.
area that encompasses various subareas, and ML is one of We only selected articles relevant to software testing using
the most prominent and widely applied subareas within AI. machine learning.
In this paper, we discuss software testing using Machine
Learning (ML). We also focus on software testing using Deep B. Search String Strategy
Learning (DL). After lots of searching in the research databases and
google scholar, we found many articles about software testing
A. Software Testing Using Machine Learning using machine learning but only the most relevant articles
Machine Learning (ML) is a process where machines learn were selected. In this process, we have used the advanced
from data using algorithms and can further predict or make search string strategy [20]. In this search string strategy,
decisions based on the data [14]. The data-centric learning Boolean operators (AND, OR, NOT) have been used to
approach has made Machine Learning powerful and widely combine and exclude keywords in the search query.
accepted in different areas including the software industry. Our search string was [(”Software Testing” AND ”Ma-
Figure 1 shows the general approach to apply Machine chine Learning”) AND (”Testing Automation Technique” OR
Learning algorithms in software testing. There are different ”Deep Learning” OR ”Black-box Testing” OR ”Integration
activities in software testing like bug detection, generating Testing” OR ”Metamorphic Testing” OR ”White Box Test-
test data, test case generation, test optimization, API testing, ing”) NOT (”Manual Testing” OR ”Adhoc testing”)].
etc [?]. Apart from the search string strategy, one can use titles,
Bug Prediction using ML: Bug prediction can be per- keywords, or abstracts to find out relevant publications. The
formed using machine learning. ML algorithms analyze purpose of this study is to review the effective application of
software code and predict the likelihood of future bugs in AI (Machine Learning, Deep Learning) in software testing

979-8-3503-0219-6/23/$31.00 ©2023 IEEE 524

 Fig. 1. A General Approach to Apply ML Techniques in Software Testing

TABLE I TABLE II
I NCLUSION AND E XCLUSION C RITERIA S ELECTED R ESEARCH S TUDIES ACCORDING TO THE P UBLISHER

Criteria Criteria Publisher Name # Research Articles

Area
for Inclusion for Exclusion IEEE 8
Article Research article, Poster, ACM 6
type SLR Book MDPI 2
Software testing, Wiley Online Library 1
Machine learning, Science Direct 2
Keywords
testing automation SCITEPRESS 1
Searched other than
technique,
keywords ones in
Test data generation,
“Inclusion criteria”
Blackbox testing,
Whitebox testing D. Data Extraction
Software testing,
Software Data extraction means the process of retrieving relevant
Area excluding
Interest of area Engineering,
“Inclusion criteria”
data from various sources for a specific purpose, such as a
Artificial literature review [21]. In the context of software testing using
Intelligence
Languages machine learning and deep learning, data extraction may
Language English involve searching through academic journals, conference
except English
Time proceedings, and other sources to gather information on the
2015 -2022 Before 2015
period
latest developments and trends in software testing using ML
and DL. This information can then be used to summarise
a comprehensive review of the current state of the field,
by developing research questions, collecting and selecting identify gaps in existing knowledge, and provide insights into
proper relatable studies through filtering methods. By ex- future directions for research and practice. Table II shows the
amining the existing literature and answering the research details of the selected number of studies and their publishers.
questions, we aim to provide the best current practices for
software testing. V. RESULTS
This section provides insights into state-of-the-art tech-
niques and their effectiveness in improving the quality and
C. Data Screening and Analysis efficiency of software testing using machine learning and
deep learning. The review aims to provide a comprehensive
Each paper examines different aspects of applications of synopsis of the existing research in this domain by analyzing
ML, DL techniques in software testing. In most studies, the a number of studies. We have selected 20 studies for the
authors compare different ML and DL models based on their study. The details findings of these selected studies have been
performance and identify the best results they could generate presented in table III. We also investigated the answer of the
from those models based on the subject criteria and expected research questions from the relevant research papers.
outcome. For the collection process, we have purposively RQ1: Does manual testing have drawbacks?
identified 40 research papers that are related to ML, DL, and Manual testing has several drawbacks. Some of the draw-
software testing by searching keywords in google scholar. backs of manual testing are it is time-consuming, it does not
We also used a backward snowballing method where we cover all possible scenarios and use cases, it is costly, it is
checked the references of the selected papers and identified susceptible to human errors and it can not reproduce test
20 papers. After the collection of papers, we started the cases accurately [22]. Machine Learning and Deep Learning
screening process where by reading the title we would be can help to overcome the mentioned drawbacks of manual
able to differentiate whether the topic being addressed is testing. ML and DL can automate the testing process. By
relatable or not. Table I shows lists of inclusion and exclusion leveraging the power of algorithms, more accurate testing
criteria in detail during paper selection for this study. can be performed [23].
The paper selection process involved sorting based on RQ2: Can integration of AI (ML or DL techniques)
eligibility criteria with a focus on the automation of software in software testing help to overcome the drawbacks of
testing using machine learning and deep learning techniques. manual testing?
After filtering, 20 research papers were selected for the litera- Integration of ML and DL techniques in software testing
ture review study. The search strategy consisted of 5 stages: can help to overcome the drawbacks of manual testing by
identification of the research topic, screening, selection of improving the efficiency, accuracy, and effectiveness of the
eligible papers, and final inclusion of research articles. testing process. ML and DL algorithms can be trained to

979-8-3503-0219-6/23/$31.00 ©2023 IEEE 525

 TABLE III
S UMMARY OF THE S ELECTED S TUDIES

Publisher
SL Paper Id Year Findings
Name
Authors proposed an approach utilizing Deep Reinforcement
1 P1 [27] 2022 ACM Learning (RL) for automating the exploration of Android apps. Authors
developed a tool called ARES along with FATE that integrates with ARES.
This paper analyzed ML frameworks in the context of software automation and
2 P2 [28] 2022 MDPI evaluated the performance of testing tools considering various factors. Accuracy
or error rate, scope are important factors to determine the effectiveness of frameworks.
This study investigates the efficacy of machine learning, data mining,
Science
3 P3 [29] 2022 and deep learning methodologies in predicting software faults. This investigation reveals that
Direct
data mining and machine learning techniques are utilized more than deep learning techniques.
This paper introduces Keeper, a novel testing tool. Keeper adopts a unique approach where it
4 P4 [30] 2022 ACM
creates pseudo-inverse functions for ML APIs. Keeper significantly enhances branch coverage .
This study presents DeepOrder, a regression machine learning model based on deep
5 P5 [31] 2021 IEEE learning techniques. DeepOrder can prioritize test cases and identify failed test cases when
it considers various factors such as test case duration and execution status.
This study investigated reward function and reward strategy within the context of
Science
6 P6 [32] 2021 continuous integration (CI) testing. The authors proposed three strategies in terms
Direct
of the reward strategy. Proposed strategies showed promising results.
This paper introduces Deep GUI. Deep GUI utilizes deep learning techniques to create
7 P7 [5] 2021 IEEE
a model of valid GUI interactions, based solely on screenshots of applications.
This study finds that most ML libraries lack a high-quality unit test suite. Moreover, the study
8 P8 [33] 2021 IEEE
also discovers recurring trends in the unexamined code throughout the five assessed ML libraries.
This study presents a deep learning approach to predict the validity of test inputs
9 P9 [34] 2021 IEEE
for RESTful APIs. The proposed network achieved 97% accuracy for the new APIs.
This paper introduces Humanoid, a deep learning approach for generating
GUI test inputs by leveraging knowledge gained from human interactions. It learns
10 P10 [35] 2019 IEEE
from traces of interactions generated by humans, enabling the automatic prioritization
of test inputs based on their perceived importance to users.
This study finds equivalent mutants are effective for augmenting data
11 P11 [36] 2019 ACM
and improving the detection rate of metamorphic relations.
This study introduces an enhanced CNN model specifically designed to improve
the learning of semantic representations from source-code. This study also showed
12 P12 [37] 2019 MDPI
enhancements of the global pattern capture capability of the models which improve
the model’s generalization performance.
This study used three supervised machine learning algorithms for predicting software
bugs. To enhance the accuracy of models, random forest
13 P13 [38] 2019 IEEE
ensemble classifiers have been used. The developed models effectively work for various
scenarios.
This study finds ML algorithms have predominantly been employed in different areas
14 P14 [39] 2019 IEEE of software testing. Test case generation, evaluation, test oracle construction,
and cost predicton for testing activitires can be performed using ML.
This study presents an approach for automating the test oracle mechanism in software
15 P15 [40] 2018 ACM using machine learning (ML). By incorporating a captured component into the application,
historical usage data have been gathered. These data later generate an appropriate oracle.
This paper describes a tool that generates test data for programs.
SCITE
16 P16 [41] 2018 The tool operates by clustering input data from a corpus folder and creating
PRESS
generative models for each cluster. These models are recurrent neural networks.
This paper introduces a methodology called DaOBML, which offers tool support to
17 P17 [42] 2018 ACM enhance the quality of environmental models that generate complex artifacts like images
or plots. In this study, among six ML algorithms, ANN shows the best performance.
This study introduces DeepXplore, an innovative whitebox system designed to
18 P18 [43] 2017 ACM systematically test DL systems and detect faulty behaviors. DeepXplore can
solve joint optimization problems.
Wiley This study, proposed a ML approach that can predict metamorphic relations in
19 P19 [44] 2016 Online software programs. To achieve this, authors utilized a graph-based representation
Library of the program.
This study proposed an approach for prioritizing test cases in manual testing. The proposed
20 P20 [45] 2016 IEEE approach considers black-box metadata, including test case history. SVM Rank ML algorithm
is used in this study.

979-8-3503-0219-6/23/$31.00 ©2023 IEEE 526

 TABLE IV research databases using an advanced search strategy. Ini-
T ESTING ACTIVITIES AUTOMATED BY ML & DL tially, 40 articles were retrieved, and after a rigorous filtering
Software Testing Activity Total process, 20 articles were chosen for analysis.
Test Case Generation 4 Based on the findings of the selected papers, it is evi-
Defect Prediction 3 dent that AI (Machine Learning and Deep Learning) can
Test Case Prioritization 3
Metamorphic Testing 2 successfully automate several testing tasks. These tasks in-
Android Testing 2 clude Test Case Generation, Defect Prediction, Test Case
Test Case Validation 1 Prioritization, Metamorphic Testing, Android Testing, Test
White Box Testing 1
Case Validation, and White Box Testing. The integration of
AI in software testing is shown to simplify testing activities
and enhance performance. In the future, incorporating AI
automate repetitive testing tasks, which reduces the required
(machine learning and deep learning) techniques in different
effort for manual testing. This improves the efficiency of
testing activities will make it easier to perform testing
the testing process and enables faster testing. ML and DL
activities. A limited number of studies have been examined
algorithms can also analyze large amounts of data which help
in this study, which is a limitation. Conducting a review of
to identify defects in the software system. Identification of
a larger number of studies would provide the opportunity to
the defects improve the accuracy of the testing. Apart from
gain more deeper insights.
that , ML algorithms can generate test cases using historical
data or existing code, optimize the testing by prioritizing test R EFERENCES
cases [24].
[1] Sharif, A., Marijan, D. and Liaaen, M., 2021, September. DeepOrder:
RQ3: What software testing tasks can be automated Deep learning for test case prioritization in continuous integration test-
by AI (ML or DL) ? ing. In 2021 IEEE International Conference on Software Maintenance
Machine Learning and Deep Learning techniques can and Evolution (ICSME) (pp. 525-534). IEEE.
[2] Yang, Y., Li, Z., He, L. and Zhao, R., 2020. A systematic study of
automate different types of software testing tasks such as test reward for reinforcement learning based continuous integration testing.
results analysis, test case prioritization, defect prediction, test Journal of Systems and Software, 170, p.110787.
execution, test case evaluation, test case refinement, testing [3] Romdhana, A., Merlo, A., Ceccato, M. and Tonella, P., 2022. Deep
reinforcement learning for black-box testing of android apps. ACM
cost estimation, test oracle construction, identification of Transactions on Software Engineering and Methodology (TOSEM),
metamorphic relations, and test case generation [24]. Table 31(4), pp.1-29.
IV shows testing activities that can be automated by machine [4] Li, Y., Yang, Z., Guo, Y. and Chen, X., 2019, November. Humanoid:
A deep learning-based approach to automated black-box android
learning app testing. In 2019 34th IEEE/ACM International Conference on
RQ4: What techniques do researchers use to assess Automated Software Engineering (ASE) (pp. 1070-1073). IEEE.
AI (ML or DL) when used in software testing? [5] YazdaniBanafsheDaragh, F. and Malek, S., 2021, November. Deep
GUI: black-box GUI input generation with deep learning. In 2021
Researchers consider different performance matrices to 36th IEEE/ACM International Conference on Automated Software
assess ML algorithms when used in software testing. The Engineering (ASE) (pp. 905-916). IEEE.
performance matrices are cross-validation, accuracy, preci- [6] Kanewala, U., Bieman, J.M. and Ben-Hur, A., 2016. Predicting meta-
morphic relations for testing scientific software: a machine learning
sion, recall, receiver operating characteristic (ROC) curve, approach using graph kernels. Software testing, verification and relia-
area under the curve (AUC), and f1 score [25]. The column bility, 26(3), pp.245-269.
total represents the number of papers where these testing [7] Braga, R., Neto, P.S., Rabêlo, R., Santiago, J. and Souza, M., 2018,
September. A machine learning approach to generate test oracles.
activities have been automated by machine learning. In Proceedings of the XXXII Brazilian Symposium on Software
Precision: Precision is a statistical measure that quantifies Engineering (pp. 142-151).
the ratio of true positive instances out of the total positive [8] Nair, A., Meinke, K. and Eldh, S., 2019, August. Leveraging mutants
for automatic prediction of metamorphic relations using machine
predictions made. [26]. learning. In Proceedings of the 3rd ACM SIGSOFT International
Recall: Recall is a statistical indicator utilized to quantify Workshop on Machine Learning Techniques for Software Quality
the fraction of true positive outcomes within the entirety of Evaluation (pp. 1-6).
[9] Singh, P.D. and Chug, A., 2017, January. Software defect prediction
actual positive instances [26]. analysis using machine learning algorithms. In 2017 7th Interna-
ML and DL algorithms have shown promising results tional Conference on Cloud Computing, Data Science Engineering-
to automate software testing tasks. Some of the promising Confluence (pp. 775-781). IEEE.
[10] Pan, C., Lu, M., Xu, B. and Gao, H., 2019. An improved CNN model
algorithms are Neural networks, Decision Tree, Support for within-project software defect prediction. Applied Sciences, 9(10),
vector machines, and Random Forest, . p.2138.
[11] Myers, G.J., Sandler, C. and Badgett, T., 2011. The art of software
VI. CONCLUSIONS testing. John Wiley Sons.
[12] Tan, L., Liu, C., Li, Z., Wang, X., Zhou, Y. and Zhai, C., 2014. Bug
Software testing plays a key role in the development characteristics in open source software. Empirical software engineer-
ing, 19, pp.1665-1705.
of software. However, as software systems become more [13] Jamil, M.A., Arif, M., Abubakar, N.S.A. and Ahmad, A., 2016,
complex, traditional manual testing methods are becoming November. Software testing techniques: A literature review. In 2016
less practical. There has been growing interest in leveraging 6th international conference on information and communication tech-
nology for the Muslim world (ICT4M) (pp. 177-182). IEEE.
AI for software testing. The aim of this study is to compre- [14] Jordan, M.I. and Mitchell, T.M., 2015. Machine learning: Trends,
hensively explore the current state of AI in software testing. perspectives, and prospects. Science, 349(6245), pp.255-260.
This review examines various approaches, techniques, and [15] Efendioglu, M., Sen, A. and Koroglu, Y., 2018. Bug prediction
of systemc models using machine learning. IEEE Transactions on
tools employed in this field, assessing their effectiveness. The Computer-Aided Design of Integrated Circuits and Systems, 38(3),
articles selected for this study were obtained from different pp.419-429.

979-8-3503-0219-6/23/$31.00 ©2023 IEEE 527

[16] Hammouri, A., Hammad, M., Alnabhan, M. and Alsarayrah, F., 2018. [38] Immaculate, S.D., Begam, M.F. and Floramary, M., 2019, March.
Software bug prediction using machine learning approach. Interna- Software bug prediction using supervised machine learning algorithms.
tional journal of advanced computer science and applications, 9(2). In 2019 International conference on data science and communication
[17] Zhang, D., 2006, November. Machine learning in value-based software (IconDSC) (pp. 1-7). IEEE.
test data generation. In 2006 18th IEEE International Conference on [39] Durelli, V.H., Durelli, R.S., Borges, S.S., Endo, A.T., Eler, M.M., Dias,
Tools with Artificial Intelligence (ICTAI’06) (pp. 732-736). IEEE. D.R. and Guimarães, M.P., 2019. Machine learning applied to software
[18] Kitchenham, B. and Brereton, P., 2013. A systematic review of sys- testing: A systematic mapping study. IEEE Transactions on Reliability,
tematic review process research in software engineering. Information 68(3), pp.1189-1212.
and software technology, 55(12), pp.2049-2075. [40] Braga, R., Neto, P.S., Rabêlo, R., Santiago, J. and Souza, M., 2018,
[19] Papaioannou, D., Sutton, A. and Booth, A., 2016. Systematic ap- September. A machine learning approach to generate test oracles.
proaches to a successful literature review. Systematic approaches to In Proceedings of the XXXII Brazilian Symposium on Software
a successful literature review, pp.1-336. Engineering (pp. 142-151).
[20] Mohamed Shaffril, H.A., Samsuddin, S.F. and Abu Samah, A., 2021. [41] Paduraru, C. and Melemciuc, M.C., 2018, July. An Automatic Test
The ABC of systematic literature review: the basic methodological Data Generation Tool using Machine Learning. In ICSOFT (pp. 506-
guidance for beginners. Quality & Quantity, 55, pp.1319-1346. 515).
[21] Jonnalagadda, S.R., Goyal, P. and Huffman, M.D., 2015. Automating [42] de Santiago, V.A., da Silva, L.A.R. and de Andrade Neto, P.R.,
data extraction in systematic reviews: a systematic review. Systematic 2018, September. Testing environmental models supported by machine
reviews, 4(1), pp.1-16. learning. In Proceedings of the III Brazilian Symposium on Systematic
[22] Leitner, A., Ciupa, I., Meyer, B. and Howard, M., 2007, January. and Automated Software Testing (pp. 3-12).
Reconciling manual and automated testing: The autotest experience. [43] Pei, K., Cao, Y., Yang, J. and Jana, S., 2017, October. Deepxplore:
In 2007 40th Annual Hawaii International Conference on System Automated whitebox testing of deep learning systems. In proceedings
Sciences (HICSS’07) (pp. 261a-261a). IEEE. of the 26th Symposium on Operating Systems Principles (pp. 1-18).
[23] Zhang, J.M., Harman, M., Ma, L. and Liu, Y., 2020. Machine [44] Kanewala, U., Bieman, J.M. and Ben-Hur, A., 2016. Predicting meta-
learning testing: Survey, landscapes and horizons. IEEE Transactions morphic relations for testing scientific software: a machine learning
on Software Engineering, 48(1), pp.1-36. approach using graph kernels. Software testing, verification and relia-
[24] Durelli, V.H., Durelli, R.S., Borges, S.S., Endo, A.T., Eler, M.M., Dias, bility, 26(3), pp.245-269.
D.R. and Guimarães, M.P., 2019. Machine learning applied to software [45] Lachmann, R., Schulze, S., Nieke, M., Seidl, C. and Schaefer, I.,
testing: A systematic mapping study. IEEE Transactions on Reliability, 2016, December. System-level test case prioritization using machine
68(3), pp.1189-1212. learning. In 2016 15th IEEE International Conference on Machine
[25] Song, Q., Guo, Y. and Shepperd, M., 2018. A comprehensive investiga- Learning and Applications (ICMLA) (pp. 361-368). IEEE.
tion of the role of imbalanced learning for software defect prediction.
IEEE Transactions on Software Engineering, 45(12), pp.1253-1269.
[26] Tatbul, N., Lee, T.J., Zdonik, S., Alam, M. and Gottschlich, J., 2018.
Precision and recall for time series. Advances in neural information
processing systems, 31.
[27] Romdhana, A., Merlo, A., Ceccato, M. and Tonella, P., 2022. Deep
reinforcement learning for black-box testing of android apps. ACM
Transactions on Software Engineering and Methodology (TOSEM),
31(4), pp.1-29.
[28] Fatima, S., Mansoor, B., Ovais, L., Sadruddin, S.A. and Hashmi,
S.A., 2022. Automated Testing with Machine Learning Frameworks:
A Critical Analysis. Engineering Proceedings, 20(1), p.12.
[29] Batool, I. and Khan, T.A., 2022. Software fault prediction using data
mining, machine learning and deep learning techniques: A system-
atic literature review. Computers and Electrical Engineering, 100,
p.107886.
[30] Wan, C., Liu, S., Xie, S., Liu, Y., Hoffmann, H., Maire, M. and Lu, S.,
2022, May. Automated testing of software that uses machine learning
apis. In Proceedings of the 44th International Conference on Software
Engineering (pp. 212-224).
[31] Sharif, A., Marijan, D. and Liaaen, M., 2021, September. DeepOrder:
Deep learning for test case prioritization in continuous integration test-
ing. In 2021 IEEE International Conference on Software Maintenance
and Evolution (ICSME) (pp. 525-534). IEEE.
[32] Yang, Y., Li, Z., He, L. and Zhao, R., 2020. A systematic study of
reward for reinforcement learning based continuous integration testing.
Journal of Systems and Software, 170, p.110787.
[33] Wang, S., Shrestha, N., Subburaman, A.K., Wang, J., Wei, M. and
Nagappan, N., 2021, May. Automatic unit test generation for machine
learning libraries: How far are we?. In 2021 IEEE/ACM 43rd Interna-
tional Conference on Software Engineering (ICSE) (pp. 1548-1560).
IEEE.
[34] Mirabella, A.G., Martin-Lopez, A., Segura, S., Valencia-Cabrera, L.
and Ruiz-Cortés, A., 2021, June. Deep learning-based prediction of
test input validity for RESTful APIs. In 2021 IEEE/ACM Third
International Workshop on Deep Learning for Testing and Testing for
Deep Learning (DeepTest) (pp. 9-16). IEEE.
[35] Li, Y., Yang, Z., Guo, Y. and Chen, X., 2019, November. Humanoid:
A deep learning-based approach to automated black-box android
app testing. In 2019 34th IEEE/ACM International Conference on
Automated Software Engineering (ASE) (pp. 1070-1073). IEEE.
[36] Nair, A., Meinke, K. and Eldh, S., 2019, August. Leveraging mutants
for automatic prediction of metamorphic relations using machine
learning. In Proceedings of the 3rd ACM SIGSOFT International
Workshop on Machine Learning Techniques for Software Quality
Evaluation (pp. 1-6).
[37] Pan, C., Lu, M., Xu, B. and Gao, H., 2019. An improved CNN model
for within-project software defect prediction. Applied Sciences, 9(10),
p.2138.

979-8-3503-0219-6/23/$31.00 ©2023 IEEE 528