Comparative Analysis of Software Reliability

Prediction Using Machine Learning and Deep

Learning
1st Akshat Jindal 2nd Ashi Gupta 3rd Rahul
Department of Software Engineering Department of Software Engineering Department of Software Engineering
Delhi Technological University Delhi Technological University Delhi Technological University
New Delhi, India New Delhi, India New Delhi, India
[email protected] [email protected] [email protected]

Abstract— Software Reliability is an integral part to Networks (RNNs), and others that use a collection of
determine Software Quality. Software is considered to be of high algorithms to replicate the brain's actions. A neural network
quality if its reliability is high. There exist many statistical is made up of four primary components: inputs, weights, a
models that can help in predicting Software Reliability, but it is bias or threshold, and an output. However, a deep learning
very difficult to consider all the real-world factors and hence it
makes the task of reliability prediction very difficult. Therefore,
model requires more data points to enhance its accuracy,
it becomes more challenging for the IT industry to predict if a although a machine learning model requires less data due to
software is dependable or not. Machine Learning and Deep its fundamental data structure.
Learning can be used for the prediction of Software Reliability
by programming a model that assesses reliability by fault For this experiment, we propose a heuristic examination of
prediction in a more meticulous manner. Therefore, in this several Machine Learning and Deep Learning techniques on
study we intend to use predefined Artificial Intelligence a univariate software failure time series data to investigate
algorithms, mainly Artificial Neural Network (ANN), Recurrent which approach can be used extensively for predicting
Neural Network (RNN), Gated Recurrent Unit (GRU) and Long software reliability. We then used metrics like Mean
Short-Term Memory (LSTM) for predicting software reliability Absolute Error, Mean Squared Error, Median Absolute Error,
on a time series software failure dataset and compare them on and Maximum Error to determine their accuracies. The
the basis of selected performance metrics. Each of the algorithm
reason for choosing these metrics is that they can capture
trained on software failure dataset will be used to predict the
software failure time after a certain number of corrective which technique closely represents the actual software failure
modifications are performed on the software. Based on the dataset. For this paper we chose to lay an impact on the data-
result of the studies, it is discovered that LSTM produces driven approaches compared to the hardware/architecture-
superior outcomes in predicting the software failure trend as it based approaches where other factors like the environment
can capture long and short-term trends in the software failure and time play a significant role and leave a major
dataset. probabilistic factor.
Keywords— Software Reliability, Deep Learning, Time Series II. RELATED WORKS
data, comparative analysis
The most prevalent Machine Learning algorithms used for
Software Reliability Prediction and Modeling include
I. INTRODUCTION Genetic Programming, Decision Trees, Support Vector
Machines, and Particle Swarm Optimization.ML techniques
The term "software reliability" refers to operational have been proven to work better than stochastic ones due to
dependability. Software reliability may alternatively be the nature of models to learn from previous eros are thereby
defined as the probability that a software system will perform leading to close precision and fewer errors (Malhotra and
its assigned task in a given environment for a certain number Negi 2013) [1].
of input cases, assuming that the hardware and input are both
error-free. It is of crucial importance to evaluate Software In order to facilitate the acceptance of connectionist models
Reliability for determining system dependability. However, and their usage in software reliability models, Cai et al.
it is difficult to accomplish reliability given the increasing (1991) [2] and Karunanithi et al. (1992) [3] undertook
complexity in software requirements. considerable research. G Krishan et alii. (2018) [4] compared
mainly Artificial Neural Networks and SVM to conclude that
Machine Learning and Deep Learning techniques can predict geometric understanding leads to better results for SVM than
the fault rate for a given software more precisely by learning NN’s.
on past input data without human judgment thus leaving less
room for errors and assumptions contrary to statistical Prediction tests to oversee software reliability were utilized
methods. (Malhotra and Negi 2013) [1]. System behavior can by Pai and Hong (2006) [5] by using SVM algorithms. Loui
be anticipated by utilizing Machine Learning which learns et alii. (2016) [6] employed a relevance vector machine for
from its past and current software failure data as it’s a tool to the prediction of software dependability. Machine learning
automate data processing. Deep learning techniques include approaches such as fuzzy inference systems, cascade
Artificial Neural Networks (ANNs), Recurrent Neural correlation neural networks, and decision trees are used by
Kumar and Singh (2012) [7] to predict outcomes. Jaiswal and
Malhotra (2016) [8] talk about predicting software reliability Table.1 Dataset description
using ANFIS. Other methods like Bagging, GRNN, SVM,
MLP, M5P, FFBPNN, CFBPNN, Lin Reg, RepTree are Attribute Description
studied by Xingguo and Yanhua (2007) [9]
t The number of modifications made to the
In papers like (Gokhale 1998) [10], an application was run software
against some software and fixed test cases to figure out a
scientific model (architecture of application) in terms of Yt Failure time after t modification is made
criteria like branching probabilities and failure model of its
components issues that could be later solved for better
reliability. Future works have been shown to find a
systematic way of predicting software reliability by
incorporating debugging functionalities (S Trivedi 2006)
[11].

However, with moving towards Machine Learning and Deep

Learning techniques these tools and methods have been
outperformed by the data gathered. In his research, NR Kiran
(2008) [12] presented a unique soft computing-based
technique that employs a non-linear ensemble trained using
Back Propagation Neural Networks. A similar notion was
observed by (Kumar and Jayaram 2014) [13], who used
Artificial Neural Networks and Genetic Algorithms to
forecast Software Reliability.

(Amin, Grunske, and Colman, 2013) [14] proposed using

ARIMA modeling to solve the unrealistic assumptions,
environment-dependent applicability, and questionable
predictability associated with Software Reliability Growth
Fig.1 Dataset visualization: representing relationship between the time
Models (SRGMs), and showed that their method performed after which software fails to the number of modications made in the
better due to the data-driven approach used. software.

III. PROBLEM STATEMENT

The main goal of the project is to predict Software Reliability

using various Machine Learning and Deep Learning
algorithms and to compare and analyze the performance of
the chosen algorithms. The chosen algorithms are trained on
a software failure dataset to determine when the given
software would malfunction based on its relationship
between previous failures and modifications. The goal of
modeling software dependability is to determine the
likelihood of a piece of software failing in a given
environment. We have also structured a Theoretical
justification for the results of comparison for Machine and
Deep Learning algorithms.

Fig.2 Dataset visualization: The Auto Correlation of Software Failure

Time with Lag to measure the linear relationship between an observation at
IV.EXPERIMENTAL DESIGN number of modification t and the observations at previous modifications

A. Dataset B. Data Transformation

• We have used the Software Failures Dataset from [6]. • The dataset is a time series dataset. Before application
The dataset has a total of 101 samples. There are 2 of any ML/DL algorithm, it needs to be transformed
attributes in the dataset as shown in Table 1. into a suitable representation of the time series.
• The variable t as shown in the table below is taken to • We transformed it into an 86 x 16 shaped dataset
be the cumulative number of corrective changes where each row contains the current failure time and
performed on the software, after each failure. along with it a sequence of past 15 failure time values.
 • We then have divided the dataset into training,
validation and test set as shown in Table 2. This reduces the complexity and helps in memorizing the
Table.2 Dataset bifurcation
previous layer outputs.

Dataset Partition Percentage No. of Samples

Training Set 70% 61

Validation Set 10% 7

Test Set 20% 18

C. ALGORITHMS USED

1. ANN (Artificial Neural Network)

The ANN algorithm is a machine learning method that is

based on the structure of the human brain and is one of Fig. 4 RNN Model Summary
the most commonly used for regression and
classification issues.

Fig 3 shows it consists of three layers: an input layer, an Reason for selecting RNN: Since the hidden state of
arbitrary number of hidden levels, and an output layer. RNN is used for remembering information about a
sequence, it can be used for a time series prediction
problem.

Training epochs: 100, Batch Size:5

3. GRU (Gated Recurrent Unit)

GRU is a version of RNN that is designed to solve

disappearing or exploding gradient issues since the
model does not lose input from the current cell while also
transmitting important information to the next cell.
GRU decides which information should be sent to the
output for prediction by employing two vectors to
determine what information should be sent to the output
for prediction.

GRU consists of 3 gates:

Fig. 3 ANN Model Summary ● Update Gate: This gate determines how much
information should be handed on to future
generations.
Reason for selecting ANN: To get a baseline prediction
performance. ● Reset Gate: It determines how much past
knowledge is useless and hence can be forgotten.
Training epochs: 100, Batch Size:5
● Current Memory Gate: This gate is further
2. RNN (Recurrent Neural Network) incorporated into the reset gate and brings non-
linearity in the input.
The RNN algorithm is a type of Neural Network. The
variation in this algorithm is that it has a hidden state,
which takes into consideration the information stored in
a sequence as represented in Fig 4. In RNN, the
independent activations are converted into dependent
activations, which is done by using the same weights and
biases for all the layers.
 Fig. 5 GRU Model Summary Fig 6. LSTM Model Summary

Reason for selecting GRU: Since GRU is a variation of Reason for selecting LSTM: Since LSTM was specially
RNN only, hence it can be effective in a time series developed to solve the problem of long-term
prediction problem by virtue of its capability to dependencies in a sequence, it can be very effective in a
memorize some information about a sequence. time series prediction problem.

Training epochs: 100, Batch Size:5 Training epochs: 100, Batch Size: 5

4. LSTM (Long and Short term Memory)

V. RESULTS
LSTM is another variation of RNN. LSTM was created
to address the issue of long-term dependencies while also Fig. 7, Fig. 8, Fig. 9 and Fig. 10 are the graphs plotted for the
preserving information over several timestamps of input performance of the different algorithms/models on the given
data. To store information, it comprises a chain-like dataset and compared with the actual failure time values.
structure with four neural networks and cells called From the graphs the results are not very obvious therefore a
memory blocks as represented in Fig. 6. When there are more mathematical formulation of the results is needed which
long-term dependencies in a data series, the efficiency of can be provided by performing an error analysis on the results
RNN decreases. obtained.
Error analysis also helps us to represent the algorithm’s
These issues are addressed by LSTM's three gates: performance and compare with other algorithms. In this paper
we have chosen mean absolute error, mean squared absolute
i) Input Gate: It adds useful information to a cell error, median squared error and maximum error as described
state. in the following section. The following metrics have been
chosen as they help to model regression problems where
ii) Forget Gate: It is used to forget long-term the output is a real or continuous value i.e., software
information that is not required anymore. failure time in our case as well as serves as an L1 loss
function.
iii) Output Gate: It is used for extracting useful
information from the current cell.

Fig 7. Prediction for ANN on Test Set

 D. Maximum Error: This metric talks about the error
arising from difference between absolute predicted and
actual observation i.e., software failure time.

VII. RESULTS JUSTIFICATION

• It can be inferred from Table 3, and from Fig. 11 that

LSTM has the least Mean Absolute Error, Mean
Squared Error, and Maximum Error amongst the chosen
4 algorithms.

• The errors in the predictions of RNN and GRU are very

close to each other and are also close to the baseline
Fig 8. Prediction for RNN on Test Set errors as seen by Table 3.

• Software Reliability Prediction is a typical time series

problem. It can have long and short-term trends that
must be learned by any model for making decent
predictions.

• Although RNN and GRU have memory units that can

remember some information about a sequence, their
performance efficiency gets decreased when there are
long-term dependencies in the input data sequence and
therefore performed poorly when compared to LSTM.

• LSTM is specially developed to capture Long and Short

term trends in the input data sequence. Therefore,
LSTM outperformed all the algorithms and gave
Fig 9. Prediction for GRU Test Set relatively low errors in the reliability prediction task.

Table 3. Compiled error scores for all different models

Mean Mean Median Maximum

Absolute Squared Absolute Error
Error Error Error

ANN 1.6247 3.6756 1.3449 3.4827

RNN 1.6821 3.7328 1.6267 3.4486

GRU 1.6587 3.7310 1.5486 3.5465

LSTM 1.5639 3.4735 1.4992 3.4095

Fig 10. Prediction for LSTM on Test Set

VI. METRICS USED FOR MODEL EVALUATION.

A. Mean Absolute Error: For a given observation, the

provided metric calculates the average of the absolute
difference between the predicted and actual values

B. Mean Squared Error: This metric is used to compute the

average values for the square of difference between
values predicted by the algorithm and values actually
observed from the simulated domain.

C. Median Absolute Error: It is the amount of the absolute

difference between expected and actual values for a Fig 11. Graphical representation of error metrics i.e., mean absolute, mean
group of data that falls in the center. squared, median absolute and maximum error in all 4 algorithms
 VIII. CONCLUSION Systems and Software,Volume 86, Issue 7,2013,Pages 1923-
1932,ISSN 0164-1212, https://doi.org/10.1016/j.jss.2013.03.045.
We applied Machine Learning and Deep Learning [15]. Patel, Viral, Daanyaal Kapadia, Deval Ghevariya, and Shiburaj Pappu.
"All India Grievance Redressal App." Journal of Information
algorithms for Software Reliability Prediction. We Technology and Digital World 2, no. 2: 91-99.
analyzed, visualized, and transformed the dataset to make [16]. Chen, Joy Iong Zong, and Joy Iong Zong. "Automatic Vehicle License
it a time series prediction problem. Plate Detection using K-Means Clustering Algorithm and CNN."
Journal of Electrical Engineering and Automation 3, no. 1 (2021): 15-
We trained four models for software reliability prediction 23.
[17]. Mugunthan, S. R., and T. Vijayakumar. "Design of Improved Version
and reported their performance in terms of different sorts of Sigmoidal Function with Biases for Classification Task in ELM
of prediction errors. Domain." Journal of Soft Computing Paradigm (JSCP) 3, no. 02
(2021): 70-82.
Considering ANN as the baseline model, we compared [18]. Manoharan, Samuel. "Study on Hermitian graph wavelets in feature
the performance of all of the above-described models and detection." Journal of Soft Computing Paradigm (JSCP) 1, no. 01
(2019): 24-32
found out that LSTM outperformed all the algorithms by [19]. Tripathi, Milan. "Analysis of Convolutional Neural Network based
virtue of its ability to effectively capture Long and Short Image Classification Techniques." Journal of Innovative Image
term trends in a data sequence and gave a theoretical Processing (JIIP) 3, no. 02 (2021): 100-117.
justification of the results obtained. [20]. Rahul, Ayush, Divya Agarwal, Devika Vijay. "Genre Classification
using Character Networks" , 2021 5th International Conference on
Intelligent Computing and Control Systems (ICICCS), 2021
[21]. J.R. Horgan. "An analytical approach to architecture-based software
reliability prediction" , Proceedings IEEE International Computer
IX. REFERENCES Performance and Dependability Symposium IPDS 98 (Cat No
98TB100248) IPDS-98, 1998
[22]. Swagat Ranjit, Shruti Shrestha, Sital Subedi, Subarna Shakya.
[1]. Malhotra R, Negi A (2013) Reliability modeling using particle swarm "Comparison of algorithms in Foreign Exchange Rate Prediction" ,
optimisation. The society for reliability engineering, quality and 2018 IEEE 3rd International Conference on Computing,
operations management (SREQOM), India and The Division of Communication and Security (ICCCS), 2018
Operation and Maintenance, Lulea University of Technology, Sweden.
Int J Syst Assur Eng Manag. doi:10.1007/s13198-012- 0139-0
[2]. Cai YK, Wen YC, Zhang LM (1991) A critical review on software
reliability modeling. Reliab Eng Syst Saf 32(3):357–371
[3]. Karunanithi N, Whitley D, Malaiya Y (1992) Prediction of software
reliability using connectionist models. IEEE Trans Softw Eng
18(7):563–574
[4]. Mohan, G. & Yoshitha, N. & Lavanya, M.L.N. & Priya, A.. (2018).
Assessment and Analysis of Software Reliability Using Machine
Learning Techniques. International Journal of Engineering and
Technology (UAE). 7. 201-205. 10.14419/ijet.v7i2.32.15567.
[5]. Pai, F.P., Hong, C.W.: Software reliability forecasting by support
vector machines with simulated annealing algorithms. J. Syst. Softw.
79, 747–755 (2006)
[6]. Lou J, Jiang Y, Shen Q, Shen Z, Wang Z, Wang R (2016) Software
reliability prediction via relevance vector regression. Neuro-
computing 186:66–73
[7]. Kumar P, Singh Y (2012) An empirical study of software reliability
prediction using machine learning techniques. Int J Syst Assur Eng
Manag 3(3):194–208. doi:10.1007/s13198-012-0123-8
[8]. Arunima Jaiswal, & Ruchika Malhotra. (2016). Software Reliability
Prediction Using Machine Learning Techniques. Proceedings of Fifth
International Conference on Soft Computing for Problem Solving,
141–163. doi:10.1007/978-981-10-0448-3_12
[9]. Xingguo L, Yanhua S (2007) An early prediction method of software
reliability based on support vector machine. In: Proceedings
international conference on wireless communications, network- ing
and mobile computing (WiCom’07), pp 6075–6078
[10]. S. S. Gokhale, W. E. Wong, K. S. Trivedi and J. R. Horgan, "An
analytical approach to architecture-based software reliability
prediction," Proceedings. IEEE International Computer Performance
and Dependability Symposium. IPDS'98 (Cat. No.98TB100248), 1998,
pp. 13-22, doi: 10.1109/IPDS.1998.707705.
[11]. S. S. Gokhale and K. S. Trivedi, "Analytical Models for Architecture-
Based Software Reliability Prediction: A Unification Framework," in
IEEE Transactions on Reliability, vol. 55, no. 4, pp. 578-590, Dec.
2006, doi: 10.1109/TR.2006.884587.
[12]. N. Raj Kiran, V. Ravi, Software reliability prediction by soft
computing techniques, Journal of Systems and Software,Volume 81,
Issue 4,2008,Pages 576-583,ISSN 0164-1212,
https://doi.org/10.1016/j.jss.2007.05.005.
[13]. M.A, Jayaram & H.S.Vijayakumar, (2014). ON APPLICATIONS OF
SOFT COMPUTING ASSISTED ANALYSIS FOR SOFTWARE
RELIABILITY. International Journal of Innovative Research in
Advanced Engineering (IJIRAE). Volume 1.
[14]. Ayman Amin, Lars Grunske, Alan Colman,An approach to software
reliability prediction based on time series modeling,Journal of