ISSN 2255-9094 (online)

Information Technology and Management Science ISSN 2255-9086 (print)

December 2017, vol. 20, pp. 54–57
doi: 10.1515/itms-2017-0009
https://www.degruyter.com/view/j/itms

A Case Study: Software Defect Root Causes

1 2 3
Laila Bergmane , Jānis Grabis , Edžus Žeiris
1, 2
Riga Technical University, Latvia, 3 ZZ Dats Ltd., Latvia

Abstract – Software quality assurance to comply with user will improve the quality of the software. This requires a
requirements enables software development companies to be problem analysis of the causes of product defects.
competitive. Maintaining a high quality level requires continuous
A defect causal analysis has three key principles:
monitoring and development. If there are quality problems, the
company’s reputation is suffering and its costs increase because of  reducing defects to improve quality;
investing in time and eliminating the consequences of the  applying local expertise;
problems. The aim of the present article is to identify the most  focusing on systematic defects.
essential root causes of software defect. The e-service “Invoice The first principle says that we can improve software quality
Submission” of Riga City Municipality is used as an example. The by focusing on the prevention and early detection of defects.
results of the study can provide useful information for developing The second determines that the cause detection must involve a
improvement activities for e-service higher quality. The analysis is
based on the information that is available in the developer’s user
software development team that can explain why these defects
request database. The Ishikawa method is used to analyse the have occured. The third principle says that, with relatively small
causes of defects. investments, the focus on systemic defects can have a
significant impact on quality [6].
Keywords – Defect analysis, defect causes, software quality. To determine the context in this paper, the definition of the
“defect” of the ISTQB term is used – a flaw in a component or
I. INTRODUCTION system that can cause the component or system to fail to
A human being can make an error, which produces a defect perform its required function, e.g., an incorrect statement or
in a program code document. If a defect in the code is executed, data definition. A defect, if encountered during execution, may
the system may fail to function properly causing a failure. cause a failure of the component or system [11]. A program is
Defects in software, systems or documents may result in said to be buggy if that contains a large number of bugs, or bugs
failures, but not all defects do so. Defects occur because human that seriously interfere with its functionality [3].
beings are fallible and because there is time pressure, complex The quality requirements are determined by standards and
code, complexity of infrastructure, changing technologies, internal quality procedures of companies [8]. Software quality
and/or many system interactions [12]. must meet user requirements. There is no software which does
Correction of defects is costly and the cost increases not have a defect. Even in the case where no defects can be
exponentially with every subsequent stage. They also directly found in the software, this does not prove that they are not there.
affect software development cycle-time. Therefore, defect In the software development industry, both nationally and
prevention not only enables one to reduce costs but also globally, the competitiveness of a company is closely related to
minimises the development time [2]. its ability to develop high quality information technology
Development organisations that deliver software-based solutions. Therefore, quality related issues are important for any
systems have to face serious problems on how to control the software development company. Maintaining the quality level
progress of test activities and quality of software products according to user requirements requires continuous
throughout the project life cycle in order to estimate test management and preventive action.
completion criteria, and if the project will end on time [10]. E-service “Invoice Submission” is a service whose
Testing activities play a major role in quality assurance and availability is disturbed due to existing defects. The quality
their non-compliance with the requirements is often the reason problems indicate that there are systematic repetitions of
why users are dissatisfied with the product. software defects that significantly impact its full use for users.
The quality of software needs to be assured through a proper Misleading software adversely affects the company’s
development process. The development process must be reputation, and defect fixing and resources spent to fix these
improved on a regular basis according to the actual usage problems increase business costs, and instead of using the
feedback. If a bug is in software, in particular, it is necessary to resources to develop new solutions, they are used to resolving
investigate a root cause of the bug in order to work out a proper existing software defects. The aim of this study is to investigate
measure to prevent it from recurring. Many reasons contribute the root causes of defects. The e-service “Invoice Submission”
towards software bugs in the project such as product, process has been used for a case study. User request database provides
and project related reasons [3]. Companies that are not good data for identifying the root causes. User requests in the
enough to resolve defect causes, risk with their reputation, loss database are classified information. The causes of the defects
of customer loyalty and cost cutting effects. To mitigate these are given in a description way. Results of this paper can be used
risks, it is essential to perform improvement activities, which for continuous improvement activities in e-service quality
improvement.

©2017 Laila Bergmane, Jānis Grabis, Edžus Žeiris. 54

This is an open access article licensed under the Creative Commons Attribution License
(http://creativecommons.org/licenses/by/4.0), in the manner agreed with De Gruyter Open. Unauthenticated
Download Date | 1/12/18 1:40 AM
 Information Technology and Management Science
_______________________________________________________________________________________________ 2017/20

II. RELATED RESEARCH exceptional data. The authors concluded that programmers
There have been several studies which analysed root causes could not know all of exceptional data in advance [7].
of defects in different systems using one or more data sources. Almost all of the studies under review [1], [8], [9] indicated
A study published in 2001 investigated 40 incident cases of that the revealed defects were related to testing problems. Other
web site functionality failures and found out that 80 % of all reasons were related with change management, lack of
failures were software failures and human errors. A large knowledge, requirements faults, data processing problems, and
number of failures occured during routine maintenance, defects in code logic. Testing problems are typically related to
software upgrades and system integration. The authors of the testing process improvement activities. One of the most popular
paper could not find out whether these failures wetr mainly due options how to improve this process at companies is to use one
to system complexity, inadequate testing and/or poor of the test maturity models, such as TPI Next, CMMI or TMMi,
understanding of system dependecies. They also indicated that which helps companies evaluate the current testing situation
other significant causes of software failure were system and, based on the recommendations or guidelines suggested by
overload, resource exhaustion and complex fault recovery the models, move towards improving the testing process at the
routines [5]. company [14].
In a study about software development companies, in which
36 highly-qualified quality assurance testers and developers III. RESEARCH METHODOLOGY
from open source projects were interviewed, it was concluded The e-service “Invoice Submission” of Riga City Municipality
that the most common quality problems were due to the fact that was selected for the case study.
the tester did not have enough information about the software The e-service “Invoice Submission” is an online service that
to be tested. It was also mentioned that testing was rigorous, offers its users an electronic submission of invoices via the web
and no fair software quality assurance policy was available in service API, XML file uploads or manual invoice information
written form [8]. entry. The invoice data validation is against the XSD scheme,
In a study, using the defect classification approach, algorithm which provides solutions for expanding or limiting input data at
and functional type defects during the development process different levels.
were found late – during system integration and testing. The The development of the e-service project started in 2011 and
mistakes were related to human factors – individual errors and its development till today has been carried out in ten phases.
lack of domain knowledge about a specific industry and system The project team consists of project manager, tester, system
[1]. analyst and programmer. The project manager is the only one
A case study about C compiler from the GNU Compiler who works on the project from its beginning, but other team
Collection, which is an application consisting of over members have changed several times.
300 000 lines of codes and can be divided based on Within the present research, several questions have been
functionality into 13 well-defined components, showed that a formulated:
significant percentage of software failures were associated with  Why do the same defects repeat systematically?
changes that spread across the system, i.e., were due to  Does the technology used in the example solution affect
nonlocalized faults. The authors of the study also analysed fligh the quality of e-service?
software failures of NASA mission. The software includes  What are the most common causes of defects in the
multiple software applications, consisting of millions of lines of production environment?
code in over 8000 files. The authors of the paper states that the
most common sources of failures were requirements and coding The data analysed were obtained from the supplier quality
faults, each contributing to about 33 percent of the failures. management information system. All defect requests received
Requirement faults included incorrect, changed and missing from users that were classified as a “defect” were selected from
requirements. The third largest fault type was related to data the database. The selected requests in the database were
problems and it contributed to 14 percent of the failures. Design registered in the period of 5 April 2017–7 July 2017–. A total
faults led to less than 6 percent of the failures. Additionally, of 102 defect requests were received during this period. For
4 percent of failures were due to process or procedural issues, analysis, the authors used the resolved defects and the
2 percent – due to integration faults, and 1 percent – due to information provided by the programmer and tester about the
simulation or testing problems [9]. progress of defect handling and their causes. Each defect
A study, which focused only on failures caused by defects in mentioned in the request was assigned to an apropriate category
data-parallel programs, showed that most failures (84.5 %) (Table I). IEEE Standard Software Anomaly was used to
were caused by defects in data processing rather than defects in categorise the defects. They were classified by considering
code logic. The authors emphasised, “the tremendous data impact on requirement classes [4]. Functional defects were
volume and various dynamic data sources make data processing subdivided into the subcategories (categories 1–3). Thereafter,
error-prone”. They also concluded that 22.5 % of failures are the number of defects in each category was determined.
table-level and their major reasons were programmers’
mistakes and frequent changes of data schema. There were also
62 % of row-level failures and most of them were caused by

55
Unauthenticated
Download Date | 1/12/18 1:40 AM
 Information Technology and Management Science
_______________________________________________________________________________________________ 2017/20

TABLE I TABLE II
DEFECT CATEGORIES TOTAL DEFECTS BY CATEGORY
No. Category Description Number of
No. Category
1. Development Actual or potential cause of defects
process failure is due to deficiencies in 1. Development process 59
the requirements analysis,
development, testing, 2. Integration with other systems 10
implementation or maintenance 3. Data processing 20
process
Actual or potential cause of 4. Usability 1
2. Integration with
other systems failure is related to 5. Security 0
interoperability
6. Performance 1
3. Data processing Actual or potential cause of
failure is any defect affecting data 7. Serviceability 0
integrity
8. Other 11
4. Usability Actual or potential cause of
failure is related to usability (ease
of use) requirements. According to the results provided in Table II, the missed
5. Security Actual or potential cause of
failure is related to security defects in the production environment are due to deficiencies in
requirements, such as those for the development process. Among all user requests, in 74 cases
authentication, authorisation, of defect causes, it was mentioned that they had not been found
privacy/confidentiality,
accountability (e.g., audit trail or
during testing. Part of the defects could have been discovered
event logging), etc. earlier if the test method based on testing the characteristics had
6. Performance Actual or potential cause of been used.
failure is related to performance In some cases, the nature of the defects makes it clear that the
requirements (e.g., capacity,
computational accuracy, response tester lacks general knowledge about the e-service solution
time, throughput, or availability). architecture and it is not enough to do integrity tests. In four
7. Serviceability Actual or potential cause of cases, defects were reported in the e-service because the tester
failure is related to requirements
for reliability, maintainability, or
had not notified about new software defect removals that would
supportability (e.g., complex require testing. Other five cases showed that defects occurred
design, undocumented code, because functionality had been changed in another part of the
ambiguous or incomplete error system. Several defect requests showed that in many cases the
logging, etc.).
Would not cause any of the tester and the system analyst had not been informed about
8. Other
effects above changes implemented in functionality, thus creating situations
where the described functional requirements were not actual in
To analyse the causes of software defects, the Ishikava documents. All of these cases indicate problems in the change
method was used [13]. Based on the examples of method management process.
approach, four categories of causes were identified [6]: The second highest number of defects is related to data
 methods, which might be incomplete, ambiguous, processing. These defects relate to generation of a PDF
wrong, or unenforced; document that is affected by the degree of complexity of
 tools and environment, which might be clumsy, invoices. These problems cannot be completely avoided due to
unreliable, or defective; the technology used in the solution. Because the number of
 people, who might lack adequate training or functionality defects is significantly higher than in other
understanding; categories, this proportion indicates that the shortcomings of the
 input and requirements, which might be incomplete, development process and the problems with generating PDF
ambiguos, or defective. documents most significantly affect the quality of the e-service.
As a result of the analysis, the most important and most According to the defect rate, the third largest defect category is
common causes were summarised and grouped by cause defects that have various other reasons, such as the temporary
categories, depicting them in the form of Ishikawa diagram. unavailability of a web service or a server. Some of the cases
are affected by the human factor.
IV. RESULTS As a result of the analysis, the main causes of the defects are
In 2015, 17 defect requests were received from e-service summarised in Fig. 1. It shows that the main causes of problems
users and this number increased to 64 in the subsequent year. include categories “methods” and “human”. It demonstrates
Compared with 2015, it is at least three times more. The trend that there is an opportunity to improve software development
of 2017 in the first three months shows that the number of process.
requests does not decrease significantly, but rather increases. The study identified that systematic repetition of defects was
Their number at the moment of the study has already reached due to the fact that the solution used restrictive technologies that
21 defect requests. could affect the quality of the service.
The number of defect requests obtained by classifying all
defects by category is given in Table II.

56
Unauthenticated
Download Date | 1/12/18 1:40 AM
 Information Technology and Management Science
_______________________________________________________________________________________________ 2017/20

REFERENCES
[1] M. Leszak, D. E. Perry, and D. Stoll, “A Case Study in Root Cause Defect
Analysis,” Proceedings of the 22nd international conference on Software
engineering – ICSE ’00, pp. 433–434, Jun. 2000.
https://doi.org/10.1145/337180.337232
[2] A. A. Shenvi, “Defect Prevention with Orthogonal Defect Classification,”
Proceeding of the 2nd annual conference on India software engineering
conference – ISEC ’09, p. 83, 2009. https://doi.org/10.1145/1506216.1506232
[3] V. Gupta, N. Ganeshan, and T. Singhal Kumar, “Determining the Root
Causes of Various Software Bugs Through Software Metrics”, 2015 2nd
International Conference on Computing for Sustainable Global
Development (INDIACom), 11–13 March, New Delhi, p. 1212, 2015.
Fig. 1. Defect causes. [4] “IEEE Standard Classification for Software Anomalies,” Revision of IEEE
Std 1044-2009, p. 8, 2010. https://doi.org/10.1109/ieeestd.2010.5399061
[5] S. Pertet and P. Narasimhan, Causes of Failure in Web Applications.
The results of the analysis show that functionality defects are Parallel Data Laboratory Carnegie Mellon University Pittsburgh, PA
the most common ones in the production environment, a total 15213-3890, CMU-PDL-05-109, pp. 3–8, Dec. 2005 [Online]. Aviable:
http://www.cs.cmu.edu/~priya/PDL-CMU-05-109.pdf
of 89 out of 102. The main causes of defects are related to [6] D. N. Card, “Learning from Our Mistakes with Defect Causal Analysis,”
problems in the development process, the technologies used in IEEE Software, vol. 15, no. 1, pp. 56–63, 1998.
the solution, insufficient testing and lack of knowledge of the https://doi.org/10.1109/52.646883
solution architecture. [7] S. Li, H. Zhou, T. Xiao, H. Lin, W. Lin, and T. Xie, “A characteristic
study on failures of production distributed data-parallel programms,”
Proceeding of the 2013 International Conference on Software
V. CONCLUSION Engineering, p. 963–972, 2013. https://dl.acm.org/citation.cfm?id=
2486921&CFID=1010944701&CFTOKEN=99591560
The results of the study have confirmed the causes of the [8] N. Nuzhat, K. Aihab, and K. Ahmed, “Survey to Improve Software
defects mentioned in the related studies; namely, the most Quality Assurance in Developing Countries,” International Journal of
common causes of defects are related to testing problems [9] Technology and Research, Islamabad 3.1 1–6, pp. 3–5, 2015.
[9] M. Hamill and K. Goseva-Popstojanova, “Common Trends in Software
and awareness of changes in software [8]. Fault and Failure Data,” IEEE Transactions on Software Engineering,
The main conclusions of the research are as follows: vol. 35, no. 4, pp. 484–496, Jul. 2009. https://doi.org/10.1109/tse.2009.3
 the example studied shows that the technology used in [10] N. Hrgarek, “Fabasoft Best Practices and Test Metrics Model,” Journal
of Information and Organizational Sciences, vol 31, no 1, Austria, pp. 75,
the solution may limit the choices of quality
2007 [Online]. Available:
improvements, but does not prove that this is a common http://jios.foi.hr/index.php/jios/article/view/31/29
problem; [11] “ISTQB Glossary” [Online]. Available: http://glossary.istqb.org/search/defect
 one of the most common root causes of defects is related [12] ISTQB, “Why is Testing Necessary,” Certified Tester, Foundation Level
Syllabus, p. 11, 2012 [Online]. Available:
to deficiencies in the development process, which is http://www.istqb.org/downloads/send/2-foundation-level-documents/3-
also confirmed by the example under consideration; foundation-level-syllabus-2011.html4
 the used example shows that it is necessary to create [13] D. Dhandapani, “Applying the Fishbone diageam and Pareto principle to
Domino,” 2004 [Online]. Available:
new defect cause categories, which is useful for a https://www.ibm.com/developerworks/lotus/library/fishbone/
systematic defect identification step in the defect cause [14] ISTQB, “Types of Process Improvement,” Certified Tester, Advanced
analysis process; Level Syllabus Test Manager, p. 60, 2012 [Online]. Available:
http://www.istqb.org/downloads/send/10-advanced-level-syllabus-
 defect cause classification will help more precisly 2012/54-advanced-level-syllabus-2012-test-manager.html
identify if the problems in the development process are
related to design/analysis, coding, testing or Laila Bergmane is a student at the professional Master study programme
infrastructure fields; “Information Technology” at the Institute of Information Technology of Riga
Technical University (Latvia). She holds ISTQB certificate. This is her first
 in the example above, 71 % of all defects were not
publication. She also works as a Software Tester at ZZ Dats Ltd.
found during testing and this was mentioned in the E-mail: [email protected]
related studies as one of the main causes of failures.
The limitation of the present study is related to the fact that Jānis Grabis holds a Doctoral degree and is a Professor at Riga Technical
the causes of the example software defects are identified based University (Latvia) as well as the Head of the Institute of Information
Technology. His main research interests lie within the application of
on the information provided in defect requests, the number of
mathematical programming methods in information technology, enterprise
which may not be sufficient to ensure more objective applications and system integration. He has published more than 60 scientific
determination of their causes. papers, including a monograph on supply chain configuration. He has led a
Future research may be devoted to the implementation of number of national projects and participated in five projects in collaboration
new software defect cause classification at the company in with the University of Michigan-Dearborn (USA) and funded mainly by
industrial partners, such as SAP America and Ford Motor Company.
order to support the defect cause analysis process. E-mail: [email protected]

Edžus Žeiris holds a Doctoral degree and is a Deputy Director at the ZZ Dats
Ltd. He holds CISM and PMP certificates. Research interests include design of
e-services, security and evaluation of electronic services.
E-mail: [email protected]

57
Unauthenticated
Download Date | 1/12/18 1:40 AM