Enhancing Product Reliability Through Maturity

Analysis: The Case Study of PT. XYZ

Thomson Palito Napitupulu Maulahikmah Galinium Mohammad Achmad Amin Soetomo
Master of Information Technology Master of Information Technology Master of Information Technology
Swiss German University Swiss German University Swiss German University
Tangerang, Indonesia Tangerang, Indonesia Tangerang, Indonesia
[email protected] [email protected] [email protected]

Abstract—This research focuses on enhancing product relia- In Southeast Asia, the banking sector faces various risks in
bility through System Maturity Analysis, specifically within the a dynamic and rapidly changing global financial environment.
context of PT. XYZ’s application in the open banking market. While specific information on open banking implementation
Low success transaction rates and recurring defects are the
background for conducting this research. The problem lies in in Southeast Asia is limited, the region’s banks are adapting
the complexity of Integration service as it integrates around 80 to technological advancements. The Association of Southeast
financial institution services. The research aims to measure ma- Asian Nations (ASEAN) banks focus on managing risks,
turity using relevant Software Quality Metrics including Defect particularly credit risk, in the face of financial innovation and
Density, Defect Severity Index, Unit Test Coverage, and Operator economic growth [3].
Intervention Rate and provide software development practices
using Capability Maturity Model Integration for Development In this research, the author studied PT. XYZ, one of
(CMMI-DEV) framework that can fill the gap between current the pioneer companies exploring the Open Banking market
practices and best practices in software quality and reliability. in Southeast Asia. PT. XYZ has enabled services in open
The research employs a mixed approach, using quantitative data banking since 2016, including direct payment, bulk transfer,
to understand the current maturity of the Integration service and customized API payment solutions, and all-in-one payment
qualitative methods to gather insights from stakeholders. The
study maps system properties of maturity in Software Improve- infrastructure. In the third quarter of 2024, the executive
ment Group (SIG) Reliability Model—fault isolation, autonomy, board demanded improved web application reliability from the
monitoring, and testing—to the CMMI-DEV process areas which engineering department, with a commitment to achieve up to
are then incorporated into the Scrum development cycle. As a a 95% addressable success rate.
result, PT. XYZ managed to decrease Defect Density and Defect This demand is emphasized by reports that even though
Severity Index below 2.0, reduce Operator Intervention Rate to
below 7 minutes, and increase Unit Testing Coverage to more large banks increasingly provide open API architecture to
than 50%. facilitate collaboration, unique challenges remain in imple-
Index Terms—System Maturity Analysis, Software Quality menting API standards [4]. For example, in the Philippines,
Metrics, CMMI-DEV, SIG Reliability Model, Scrum standardization relies on a ”nudge” rather than ”mandate” ap-
proach, which can lead to inconsistencies and slower adoption
I. I NTRODUCTION compared to countries with stricter regulations. Since PT. XYZ
has more than 10 million API hits monthly, web application
Financial technologies, commonly referred to as fintech, reliability has become a priority.
have fundamentally transformed the banking and financial The research problem focuses on two main areas:
services sector by presenting groundbreaking solutions that 1) The lack of standardized workflow in quality assurance,
significantly improve operational efficiency, accessibility, and particularly in defect prevention processes and testing.
the overall customer experience. The integration between 2) The need to establish reliability measurement specific to
banks and fintech companies has evolved through various lev- the maturity level of Integration service modularity.
els, from traditional banking to digital banking, open banking,
This research aims to address these problems by:
and the emerging Open-X banking model [1].
In March 2024, the proportion of digitally active banking 1) Analyzing the maturity of Integration service to enhance
customers utilizing open banking services showed a continued the reliability of PT. XYZ’s fintech app product by
upward trend, reaching 14% [2]. This implies that one in measuring with relevant metrics.
seven digital banking consumers currently possesses an active 2) Providing software development practices to increase the
open banking linkage or has executed a transaction through system maturity level of Integration service by aligning
open banking mechanisms. Open banking transactions have with software development best practices.
surpassed data connections in terms of market penetration, The significance of this study is threefold:
with 8.2% of digitally enabled customers conducting an open 1) Helping PT. XYZ measure reliability and prioritize
banking transaction in January 2024, compared to 7.2 improvement areas in Integration service.
 2) Helping PT. XYZ incorporate reliability concerns into 2) Reliability in ISO 25010: ISO 25010 divides Reliability
their current software product development process. characteristics into four sub-characteristics [8]:
3) Providing insights for software product companies and • Maturity: Assesses the frequency of failures in the soft-
academic communities to improve their software de- ware.
velopment processes regarding reliability and capability • Availability: Measures the ratio of time during which the
maturity level. software remains functional and accessible.
II. L ITERATURE R EVIEW • Fault Tolerance: Pertains to the software’s capacity to sus-

A. Product Reliability in Software Engineering tain proper functionality despite occurrences of hardware
or software malfunctions.
In software engineering, ”product” refers to a software • Recoverability: Evaluates how quickly a system can re-
application or system developed to meet specific user require- cover from failures or crashes.
ments and deliver certain functionalities. Software products
may vary from simple applications to complex systems serving 3) SIG Reliability Model: The Software Improvement
extensive organizations. Product reliability is conceptualized Group (SIG) Reliability Model, grounded in ISO 25010, sys-
as the extent to which a software product reliably executes its tematically evaluates eight system attributes to determine ma-
designated functions without experiencing failure [5]. turity, fault tolerance, and recoverability [9]. For this research,
The core concept of product reliability revolves around four system properties related to maturity were selected:
the systematic assessment and enhancement of a software • Fault Isolation: Examines how well faults in a component
product’s performance over time. This involves not only the are contained.
initial development phase but also ongoing evaluation and • Autonomy: Assesses the system’s capacity to deliver
improvement throughout the software lifecycle. Key metrics anticipated services without human intervention.
for assessing reliability include Mean Time Between Failures • Monitoring: Gauges the extent to which operations,
(MTBF), Mean Time to Failure (MTTF), defect density, and events, and resources are scrutinized.
failure rates [6]. • Testing: Reflects the system’s capability to manage an-
ticipated peak loads.
B. Software Reliability Metrics
4) CMMI-DEV Framework: The Capability Maturity
Several metrics can be used to assess software reliability.
Model Integration for Development (CMMI-DEV) provides
For this research, the following metrics were selected based
a comprehensive framework of best practices for process
on their relevance to the PT. XYZ case:
1) Defect Density: Defect Density is calculated by dividing improvement [10]. For this research, three process areas were
the total number of defects by the size of the code (in selected:
thousands of lines of code): • Configuration Management (CM)
• Project Monitoring and Control (PMC)
TD • Verification (VER)
DD = (1)
KSLOC
where DD is Defect Density, T D is Total Defects, and III. R ESEARCH M ETHODS
KSLOC is the total lines of source code in thousands. A. Research Framework
2) Defect Severity Index: Defect Severity Index is calcu-
lated using: The research framework was designed based on mapping
Pn system properties of maturity to relevant CMMI-DEV process
T Di × Wi areas, as shown in Fig. 1. The flow starts with implementing
DSI = i=1 (2)
TD Configuration Management (CM), which supports Verification
where DSI is Defect Severity Index, T Di is Total Defect (VER) processes. During implementation, Project Monitoring
per severity category, Wi is the severity weight, and T D is and Control (PMC) is continuously performed. Measurements
the total number of defects. of reliability metrics (Defect Density, Defect Severity Index,
3) Unit Test Coverage: Unit Test Coverage measures the Operator Intervention Rate, and Unit Testing Coverage) are re-
percentage of code lines executed during testing. viewed and discussed with relevant stakeholders. The practices
4) Operator Intervention Rate: Operator Intervention Rate are incorporated into the Scrum sprint cycle for continuous
measures how often operators intervene to maintain the sys- improvement.
tem, measured in minutes per intervention.
C. Software Quality Best Practices B. Data Collection
1) Scrum Framework: Scrum is an agile framework for Data collection was conducted through:
managing complex projects, particularly in software develop- • Observation of secondary data (engineering documenta-
ment. It emphasizes iterative progress through short cycles tion, testing results, source codes, and operational situa-
called sprints. Scrum incorporates a series of events: Sprint tion)
Planning, Daily Scrum, Sprint Review, and Sprint Retrospec- • Interviews with Integration team members regarding cur-
tive [7]. rent process conditions
 IV. R ESULTS AND D ISCUSSIONS
A. Data Collection Results
Initial measurements of the Integration service revealed:
1) Defect Density: The highest defect density score was in
the Fetch Statements use case with a score of 6.54. Several
use cases had scores above 2.0, including Callback, Payment
QR, Session Controller, and Session Tasks.
2) Defect Severity Index: Several use cases had scores
above 2.0, including Callback, Session Controller, Transfer
Funds Controller, and State Machine Manager.
3) Operator Intervention Rate: All occurrences exceeded 7
minutes, with durations ranging from 8 to 15 minutes.
4) Unit Test Coverage: Most use cases had coverage below
50%.
5) Process Assessment: Assessment of selected CMMI-
DEV process areas showed:
Fig. 1. Research Framework • 10 specific practices were fully performed
• 11 specific practices were partially performed
• 3 specific practices were not performed
For the maturity measurement, the following techniques
were used: B. Gap Analysis
• Defect Severity Index (for Fault Isolation)
Based on the interview with the Engineering Manager, the
• Operator Intervention Rate (for Autonomy)
success criteria for improvement were established:
• Defect Density (for Monitoring)
• Defect Severity Index: 2% of total major defects
• Unit Test Coverage (for Testing)
• Operator Intervention Rate: Below 10 minutes
C. Gap Analysis • Defect Density: <2.0
Gap analysis was performed by comparing current and • Unit Test Coverage: >50%
desired states of measurement results about the maturity of
C. Designed Practices
Integration service. The gap in current processes was identified
from unperformed processes resulting from the interviews. For each process area, specific practices were designed:
1) Configuration Management (CM):
D. Designing Practices
• Identify and select important configurations that follow
Practices were designed by adopting specific goals and certain criteria of importance
subpractices from the selected CMMI-DEV process areas, • Specify the ownership of work products
tailored to the maturity of Integration service. • Document key-value information

E. Incorporating Practices into Scrum Sprint Cycle 2) Project Monitoring and Control (PMC):
The designed practices were incorporated into the Scrum • Document identified risks and progress in managing the
events: risk
• Sprint Planning • Communicate significant changes in risk status to internal

• Daily Scrum stakeholders

• Sprint Review • Gather stakeholder representation for issue analysis

• Sprint Retrospective • Document corrective action plans

• Monitor implementation of corrective actions
F. Validation • Document action progress and results
Validation was conducted through: 3) Verification (VER):
• Re-measurement of maturity metrics after implementing • Submit verification plans including methods and tools
the practices • Explore verification tools
• Interviews with external subject matter experts • Establish verification criteria and parameters
The success criteria for validation were: • Establish satisfaction criteria for peer review
• Defect Severity Index: <2.0 of DSI score and <2% of • Perform peer review based on the given checklist
remaining major defects • Communicate issues to relevant stakeholders
• Operator Intervention Rate: <7 minutes • Conduct and improve unit tests and acceptance criteria
• Defect Density: <2.0 • Analyze defects and provide information on reproduction
• Unit Test Coverage: >50% and resolution
D. Incorporating Practices into Scrum Sprint Cycle G. Expert Validation
The practices were incorporated into Scrum events as fol- Four external experts with backgrounds in e-commerce,
lows: banking, stock exchange, and government technology vali-
dated the approach. Key insights included:
TABLE I
I NCORPORATED P RACTICES INTO S CRUM S PRINT C YCLE • The need to include non-functional requirements such as
response time
Scrum Event Incorporated Practices
• The recommendation to involve QA staff in requirement
1. Identify important configurations
2. Specify ownership of configuration items gathering
Sprint Planning 3. Submit verification plans • The suggestion to automate data collection processes
4. Document identified risks
• The importance of establishing clear RACI matrices for
5. Establish verification criteria
1. Explore verification tools responsibilities
2. Perform peer reviews
3. Communicate issues to stakeholders V. C ONCLUSIONS AND R ECOMMENDATIONS
Daily Scrum
4. Conduct and improve unit tests
5. Analyze defects for reproduction/resolution A. Conclusions
6. Monitor corrective action implementation
1. Document key-value information This study successfully analyzed the maturity of PT. XYZ’s
Sprint Review
2. Communicate risk status changes Integration service using relevant software reliability metrics:
Sprint 1. Document corrective action results
Retrospective
Defect Density, Defect Severity Index, Unit Test Coverage,
and Operator Intervention Rate. These measurements provided
visibility into critical areas of the Integration service and
E. Framework Implementation helped prioritize improvements.
The research also provided tailored software development
Implementation of the research framework included:
practices derived from CMMI-DEV process areas (Configura-
• Identification of configuration items by Product Manager
tion Management, Project Monitoring and Control, and Ver-
and Senior Developer ification) and incorporated them into the Scrum sprint cycle.
• Establishment of change tracking for configuration items
After three sprint cycles, three out of four success criteria were
• Setup of Terraform integration with Jira for configuration
fully achieved, with Defect Severity Index partially achieved.
integrity
• Creation of discussion sessions for requirements and B. Recommendations
acceptance criteria
Based on the results, the following recommendations are
• Peer review sessions with checklists
made:
• Regular progress monitoring and communication
• Analysis of alternatives for authentication methods • Include feature flags and A/B testing configurations in

• Documentation of issues and resolutions configuration management

• Develop real-time visualization of quality metrics using
F. Validation Results Grafana
After three sprint cycles of adopting the tailored practices, • Improve spike ticket documentation with detailed infor-
the results showed: mation about approaches
• Implement monitoring for transaction performance during
TABLE II peak hours
VALIDATION R ESULTS
• Explore the TMMI framework to enhance testing prac-
Metric Success Crite- Result Status tices
ria
• Define clear RACI matrices for each responsibility
Defect Severity <2.0 All <2.0 with 1 Partially
Index major defect Achieved • Conduct performance testing alongside functional testing
Operator <7 minutes 4-5 minutes Achieved
Intervention C. Future Work
Rate
Defect Density <2.0 All <2.0 Achieved Future work could include:
Unit Test Cover- >50% All >50% Achieved
• Development of real-time measurement dashboards
age
• Exploration of fault tolerance measurement and improve-
ment
Improvements were achieved through: • Investigation of recoverability measurement and enhance-
• Distribution of senior engineers to critical use cases ment
• Intensive communication in daily scrum events • Analysis of methods for defining bug criticality scientif-
• Integration with third-party authentication platform ically
• Implementation of Test-Driven Development • Continuous validation of the framework to identify miss-
• Optimization of deployment pipelines ing or irrelevant practices
 R EFERENCES
[1] P. M. Rubanov, ”Transformation of the banking sector in the digital era,”
, vol. 4, 2019.
[2] ”The Open Banking Impact Report,” Openbanking.org.uk, Mar. 2024.
[3] A. B. Khan et al., ”Financial innovation, sustainable economic growth,
and credit risk: A case of the ASEAN banking sector,” Frontiers in
Environmental Science, vol. 9, p. 729922, 2021.
[4] ”Is Southeast Asia ready to embrace open banking? An Executive
Insights Report on Open Banking Readiness in the Philippines, Indonesia
and Thailand,” Brankas, 2024.
[5] S. K. Park, ”A comparative study on the property of nhpp software
reliability model based on shape parameters of exponential family
lifetime distribution,” International Journal of Engineering Research and
Technology, vol. 13, no. 5, p. 873, 2020.
[6] Y. Wang, J. J. Yang, and N. M. Mbiye, ”Effectiveness and suitability
of the automotive ehps software reliability and testing,” Advances in
Science, Technology and Engineering Systems Journal, vol. 6, no. 3,
pp. 205–212, 2021.
[7] K. Schwaber and J. Sutherland, ”The scrum guide,” Scrum Alliance,
vol. 21, no. 1, pp. 1–38, 2020.
[8] ”ISO/IEC 25010:2011 Systems and software engineering – Systems and
software quality requirements and evaluation (SQuaRE) – System and
software quality models,” International Organization for Standardization,
Geneva, Switzerland, 2011.
[9] Software Improvement Group, ”SIG Evaluation Criteria for Reliability,”
2020.
[10] CMMI Product Team, ”Capability Maturity Model Integration (CMMI-
DEV),” Software Engineering Institute, 2009.