Software Testing Framework For ERP Systems Based On Agile Development
Software Testing Framework For ERP Systems Based On Agile Development
Keywords: Agile, Critical Success Factors (CSF), ERP technical upgrades in terms of hardware and network to
implementation, software validation, complement the implementation, and lack of data integration.
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on February 27,2023 at 04:11:29 UTC from IEEE Xplore. Restrictions apply.
until cloud ERP clients become more sophisticated in 3 The Proposed Framework
managing its current limitations.
Many industries depend on Information Technology and ERP Figure 1 shows the four primary processes of improving and
software to manage their operations. Gill, Shahzad, and testing ERP software collecting input data, ERP testing
Ramalu [5] used an explanatory and predictive orientation framework, software validation, and output data. Input data is
approach to identify critical factors of successful collected for live testing and to develop appropriate test cases.
implementation of ERP and their impact in the banking sector. In the validation phase, the test cases are run through the
The outcome showed that management buy-in, information system to check if it is functioning as designed. Agile
technology infrastructure, and end-user involvement have a methodology is then deployed to ensure that the testing process
positive impact on the successful implementation of ERP. evolves and improves. The output communicates the extent to
Metaxiotis and Liagkouras [6] reviewed literature to examine which the system is functioning as per end-user requirements.
the implementation and operation of ERP systems in the
finance industry. For successful implementation, the authors
recommend customizing ERP systems to meet the demands of
the organization and the integration of existing systems.
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on February 27,2023 at 04:11:29 UTC from IEEE Xplore. Restrictions apply.
unrelated to the system data and (2) Test data correlation - priorities for customer requirement is the MoSCoW Method
describing the relation between system and input data [16]. (developed by Dai Clegg of Oracle UK) (see fig. 3). MoSCoW
is a prioritization procedure consisting of Must Have (M),
Should Have (S), Could Have (C), and Will Not Have (W)
[24]. This method requires the developers to coordinate with
the client to create a core testing team which will formulate a
set of product requirements needed for the ERP system.
Conducting a survey will help to categorize and prioritize the
requirements using MoSCoW.
The agile testing process involves three phases Test Case Figure 3: MoSCoW method hierarchy.
Design (TCD), Test Execution (TEX), and Test Improvement
(TIP) (see tab. 1). The TCD phase quantifies the scope of The developer perceived code implementation complexity
testing required for ERP implementation. After the scope is helps to identify, during testing, which requirements will have
quantified, the required modules can be identified. The a higher number of errors due to the complexity of code
modules are categorized according to their ability to be tested implementation. The complexity of code implementation can
automatically or manually. Stakeholders related to each cause various issues if developers do not focus on it in the early
module would be identified and communication channels stages of testing and implementation. To avoid spending time
established. The testing infrastructure would then be designed, and money on fixing faults, it is advisable to address them in
and the implementation integrated with the core testing team. the early stages of testing, by prioritizing complex codes. The
technicality of the second factor requires developers to decide
Table 1: Core phases of ERP testing. among themselves. Ranking codes according to complexity
can help in prioritizing and allocating resources towards the
most complex codes.
TCD TEX TIP
x Quantify scope of x Design test cases for all x Quality assurance to
testing required business processes monitor quality of Changes in requirements indicate the number of times a
x Identify modules x Ensure that the testing testing and recommend
improvements
requirement has been altered in the development cycle with
required for ERP tool is configured
implementation. properly x Develop a knowledge respect to its origin date. If a requirement is altered more than
x Classify modules that x Execute validation of base of manual and
10 times, the volatility values for all the requirements are
require manual and manual test cases automated testing
automated testing x Run automated test tools, which can help quantified on a 10-point scale [20]. The requirement changes
x Identify stakeholders to
establish communication
cases using appropriate
tool
identify the right tool
quickly
for a requirement i (RCi) can be computed by dividing the
channels x Generate report for all x Develop scripts for number of changes for requirement i to the highest number of
x Design testing reusing automated
infrastructure based on
test cases for verification
testing routines changes for any requirement. If ith requirement is changed N
project size/requirements times, and the maximum number of requirement changes
x Integrate Core and
Implementation testing amongst all the requirements is M, then the requirement change
teams of i can be computed using the following equation:
To ensure proper supply of test case data, the framework (fig. ܴܥ =
ே
× 10 (1)
2) maps out new test case factors customer assigned priority of ெ
requirements, developer perceived code implementation, and
changes in requirements. The customer assigned priority of The change factor is calculated because the changes might
requirements as a factor in the system test data supply, provides cause errors. Sometimes developers are unable to reduce the
information related to requirements demanded by the customer number of times a project requirement changes but noting the
and their priorities. To acquire data for this first factor, the number of changes can give foresight as to the number of faults
developer and the customer, must have a clear understanding they could expect so that they can develop a counter measure.
of business processes and keep track of the changes that the
company makes (CSF 3). The developer must consider the The TEX phase starts with designing test cases for all business
specific needs of the business when customizing the ERP processes. Combining manual testing with automatic helps to
solution (CSF 4). These two critical factors are necessary in improve the efficiency of the process. To optimize the ERP
creating a list of requirements. The method used to assign system testing process, one must focus on facilitating relevant
test data and ensure that the data is stable and meets the input
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on February 27,2023 at 04:11:29 UTC from IEEE Xplore. Restrictions apply.
constraints and correlation requirements. Once a proper Since ERP implementation involves customization and
configuration is assured, the validation of test cases could be dynamic changes, it must also include performance and user
executed starting from manual to automated tests. Afterwards, acceptance testing.
a report can be generated.
4 Case Study
Table 2: Five Critical Success Factors for ERP The case study conducted in [21] focused on Omantel. Omantel
implementation. [22] is a government-owned telecommunications service
CSF 1 CSF 2 CSF 3 CSF 4 CSF 5 provider in Oman. It is Oman’s first and leading integrated
telecommunications services provider. In 2005, Omantel
Ensure Compatibili Understand Customiza Engage
complianc ty with ing tion of ERP ment
decided to initiate an ERP project to prepare for the anticipated
e with legacy business solution to with liberalization of the economy and meet the development
project systems to process suit end- requirements of the nation [21]. The company hired Oracle to
scope and avoid and specific users to provide ERP solutions. Initially, the project was to be
goals errors and keeping business ensure completed within a year, but due to the lack of ERP expertise
incompatibi track of needs satisfacti
in Omantel’s team, the project faced delays and Oracle had to
lity changes on
take on a consultancy role to deliver training and transfer
knowledge. The project also included the validation of quality
of service, defining service architecture, and developing testing
protocols.
3.3 ERP Software Validation
Due to Omantel’s various legacy systems, Oracle faced many
A test case comprises the sequence of steps involved in compatibility issues that required significant customization
validating a system’s intended function. The processes and system reengineering. The project plan was monitored and
involved in ERP software validation stage are Setup, Load, controlled through a formal change control process that sought
Validation, and Reset (see fig. 3). The process begins with to track and manage requests for change, sequencing, timing of
setting up the ERP system. The load phase sets in motion the the plan, and modifying the plan [21]. Regarding the system’s
relevant sequence of inputs to test the system, while validation testing and software customization, a joint system testing and
checks and analyses whether the obtained results are accurate. software customization process was conducted by Omantel and
The reset phase returns the system to its original state to Oracle, whereby specialists were called to assess Omantel’s
process the next input. The four phases can be improved products and services to determine the potential effects of the
through iteration planning and acceptance testing to ensure ERP system. The project suffered from four main challenges a
high quality test cases are provided to the system [19]. ERP lack of ERP expertise, presence of various legacy systems,
testing involves core testing and implementation testing teams. insufficient test case scenarios, and lack of employee
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on February 27,2023 at 04:11:29 UTC from IEEE Xplore. Restrictions apply.
engagement during system design [23]. The following section system. The requirements would then be assigned a number on
elaborates these challenges in more detail. a scale of one to ten for prioritization.
Lack of ERP Expertise: Oracle had to take on a consultancy The perceived code implementation complexity step will
role to deliver training and knowledge transfer, due to the lack identify which aspects of the implementation are most complex
of expertise on Omantel’s team. Omantel only had a few IT/IS and give more focus to the most complex aspects, to facilitate
employees with Oracle integration expertise. This caused optimal time utilization and avoid errors becoming more
communication problems, as Oracle team members would complex. Should there be any changes in requirements, the
have to explain technical aspects of the process to Omantel’s proposed formula, (RCi = (N/M) *10), can be applied. Noting
staff. The staff also had limited expertise on Oracle products, the number of changes made can help the developers anticipate
thereby causing more delays. the possible number of faults during testing.
Presence of Legacy Systems: Incompatibilities in integrating Our approach proposes dealing with the identified delay issues
the ERP system with the legacy systems compelled Oracle to using a survey-type questionnaire distributed to the staff. The
spend more time investigating the incompatibilities and proposed survey shall include basic information to equip
explaining them to Omantel staff, to help them understand Omantel staff with the knowledge they need regarding the ERP
which features of the various legacy systems could be process and integration and a walkthrough of the testing and
integrated into the ERP system. Furthermore, breaks-in-service validation process. It would also include the MoSCoW method,
impacted end-consumer experience. to teach Omantel staff how the data gathered would be treated.
A FAQ and contact numbers of the core testing team members
Insufficient Test Cases: The test case scenarios developed by shall be provided, in case of further inquiry. FAQs will
Omantel were insufficient, as they lacked the necessary depth eliminate redundancies and save time. A questionnaire and
and robustness. The lack of robust test cases resulted in answer sheet will be attached to the survey.
inefficient testing, delays and last-minute changes to the
process. This occurred because either the requirements for Survey is preferred over interviews because it is less time-
change had been observed before testing began but were not consuming and can gather information from many people in a
sufficiently addressed, or changes had to be made following a shorter time. Also, assessing requirements and prioritizing
problem identified during testing. them through the MoSCoW method, can speed-up the testing
process [25]. It would also make the overall process more
Lack of Employees Engagement During System Design: At robust by providing insight into test cases and scripts created
the time of commissioning, approximately 1 in 3 employees for testing the ERP.
was not trained on the system's use. About 15% of the project
staff did not receive any training at all. Furthermore, the trained Phase 2: Test Execution
staff were required to use their newly acquired skills about Applying the TEX phase to Omantel’s case will help the
seven months after training. Due to lack of practice, refresher organization to improve the process of designing test cases, as
courses were required resulting in increased company Omantel’s test cases lacked the necessary calibre and depth to
expenditure. ensure the robustness necessary for successful ERP system
implementation. Omantel’s failure to adopt a rigorous testing
Application of Proposed Framework to Omantel process compromised overall compatibility and reliability of
The proposed framework employs an agile approach to test the system and resulted in last minute changes and delays. The
Omantel’s ERP software using the three phases Test Case TEX phase in the proposed framework would help to eliminate
Design (TCD), Test Execution (TEX), and Test Improvement such delays by ensuring that the provided test case data is stable
(TIP). and meets the input constraints and correlation requirements.
Also, the TEX phase would help Omantel to verify if the
Phase 1: Test Case Design testing tools being used are appropriate and deliver optimal
By applying TCD to Omantel’s case study, we can quantify the performance.
scope of testing and design test cases based on the collected
data. The framework approach will ensure proper test case data Phase 3: Test Improvement
categorization. The information fed to the system would be Applying TIP would help the organization to develop a
grouped based on customer assigned priority of requirements, knowledge base for testing and validation. That is,
developer perceived code complexity, and changes in the documenting the knowledge gained and improvements made
requirements. The framework also recommends creating a core in the implementation process. Providing easy access to
testing team of developers from Oracle and qualified staff from valuable information can be used as a guideline for improving
Omantel to ensure proper understanding of the project scope, technical expertise among Omantel employees. TIP would help
business processes, and technical concepts. Omantel to overcome the issue of poor test case design, as the
In terms of customer assigned priority of requirements, Oracle knowledge would empower employees to develop better test
consultants would engage Omantel’s staff and stakeholders in cases for testing and validation. As TIP recommends
discussions to identify desired requirements for the ERP improvements by aligning with CSFs, it would help Omantel
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on February 27,2023 at 04:11:29 UTC from IEEE Xplore. Restrictions apply.
to learn from its past mistakes and reduce reliance on third- systematic literature review”, Future Computing and
party vendors such as Oracle. Informatics Journal, 1, pp.1-9. (2016).
[5] A. A. Gill, A. Shahzad, S. S. and Ramalu, S.S. “Enterprise
5 Limitations Resource Planning (ERP) implementation issues: An
The first limitation is threats to validity. Theoretical empirical study in commercial banking industry of
frameworks do not operate in a vacuum, as they depend on the Pakistan, Journal of Resource Development and
organizational environment. Therefore, the proposed Management, 18, pp. 86-92, (2016).
framework would work effectively only when cooperation [6] K. Metaxiotis, and K. Liagkouras. “The Current State of
exists between the ERP system provider’s team and the client ERP Systems in Banking Sector”, International Research
company’s staff. Without cooperation, the framework could Journal of Electronics & Computer Engineering, 3, pp. 23-
fail because data collection requires cooperation. 26, (2017).
[7] B. A. Goedhard, “On the determination of the optimal
Another threat to validity is the incompleteness of the methodology for ERP projects”, Doctoral Dissertation,
information collected using the survey forms, while defining Faculty of Science, Radboud University, Nijmegen, NL,
customer assigned priority of requirements. The ERP provider (2016).
team could fail to collect all necessary information and the [8] M. Shameem, R. R. Kumar, M. Nadeem, & A. A. Khan.
employees can provide irrelevant or inaccurate information. “Taxonomical classification of barriers for scaling agile
While the proposed framework could help address technical methods in global software development environment
issues such as incompatibility with legacy systems to a certain using fuzzy analytic hierarchy process”, Applied Soft
extent, it cannot completely resolve certain fundamental Computing, 90, (2020).
incompatibilities. [9] P.Efe, and O. Demirors. “A change management model and
its application in software development
projects”, Computer Standards & Interfaces, 66, pp. 1-12,
6 Conclusions
(2019).
The set of guidelines in this research's proposed framework
[10] K. Könnölä, S. Suomi, T. Mäkilä, T. Jokela, V. Rantala,
integrates agile principles to simplify and improve robustness
and T. Lehtonen. “Agile methods in embedded system
of testing and verification processes during ERP
development: Multiple-case study of three industrial
implementation. It addresses issues such as testers not having
cases, Journal of Systems and Software, 118, pp.134-150,
adequate knowledge about the subject matter, inability to
(2016).
identify the right test cases for testing and validation, and
[11] C. Tam, E. J. da Costa Moura, T. Oliveira, and J. Varajão.
inability to find the right tools for testing automation. The
“The factors influencing the success of on-going agile
proposed framework centres around three phases of the ERP
software development projects”, International Journal of
system namely TCD, TEX and TIP. The proposed framework
Project Management, 38, pp.165-176, (2020).
was applied to a case study on Omantel, to demonstrate how
[12] H. Kagdi, M. Hammad, and J. I. Maletic. “Who can help
challenges faced by the company could be addressed. The
me with this source code change?”, IEEE International
framework has the potential to help organizations to efficiently
Conference on Software Maintenance, Beijing, China,
implement ERP systems and avoid project delays. Future
2008, pp.157-166.
research should apply the proposed framework to other case
[13] J. Anvik, and G. Murphy. “Determining implementation
studies, to further evaluate its validity and make improvements.
expertise from bug reports”, Proceedings of Fourth
International Workshop on Mining Software Repositories,
Minneapolis, 2007, pp. 20-26.
References [14] J. Anvik, L. Hiew, and G. C. Murphy, “Who should fix
this bug?”, Proceedings of 28th international conference
[1] R. C. Thompson, O. O. Olugbara, and A. Singh. “Deriving on Software engineering Shanghai, China, 2006, pp.361-
critical success factors for implementation of enterprise 370.
resource planning systems in higher education [15] M. Fatma, Z. Rahman, and I. Khan. “Multi-item
institution”, African Journal of Information Systems, 10, stakeholder-based scale to measure CSR in the banking
pp. 21-44, (2018). industry”, International Strategic Management Review, 2,
[2] F. M. Elbahri, O. I. Al-Sanjary, M. A. M. Ali, and O. A. pp.9-20, (2014).
Ibrahim. “Technical comparison of cloud Enterprise [16] S. Wieczorek, A. Stefanescu, and I. Schieferdecker. “Test
Resource Planning using web systems”, 2019 IEEE 7th data provision for ERP systems”, In 2008 1st International
Conference on Systems, Process and Control (ICSPC), Conference on Software Testing, Verification, and
Melaka, Malaysia, 2019, pp. 90-95. Validation, Lillehammer, Norway, 2008, pp. 396-403.
[3] M. Tavana, V. Hajipour, and S. Oveisi. “IoT-based [17] A. Al-Hossan, and A. S. Al-Mudimigh. “Practical
Enterprise Resource Planning: Challenges, open issues, guidelines for successful ERP testing”, Journal of
applications, architecture, and future research directions”, Theoretical and Applied Information Technology, 27, pp.
Internet of Things, 11, pp. 1-27, (2020). 11-18, (2011).
[4] M. A. Abd Elmonem, E. S. Nasr, and M. H. Geith. [18] N. Osman, and A. K. Sahraoui. “A Software Requirement
“Benefits and challenges of cloud ERP systems–a Engineering Framework to Enhance Critical Success
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on February 27,2023 at 04:11:29 UTC from IEEE Xplore. Restrictions apply.
Factors for ERP Implementation, International Journal of Available:
Computer Applications, Foundation of Computer Science, http://mrmewr.gov.om/Financials%20for%20website/200
180, pp.32-37, (2018). 6/Year%20End%20English/Management%20Discussion
[19] Y. Chernak. “Validating and Improving Test-Case %20And%20Analysis_2006_Final.pdf
effectiveness”, IEEE Software, 18, pp. 82-86, (2001). [24] Agile Business Consortium, MoSCoW Prioritisation,
[20] R. Krishnamoorthi, and S. A. Mary. “Factor oriented [Online]. Available:
requirement coverage-based system test case prioritization https://www.agilebusiness.org/page/ProjectFramework_1
of new and regression test cases”, Information and 0_MoSCoWPrioritisation
Software Technology, 51(4), pp.799-808, (2009). [25] K. S. Ahmad, N. Ahmad, H. Tahir, and S. Khan.
[21] S. Maguire, U. Ojiako, A. Said. “ERP implementation in “Fuzzy_MoSCoW: A fuzzy based MoSCoW method for
Omantel: a case study”, Industrial Management and Data the prioritization of software requirements”, International
Systems, 110, pp.78-92, (2010). Conference on Intelligent Computing, Instrumentation
[22] Omantel. “About us”, Omantel.com. [Online]. Available: and Control Technologies, Kerala, India, 2017, pp.433-
https://www.omantel.om/About%20us2020 437.
[23] M. Al Wohaibi. “Omantel – Management discussion &
analysis report”, 2006, Oman Government. [Online].
Authorized licensed use limited to: UNIVERSITAS GADJAH MADA. Downloaded on February 27,2023 at 04:11:29 UTC from IEEE Xplore. Restrictions apply.