Article

Quality Control Methods Using Quality Characteristics in

Development and Operations †
Daiju Kato 1, * and Hiroshi Ishikawa 2

1 Nihon Knowledge Co., Ltd., Tokyo 111-0042, Japan

2 Graduate School of System Design, Tokyo Metropolitan University, Tokyo 191-0065, Japan;
[email protected]
* Correspondence: [email protected]; Tel.: +81-80-4052-2372
† This paper is the conference extension of Proceedings of the 14th International Conference on Management of
Digital EcoSystems, Venice, Italy, 19–21 October 2022.

Abstract: Since the Software Quality Model was defined as an international standard, many quality
assurance teams have used this quality model in a waterfall model for software development and
quality control. As more software is delivered as a cloud service, various methodologies have been
created with an awareness of the link between development productivity and operations, enabling
faster development. However, most development methods are development-oriented with awareness
of development progress, and there has been little consideration of methods that achieve quality
orientation for continuous quality improvement and monitoring. Therefore, we developed a method
to visualize the progress of software quality during development by defining quality goals in the
project charter using the quality model defined in international standards, classifying each test
by quality characteristics, and clarifying the quality ensured by each test. This was achieved by
classifying each test by quality characteristics and clarifying the quality ensured by each test. To use
quality characteristics as KPIs, it is necessary to manage test results for each test type and compare
them with past build results. This paper explains how to visualize the quality to be assured and
the benefits of using quality characteristics as KPIs and proposes a method to achieve rapid and
high-quality product development.
Citation: Kato, D.; Ishikawa, H.
Quality Control Methods Using Keywords: DevOps; quality control; quality characteristics; SQuaRE; quality analysis
Quality Characteristics in
Development and Operations. Digital
2024, 4, 232–243. https://doi.org/
10.3390/digital4010012 1. Introduction
Academic Editors: Mirjana Ivanović, In many software development environments, work is often divided between devel-
Richard Chbeir and Yannis opment (Dev) teams, which are responsible for developing features, and operations (Ops)
Manolopoulos teams, which are responsible for running the service. The Dev team’s primary mission is to
add new features to the software, while the Ops team’s mission is to maintain the current
Received: 15 October 2023
environment and keep the service stable and continuous. The Dev and Ops teams have
Revised: 25 January 2024
Accepted: 2 February 2024
very different missions, and this can lead to team disagreements. However, faster software
Published: 1 March 2024
development is required to keep up with the rapidly changing business environment. To
this end, teams should avoid becoming exhausted due to internal conflicts, and DevOps [1]
is attracting attention as a way of thinking that “resolves common conflicts between teams
and promotes smooth development through collaboration”.
Copyright: © 2024 by the authors. DevOps has a seven-step lifecycle: Plan, Code, Build, Test, Deploy, Operate, and
Licensee MDPI, Basel, Switzerland. Monitor.
This article is an open access article
distributed under the terms and
• Plan: Defines the task management and development requirements for the entire project;
conditions of the Creative Commons
• Code: Programmers create code according to development requirements;
Attribution (CC BY) license (https:// • Build: The application that will actually run is built from the source code;
creativecommons.org/licenses/by/ • Test: Test the built application for bugs and other defects;
4.0/). • Deploy: Deploy the application into production;

Digital 2024, 4, 232–243. https://doi.org/10.3390/digital4010012 https://www.mdpi.com/journal/digital

Digital 2024, 4 233

• Operation: Perform maintenance and management tasks to ensure continuous service;

• Monitor: Review information obtained from operations, user assessments, requests, etc.
These steps are performed sequentially and continuously to practice DevOps. The
main benefits of implementing DevOps are achieving smooth development, increasing
productivity, and speeding up releases.
The essence of DevOps, as mentioned above, is to resolve conflicts between Dev and
Ops teams. By eliminating the internal drag-and-tug that has been common in the past, the
goal is to achieve smooth development and operations.
To implement DevOps, it is necessary to implement a variety of supporting tools
at each step of the lifecycle. Typical tools include version control systems that track file
changes under project management and CI/CD tools such as Jenkins [2] that automate tasks
previously performed manually. Effective use of these tools leads to increased productivity.
This increased productivity means that more human resources can be allocated to improving
quality and developing new services, which in turn increases the value of the service.
Automating testing and delivery with tools brings benefits in the form of increased
productivity and faster work. Acceleration is simply the speed at which the DevOps
lifecycle can be executed. In other words, you can increase the number of DevOps lifecycle
cycles in a given period of time. More lifecycle cycles mean faster adoption of market
requirements and feedback, which in turn means greater responsiveness to changing
market needs.
For example, a developer runs unit tests against locally implemented code, and if
the unit tests pass, the developer periodically commits changes to the code to a central
repository as part of the CI process. The process is then delegated to Jenkins, the CI enabler,
to run build, unit test, and build verification tests (BVT). If the BVT passes, the build is
automatically deployed. Automated tests such as functional, performance, security, and
regression tests are then run to ensure that the added code does not degrade the quality of
existing functionality.
In addition to testing for quality, quality control is performed using various metrics
and cascading models such as the V-model. Typically, the pass rate, defect rate, and
test coverage of each test type (Table 1) are used to monitor project quality. However,
these quality indicators are often difficult to use to assess the quality of software under
development. Even if the coding rules and test design and implementation rules are clearly
defined, it is difficult to objectively judge whether the quality is good or bad unless it is a
derivative development project.

Table 1. Famous quality metrics for quality control.

Metrics Types Metrics

Ratio per page of design documents
Defect density Ratio per line or step of codes
Ratio per test cases
Removal rate per phases
Defect removal rates Removal rate per components
Removal rate per test cases
Pass rates per test sets
Pass rates Pass rates per test cases
Pass rates per function points
Documents review coverage
Coverages Test coverage per line of codes
Test coverage per test cases

When considering software quality, the quality model [3] defined in the ISO 25000
series (SQuaRE) [4] can be used to classify the quality required of the software under
development in terms of the quality characteristics provided by this quality model, shown
 Table 1. Famous quality metrics for quality control.

Metrics Types Metrics

Digital 2024, 4 Ratio per page of design documents 234
Defect density Ratio per line or step of codes
Ratio per test cases
in Figure 1, and to judge from test results whether
Removal each
ratecharacteristic
per phases has been assured.
SQuaRE alsoDefect
defines the metrics
removal rates for each quality attribute, which can be used to determine
Removal rate per components
whether quality has been assured. Removal rate per test cases
Ito et al. [5] proposed a framework that allows the creation of a strategic test plan
Pass rates per test sets
to achieve incremental quality building in agile development projects. The framework
Pass rates Pass rates per test cases
provides a state in which quality can be explained by releasing the product in a ratable
manner and comprehensive quality assurance Pass
by rates per function
incrementally pointsproduct quality.
building
To use quality characteristics effectively, Documents
we recommend review
thatcoverage
project managers classify
Coverages
the quality of software goals by quality characteristics and, ifline
Test coverage per of codesalso define the
possible,
Test coverage
quality to be achieved at each milestone by quality per test cases
characteristics.

Figure 1. ISO/IEC 25010 quality model for system/software product quality.

Figure 1. ISO/IEC 25010 quality model for system/software product quality.
Therefore, we have studied how to effectively use quality characteristics and existing
qualityTherefore, we have studied
data in development howusing
projects to effectively
the V-model.use quality
We have characteristics and existing
studied the international
quality data in development projects using the V-model.
standard, which is also used in countries such as Germany and South Korea. We have studied theIninterna-
the JIS
tional standard, which is also used in countries such as Germany and
(Japanese Industry Standard) standard certification for software [6], we have studied South Korea. In the
the
JIS (Japanese Industry Standard) standard certification for software
development process with an awareness of compliance with this standard and have applied[6], we have studied
thetodevelopment
it process
several projects. Withwith
the an awareness
expansion of compliance
of the with thisseveral
recurring business, standard and have
projects are
applied itthe
adapting to several projects.soWith
agile process, the expansion
we decided of the
to further recurring
expand the business,
development several projects
process for
are use
the adapting the agile
of quality process,
features so we decided
and improve to it
it so that further expand
can also theindevelopment
be used projects that process
use the
for the
agile use of quality features and improve it so that it can also be used in projects that use
process.
the agile process.
2. Considerations for Using Quality Model in Agile Development Processes
2. Considerations
The combination for Using Quality Model
of continuous in Agile
integration (CI)Development
and continuous Processes
deployment (CD)
processes as a software engineering
The combination of continuous practice for rapidly
integration (CI) developing
and continuousand deploying
deployment software
(CD)
applications into production is called the CI/CD pipeline. The pipeline
processes as a software engineering practice for rapidly developing and deploying soft- is a collection of
tools developers, test engineers, and IT operations staff use throughout
ware applications into production is called the CI/CD pipeline. The pipeline is a collection the continuous
software development,
of tools developers, testdelivery,
engineers, and deployment
and lifecycle.
IT operations staff use throughout the continuous
The development,
software test pyramid, as shownand
delivery, in Figure 2, is a useful
deployment technique that allows us to con-
lifecycle.
ceptualize how
The test to prioritize
pyramid, the tests
as shown in a CI/CD
in Figure 2, is apipeline in terms ofthat
useful technique theirallows
relative
usnumber
to con-
and order of execution. This technique was defined by Mike Cohn [7], with
ceptualize how to prioritize the tests in a CI/CD pipeline in terms of their relative number unit tests at
the bottom, service tests in the middle, and UI tests at the top. By prioritizing
and order of execution. This technique was defined by Mike Cohn [7], with unit tests at with a test
pyramid,
the bottom,youservice
can build
testsa in
strong foundation
the middle, andofUIautomated
tests at theunit
top.tests that are quick
By prioritizing andaeasy
with test
to
pyramid, you can build a strong foundation of automated unit tests that are quick take
execute, then move on to more complex tests that are more complex to write and and
longer
easy totoexecute,
execute,then
andmove
finally,on the
toleast
morecomplex
complex tests that
tests areare
that fewer
more in complex
number. The pipeline
to write and
provides more prioritized feedback.
take longer to execute, and finally, the least complex tests that are fewer in number. The
As in
pipeline the development
provides processfeedback.
more prioritized established for the V-model, quality requirements in
agile projects are classified and normally described in the project plan. An international
standard for quality requirements [8] has already been established, and functional and
non-functional requirements can be classified using quality characteristics according to
the methodology described in this standard. This method makes it possible to clarify the
quality requirements for each actor. The initial quality control is carried out through the
various activities of the development process and the rules defined in the project charter.
Digital 2024, 4 235

The project charter is a short, formal document that summarizes the entire project and
describes the project objectives, how it will be executed, and who will be involved. It is
used throughout the project lifecycle and is an important factor in project planning. These
Digital 2024, 4, FOR PEER REVIEW 4
projects can use quality characteristics for key quality management indicators, as shown in
Table 2.

Figure 2.
Figure 2. Test
Test pyramid.
pyramid.

By
As including all the practices
in the development process related to testing
established in the
for the Agile process
V-model, qualitypipeline
requirementsand thein
review activities necessary to build quality, such as design reviews,
agile projects are classified and normally described in the project plan. An international it is possible to organize
which
standardquality-enhancing activities are
for quality requirements [8]included
has already in each
beenpractice.
established, and functional and
Build, staticrequirements
non-functional analysis, unit cantesting, front-end
be classified usingandquality
back-end integration according
characteristics testing, and to
the
the E2E test pipeline
methodology can be used
described in this to standard.
identify commonThis methodimplementation
makes it possibleerrors and increase
to clarify the
software maturity. It isfor
quality requirements also possible
each actor. to Themeasure individual
initial quality execution
control is carriedtimes
outwithin
through the the
CI
process to immediately detect degradation due to code additions.
various activities of the development process and the rules defined in the project charter. For example, if the execu-
tion
The time of acharter
project test increases compared
is a short, formaltodocument
the previous test,
that it is likely that
summarizes the some
entireperformance
project and
degradation is occurring, and the impact of the added code should
describes the project objectives, how it will be executed, and who will be involved. It is be suspected.
usedDeploying
throughout builds in a continuous
the project lifecycle and delivery (CD) process
is an important allows
factor portability
in project properties
planning. These
to be checked
projects can useduring
quality thecharacteristics
installation process. for key quality management indicators, as shown
Implementing
in Table 2. a general CI/CD process and managing the results will help ensure that
quality is continuously
By including all the improved.
practicesInrelated
addition, by automating
to testing in the Agile and process
integrating moreand
pipeline testing
the
activities, more quality characteristics can be covered in addition to
review activities necessary to build quality, such as design reviews, it is possible to organ- functional conformance.
By
izedefining criteria for testingactivities
which quality-enhancing activities,are each quality in
included attribute can be used as a quality KPI
each practice.
on anBuild,
ongoing basis.
static analysis, unit testing, front-end and back-end integration testing, and the
In many
E2E test pipeline development
can be used projects, GitHub
to identify [9] andimplementation
common Atlassian tools [10] canand
errors be increase
used as
project management tools, task ticket activities can be visualized
software maturity. It is also possible to measure individual execution times within the as kanban boards, andCI
development and bug-fixing activities can be effective for source code control. Project
process to immediately detect degradation due to code additions. For example, if the ex-
management tools are effective for managing project progress, but they cannot manage
ecution time of a test increases compared to the previous test, it is likely that some perfor-
quality progress. Test management tools, on the other hand, can manage test cases and
mance degradation is occurring, and the impact of the added code should be suspected.
test progress and can play a role in quality management, but it is often difficult to use test
Deploying builds in a continuous delivery (CD) process allows portability properties
results in history management.
to be checked during the installation process.
Shimizu et.al [11] proposed a test result management tool to analyze and extend the
Implementing a general CI/CD process and managing the results will help ensure
coverage of automated tests and our team enhanced the test result management tool. The
that quality is continuously improved. In addition, by automating and integrating more
tool is written in C# and stores test results in SQL Server; for unit tests with API calls, we
testing activities, more quality characteristics can be covered in addition to functional con-
decided to implement and integrate a report class that can capture CPU load, memory
formance. By defining criteria for testing activities, each quality attribute can be used as a
consumption, I/O load, and network load along with functional tests and output this
quality KPI on an ongoing basis.
quality data along with the test results. The purpose of the report class is to provide a
In manyefficiency
performance development projects,
assessment. TheGitHub
purpose[9] of and Atlassian
the report classtools [10] canthe
is to extend becoverage
used as
project management tools, task ticket activities
of unit testing by providing a performance efficiency evaluation. can be visualized as kanban boards, and
development and bug-fixing activities can be effective for source
The report class we built called MSTest [12], an extension of NUnit [13], a unit testing code control. Project
managementwith
framework, toolsanare effectivemethod
extension for managingto obtainproject progress,
quality but they cannot
data necessary manage
for evaluating
quality progress. Test management tools, on the other hand,
performance efficiency and to generate test results in XML format along with functional can manage test cases and
test progress and
test results (Figure 3).can play a role in quality management, but it is often difficult to use test
results in history management.
Shimizu et.al [11] proposed a test result management tool to analyze and extend the
coverage of automated tests and our team enhanced the test result management tool. The
tool is written in C# and stores test results in SQL Server; for unit tests with API calls, we
decided to implement and integrate a report class that can capture CPU load, memory
consumption, I/O load, and network load along with functional tests and output this qual-
Digital 2024, 4 236

Table 2. Famous quality metrics for quality control.

Activities Quality Characteristics Quality Sub-Characteristics

Activities in the Whole Development Process
Security Confidentiality
Integrity
Coding Rules Maintainability Modularity
Reusability
Modifiability
Testability
Function completeness
Functional Suitability
Function correctness
Maturity
Design Review Availability
Reliability
Fault tolerance
Recoverability
Maintainability Testability
Inspection Review Reliability Maturity
Activities in CI process
Reliability Maturity
Static Analysis Security Integrity
Maintainability Analysability
Function completeness
Unit test Functional Suitability Function correctness
Integration test Function appropriateness
Regression test
Reliability Maturity
Function completeness
Functional Suitability Function correctness
Function appropriateness
E2E test Operability
Usability
User error protection
Reliability Maturity
Trough out the CI process
Time behavior
Measurement of Performance Efficiency
Resource utilization
processing time
Maintainability Analysability
Activity in CD operations
Digital 2024, 4, FOR PEER REVIEW 6
install test Portability Installability

Figure 3.
Figure 3. Created
Created report
reportclass
class(upper
(upperisisprocess
processand lower
and is operation.
lower Square
is operation. frame
Square is developed
frame is devel-
area ).
oped area).

The generated XML file can be used with the command line generator tool to generate
a test result report in HTML format. The class also reads the .trx file of MSTest results
generated by Visual Studio and generates a list of API methods called in HTML format to
help understand test coverage.
Although it is possible to open a .trx file in Visual Studio and check the test results,
there are problems in analyzing the results as it is difficult to see the cause of the failure.
Digital 2024, 4 237
Figure 3. Created report class (upper is process and lower is operation. Square frame is developed
area ).

Thegenerated
The generatedXML XMLfilefilecan
canbebeused
used with
with the
the command
commandline line generator
generatortool tooltoto generate
generate
aa test
test result
result report
report in inHTML
HTML format.
format. The
The class
class also
also reads
reads the
the .trx
.trx file
file of
of MSTest
MSTest results
results
generated
generatedby byVisual
VisualStudio
Studioand andgenerates
generatesaalist listofofAPI
APImethods
methodscalled
calledin inHTML
HTMLformatformatto to
help
helpunderstand
understandtest testcoverage.
coverage.
Although
Although itit isis possible
possible to to open
open aa .trx
.trx file
file in
in Visual
Visual Studio
Studio andand check
check thethe test
test results,
results,
there are problems in analyzing the results as it is difficult
there are problems in analyzing the results as it is difficult to see the cause to see the cause of the failure.
We
Wedeveloped
developedthis thistime
timeto to
notnot
only generate
only generate the the
results of functional
results of functionaland performance
and performance tests
by outputting
tests reportsreports
by outputting that automatically describedescribe
that automatically the pass/fail list, the logs
the pass/fail list, during
the logsthe tests,
during
and the performance
the tests, measurement
and the performance results butresults
measurement also to but
convert
also this result information
to convert into
this result infor-
HTML
mationfiles
intoto visualize
HTML filesthe test resultthe
to visualize information, Figure 4.
test result information, Figure 4.

Figure4.4. Easy
Figure Easy visualization
visualizationof
oftest
testresults.
results.

Also,
Also,for
forthe
the API
API tests
tests created
created in
in JMeter,
JMeter,we we decided
decided to
to obtain
obtain performance
performancedata
data from
from
the
theAPI
API test
test execution
executionasaswell
well as
as the
the unit
unit tests
tests and
and evaluate
evaluate whether
whether the
the updated
updatedbuild
buildhashas
any
anyperformance
performancedegradation.
degradation.
Both
Both unit and
and integration
integrationtests
testswere
were able
able to produce
to produce thethe following
following benefits
benefits whilewhile
add-
adding performance efficiency assessments.
ing performance efficiency assessments.
•• Increaseddevelopment
Increased development productivity
productivity by
by being
being able
able to
to find
findfeatures
featuresand
andperformance
performance
andload
and loaddegrades
degradesin inthe
thebuild
buildpipeline;
pipeline;
•• Increased
Increasedtesting
testingefficiency
efficiencythrough
throughautomation;
automation;
•• Increased
Increasedcoverage
coverageof ofautomated
automatedtests
teststo
tocapture
capturemore
morequality
qualitydata.
data.
As an
As an example
example ofof this
this project,
project, we
we are
are using
using Ranorex
Ranorex [14]
[14] as
as an
an E2E
E2E testing
testing tool
tool to
to
automate
automate scenario testing. The scenarios include use cases that involve more
testing. The scenarios include use cases that involve more standard standard op-
erating procedures
operating procedurescalled
calledgolden
goldenroutes,
routes,useusecases
cases that
that check screen transitions
transitions when
when
functionsare
functions areinvoked,
invoked,andanduse
usecases
casesthat
thatresult
resultin
in errors
errorsdue
due toto operational
operational errors,
errors, so
so they
they
areconsidered
are consideredto toinclude
includeusability
usabilityevaluations.
evaluations.
In this
In thisway,
way,classifying
classifyingautomated
automatedteststestsusing
usingquality
qualitycharacteristics
characteristicsnot
notonly
onlyallows
allows
the quality improvement situation to take test coverage into account but
the quality improvement situation to take test coverage into account but also increases also increases
development
developmentproductivity.
productivity.

3. Enhance of Management of Test Results for Agile Projects

In software package development and service development, DevOps makes it possible
to increase development productivity and increase the frequency of product releases. To
ensure stable product releases, it is important to maintain the quality of the product and
the test results within the project must be well managed. Since the DevOps project has
achieved test automation with CI tools, we will also consider using a pipeline for test
result management.
Engineers can quickly detect quality degradation if the test management tool can save
the test results of unit tests, integration tests, and E2E tests as Jenkins jobs to be developed
and easily report comparisons with multiple past versions. The comparison with the
previous build is done in the build pipeline, and the comparison with past versions is easily
performed in the tool.
 By running it as a Jenkins job, you can automate the process of importing the results
of each test performed in the build pipeline and generating a test report.
The ability to manage test results and compare them with past versions makes it eas-
ier to ensure maturity and allows for the immediate detection of quality degradation,
Digital 2024, 4 whether functional or performance. 238
The quality guaranteed by the expanded coverage of test automation in the CI pro-
cess, the comparison process in the test result management tool, the distribution in the CD
process,
Whenandimporting
the development rules in
the test results, it the development
compares the test project are shown
results with in Table
the specified 3.
multiple
past Our projects
versions, are able
compares thetopass
automate
rate ofmore quality-related
the functional tasks,
tests and the such as static data,
performance analysis,
and
known
generatesas aSAST,
simpleand dynamic
report with analysis, known
the judgment as DAST,orthan
of whether not typical DevOps, although
the performance is within
the
it registered
requires somethreshold
time andrange.
effort The URL of the
to integrate generated
it into both theresult will be
CI process notified
and the CDtoprocess.
you via
Teams or e-mail.
Therefore, If you want
by preparing to check
multiple test the results of past
environments, versionstesting
resilience in detail, you to
is able canperform
use the
comparison function of the tool to generate
security-related testing for automated testing. a detailed report.
The image
Also, the useof of
thetest
jobresult
to process the import
management of makes
tools test results from Jenkins
it possible is shown
to monitor in
quality
Figure 5. from test results obtained in the CI process.
progress

Figure
Figure 5. Architecture
Architecture of test results management
management (The
(The red
red part
partisisthe
theprocess
processflow
flowand
andthe
theblue
bluepart
part
is theistest
the result
test result management
management process).
process).

By running it as a Jenkins job, you can automate the process of importing the results
of each test performed in the build pipeline and generating a test report.
The ability to manage test results and compare them with past versions makes it easier
to ensure maturity and allows for the immediate detection of quality degradation, whether
functional or performance.
The quality guaranteed by the expanded coverage of test automation in the CI process,
the comparison process in the test result management tool, the distribution in the CD
process, and the development rules in the development project are shown in Table 3.
Our projects are able to automate more quality-related tasks, such as static analysis,
known as SAST, and dynamic analysis, known as DAST, than typical DevOps, although it
requires some time and effort to integrate it into both the CI process and the CD process.
Therefore, by preparing multiple test environments, resilience testing is able to perform
security-related testing for automated testing.
Also, the use of test result management tools makes it possible to monitor quality
progress from test results obtained in the CI process.

Table 3. Quality covered by CI/CD pipeline process.

Quality Characteristics Quality Sub Characteristics Evaluation by

Functional Completeness
Function Suitability Functional Correctness ■ Unit test
■ Static analysis (SAST)
Functional Appropriateness
■ Integration test
Time Behaviour ■ Dynamic analysis (DAST)
Performance Efficiency Resource Utilization ■ Performance test
■ Regression test
Capacity ■ E2E test
Co-existence
Compatibility
Interoperability -
Digital 2024, 4 239

Table 3. Cont.

Quality Characteristics Quality Sub Characteristics Evaluation by

Appropriateness Recognizability -
Learnability -
Operability
Usability ■ Unit test
User Error Protection
■ E2E test
User Interface Aesthetics
Accessibility -
Maturity ■ Unit test
■ Static analysis (SAST)
■ Integration test
■ Dynamic analysis (DAST)
Availability
Reliability ■ Performance test
■ Regression test
■ E2E test
Fault Tolerance ■ Test results management
Recoverability -
■ Static analysis (SAST)
Confidentiality
■ Dynamic analysis (DAST)
Integrity ■ Regression test
Security Non-repudiation -
Authenticity -
Accountability -
Modularity
Reusability
■ Covered by the development process and
Maintainability Analysability
project charter
Modifiability
Testability
Adaptability -

Portability Installability ■ CD process

Replaceability -

4. Using Quality Characteristics for KPI under Agile Development Projects

We have always used quality characteristics as KPIs for project quality in waterfall
software development and have made quality progress visible [15]. In addition, this
quality management approach is aware of the international standards [16] used in software
certification and the international standards for software testing [17]. Therefore, the use of
quality characteristics as KPIs to understand the quality build has also been considered
for agile software development projects [18]. In the waterfall model, acceptance testing
is planned as a condition for starting testing as the entry criteria for each test level, and
acceptance testing is performed by extracting test cases from the test types to be performed
at that test level using a stratified method. Since it is impossible to perform acceptance
testing in every sprint of the agile process due to time constraints, we decided to investigate
the need for acceptance testing and the relationship between the tasks performed in each
sprint and quality characteristics through experiments.
We conducted two types of experiments on the impact on quality by conducting
software development in an agile process with the following measures to make more
effective use of quality characteristics in the agile process.
Digital 2024, 4 240

In the first team, the project manager chose the method of adding quality attributes
to the task ticket items so that quality attributes were set when each task was raised, and
sub-quality attributes were set for all test cases in the sprint. Second, many rules are defined
in the project charter for using quality characteristics for KPIs.
The project manager in the second team adds the definition for the analyzed functional
requirements and introduces a process for creating quality requirements at the time of the
requirements review, and the following activities are introduced:
• Designing quality requirements created at the time of the business requirements review
and quality requirements for the design and development of the software. Ensure
traceability of each issue so that the quality requirements created in the business
requirements review can be checked for validity in design and evaluation;
• Add quality attribute items to Jira and create a Kanban that quality attributes can
reference;
• Before starting a sprint, we created a sprint backlog by selecting tickets to be worked
on by the sprint from the product backlog containing the business requirements and
mapped the quality characteristics required by the business requirements.
In addition, the following practices are carried out for each sprint;
• Define the quality requirements for each functional requirement and set the criteria by
considering the metrics to be used;
• Evaluate the design and implementation checkpoints of the functional elements for
each quality attribute;
• Establish evaluation criteria for each quality characteristic and map them to the quality
characteristics to be ensured for each test activity;
• Visualize the progress of the established criteria by assessing the conformance of the
quality characteristics at each sprint.
Before starting the first sprint, the project prepared a mapping of activities and quality
characteristics, as shown in Figure 6.
Also, the project begins with the following quality objectives:
• Projected development scale: 2.6 KL;
• Review density: 20 man-hours/KL;
Digital 2024, 4, FOR PEER REVIEW 10
• Density of test items: 150 items/KL;
• Number of bugs: 10/KL.
suitability
Functional
efficiency
Performance
Compatibility
Usability
Reliability
Security
ability
Maintain-
Portability

Quality
characteristics

Activities
Each sprint
Spec analysis ○ ○ ○ ○ ○ ○ ○ ○
Design ○ ○ ○ ○ ○ ○ ○ ○
Coding ○ ○ ○ ○ ○ ○ ○ ○
Static analysis ○ ○
Unit test ○ ○ ○ ○
Integration test ○ ○ ○ ○
Security test ○
Performance test ○
E2E test ○ ○ ○
Only final sprint
Regression test ○ ○
Release test ○ ○ ○ ○ ○ ○ ○ ○

Figure 6. Mapping
Figure 6. Mapping quality
quality characteristics
characteristics to
to activities.
activities.
5. Results of Two Practices
5. Results of Two Practices
The first project did not establish rules for using quality features in the project and left
The first project did not establish rules for using quality features in the project and
it up to the developers, which resulted in many tasks being delayed due to the overhead of
left it up to the developers, which resulted in many tasks being delayed due to the over-
head of using quality features. In addition, most tasks were biased toward functional con-
formance. This result is likely to occur in the waterfall model as well, and without consid-
ering the balance of each quality characteristic, the result will be biased toward functional
conformance, so efforts are being made to reduce quality characteristics to 70% or less. It
was found that simply using quality characteristics would only complicate the work and
Digital 2024, 4 241

using quality features. In addition, most tasks were biased toward functional conformance.
This result is likely to occur in the waterfall model as well, and without considering
the balance of each quality characteristic, the result will be biased toward functional
conformance, so efforts are being made to reduce quality characteristics to 70% or less. It
was found that simply using quality characteristics would only complicate the work and
would not be beneficial.
On the other hand, in the second project, where the use of quality characteristics
was specified in the project charter, and the method of using quality characteristics was
defined, it became possible to break down the quality status and monitor progress. As
a result, smooth project management was realized without disturbing the balance of
QCD, and very good results were achieved. In the case of this project, the use of quality
characteristics increased the testing density by 144% compared to the case where quality
characteristics were not used, resulting in an increase in testing man-hours, but the number
of defects detected decreased to 32% of the predicted value. This was a result of the quality
characteristics of KPI and the implementation of quality-driven development. In addition,
the tasks to be implemented were able to proceed as planned. In other words, using quality
characteristics as KPIs makes it possible to proceed with quality-conscious development
while being aware of the QCD balance.
Although this experiment was conducted on a small scale of 3.7 KL, similar results
can be obtained by determining the quality characteristics to focus on for each sprint and
proceeding with development with an awareness of the quality priority order. It was also
suggested that the same effect could be achieved by being aware of quality prioritization,
such as determining the quality characteristics to focus on in each sprint.
The benefits of using quality characteristics from the experiment are as follows:
• When using quality characteristics in the agile process, it is assumed that if they
are applied to a project for which a project plan has already been drawn up, the
overhead of using quality characteristics is large and is likely to significantly affect the
sprint’s activities;
• If the use of quality features is considered from the project planning stage, the following
actions can be taken to provide evidence of quality assurance to project management;
• Declaring the use of quality features in the project charter;
• Assigning quality characteristics to different activities;
• The overhead of using quality characteristics should be included in the rough estimate
before the project starts.
Finally, an example of a qualitative analysis of a product by quality characteristic in
the project is given in Table 4.

Table 4. Example of quality analysis classified by quality characteristics.

Quality Characteristics Quality Analysis

Functional suitability is ensured because the functional requirements that satisfy the functional
Function
requirements are considered in the design document, implemented, and confirmed to work properly
suitability
through various tests.
Compared to the operation before the enhancement, the performance of the functions implemented
in the previous version has not deteriorated, and the import of new reports, a new function, is
Performance
comparable to the import speed of the report functions implemented in the previous version, thus
efficiency
meeting the performance requirements. As load and capacity requirements have not been verified for
this project, it can be concluded that ensuring only performance efficiency is not a problem.
There are no problems with interfacing with other systems, such as EXCEL output and ZIP
Compatibility compression, and compatibility is ensured because there are no problems with co-existence with
other systems in the same environment and no code-level implementation that would affect others.
The enhancement does not create new screens but enhances existing screens and identifies no new
Usability
usability issues, ensuring usability.
Digital 2024, 4 242

Table 4. Cont.

Quality Characteristics Quality Analysis

Reliability is ensured because functional requirements are met throughout the process:
requirements->specification->design->implementation->testing
Reliability
The quality of the requirements developed during the requirements review is ensured by the
associated activities, and any bugs found during the various reviews and tests are fixed.
The same level of security as the previous version has been confirmed by testing. The same level of
Security
security as the previous version is guaranteed.
Maintainability is ensured because it is based on coding conventions, various documents are written
Maintainability in the same format as the previous version, and the level of description is written at the same or
higher granularity than the previous version.
Portability The installation procedure is the same as the previous version. Portability is, therefore, guaranteed.

6. Conclusions
Regardless of whether the development process used in a project is the cascade model
method or the agile method, quality-oriented development can be achieved by using
quality characteristics.
In the case of agile processes, it is possible to confirm that quality is continuously im-
proving by defining the metrics to be achieved in each pipeline in the CI/CD pipeline. From
the test engineer’s perspective, more time can be spent on test design and automated script-
ing, leaving more time for exploratory testing. In addition, the results of automated tests
become the criteria for initiating manual tests, allowing them to efficiently focus on more
complex manual tests. This allows them to efficiently focus on further improving quality.
In an agile process, it is important to design and execute testing activities efficiently in
the CI/CD pipeline and to clarify in the sprint plan the quality characteristics that should be
prioritized in each sprint. By clarifying the quality characteristics that should be prioritized
in each sprint in the sprint plan and by developing a method to use SQuaRE [19] in the
agile process, it can be recommended to build quality more efficiently.
In order to use quality characteristics as a KPI in large-scale agile development, it is
necessary to prepare a KANBAN for this KPI and check the quality status of each team
in a centralized manner. It will be necessary to educate all project members about quality
characteristics, specify the purpose and means of using quality KPIs in the project charter,
and ensure that project members understand and follow this project charter.

Author Contributions: Conceptualization, methodology, validation, formal analysis, investigation,

resources, data curation, writing—review and editing, D.K.; supervision, H.I. All authors have read
and agreed to the published version of the manuscript.
Funding: This research received no external funding.
Data Availability Statement: Data are contained within the article.
Conflicts of Interest: Author Daiju Kato was employed by the company Nihon Knowledge Co., Ltd.
The remaining authors declare that the research was conducted in the absence of any commercial or
financial relationships that could be construed as a potential conflict of interest.

References
1. DevOps, 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr. 2009. Available online: https://www.slideshare.net/jallspaw/
10-deploys-per-day-dev-and-ops-cooperation-at-flickr (accessed on 15 January 2022).
2. Jenkins. Available online: https://www.jenkins.io/ (accessed on 1 February 2020).
3. ISO/IEC 25010; Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—
System and software quality models. ISO: Geneva, Switzerland, 2011.
4. ISO/IEC 25000; Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—
Guide to SQuaRE. ISO: Geneva, Switzerland, 2014.
Digital 2024, 4 243

5. Ito, J.; Yamaguchi, S.; Okazaki, K.; Yokosuka, S.; Kimoto, K.; Yamanaka, M.; Nagata, A.; Yamaguchi, T.; Hosoya, S. Quality
Assurance by Quality Stepwise Refinement in Agile Development. Report on the Results of the Subcommittee Meeting of the
Software Quality Control Research Group. 2018, pp. 137–144. Available online: https://www.juse.or.jp/sqip/workshop/report/
at-tachs/2018/4_aqa_ronbun.pdf (accessed on 15 January 2022). (In Japanese).
6. JIS X 25051; Software Engineering-Systems and Software Quality Requirements and Evaluation (SQuaRE)-Requirements for
Quality of Ready to Use Software Product (RUSP) and Instructions for Testing. JISC: Tokyo, Japan, 2016.
7. Cohn, M. The Forgotten Layer of the Test Automation Pyramid. Available online: https://www.mountaingoatsoftware.com/
blog/the-forgotten-layer-of-the-test-automation-pyramid/ (accessed on 1 February 2022).
8. ISO/IEC 25030; Systems and Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—
Quality Requirements Framework. ISO: Geneva, Switzerland, 2019.
9. GitHub. Available online: https://github.com/ (accessed on 1 February 2022).
10. Atlassian. Available online: https://www.atlassian.com/ (accessed on 1 February 2020).
11. Kato, D.; Shimizu, A.; Ishikawa, H. Quality classification for testing work in DevOps. In Proceedings of the 14th International
Conference on Management of Digital EcoSystems (ACM MEDES 2022), Venice, Italy, 19–21 October 2022.
12. MSTest. Available online: https://github.com/Microsoft/testfx-docs/ (accessed on 1 February 2021).
13. NUnuit. Available online: https://nunit.org/ (accessed on 1 February 2021).
14. Ranorex. Available online: https://ranorex.com/ (accessed on 1 February 2020).
15. Kato, D.; Ishikawa, H. Develop quality characteristics based quality evaluation process for ready to use software projects.
Com-put. Sci. Inf. Technol. 2016, 6, 9–21. [CrossRef]
16. ISO/IEC 25051; Software Engineering—Systems and Software Quality Requirements and Evaluation (SQuaRE)—Requirements
for Quality of Ready to Use Software Product (RUSP) and Instructions for Testing. ISO: Geneva, Switzerland, 2014.
17. ISO/IEC/IEEE 29119-3; Software and Systems Engineering—Software Testing—Part 3: Test Documentation. ISO: Geneva,
Switzerland, 2021.
18. Kato, D.; Okuyama, A.; Ishikawa, H. Introduction of test management based on quality characteristics. In Proceedings of the
1st Inter-national Workshop on Experience with SQuaRE Series & Their Future Direction IWESQ 2019), Putrajaya, Malaysia, 2
December 2019.
19. Shang, W. Bridging the divide between software developers and operators using logs. In Proceedings of the 34th International
Conference on Software Engineering (ICSE‘12), Zurich, Switzerland, 2–9 June 2012.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.