2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

Complex Software Testing Analysis using

International Standards
Satoshi Masuda Yasuharu Nishi Kazuhiro Suzuki
IBM Research University of Electro- Hitachi, Ltd.
Tokyo, Japan Communications Kanagawa, Japan
[email protected] Tokyo, Japan [email protected]
[email protected]

Abstract—International standards for software testing have that the working memory of an average human can only gold
been published in 2013 and are now widely used in industry seven plus or minus two objects [10]. The proposed method
and academia. However, these standards are difficult to use for defines the testing views that group the test types according to
the analysis of testing for complex software. Complex software
has a huge number of components and an enormous number the software architecture standards and defines the contents of
of relations between them. If we analyze such complex software testing views based on software requirements specifications. In
according to the software testing standards, the results of the this paper, we discuss a case study of applying the proposed
analysis will have a huge number of testing patterns or cases. method to the existing system that is“ bridging resources and
Therefore, planning a project of complex software testing is im- agencies in large-scale emergency management”, the BRIDGE
possible, and even if we could plan it, the plan would be infeasible.
We propose a method of testing analysis for complex software system [11].
by using definitions of software architecture in international The original contributions of this work are as follows:
standards. We believe that the outcomes of the proposed method • We established a method that supports the analysis pro-
are software-test-architecture. The software-test-architecture helps cess for complex software testing.
to analyze complex software, plan software testing, and develop
• We demonstrated an application of the proposed method
test cases. We select existing case studies that have testing on a
large and complex software, and discuss techniques to apply the in an existing system.
proposed method for the case studies. The paper is organized as follows. In Section II, we present
Index Terms—software testing, international standard, com- related work on international standards and complex systems.
plex software, software test architecture
In Section III, we give an overview and details of the proposed
method. In Section IV, we present a case study and describe
I. I NTRODUCTION
the target system. We conclude the paper in Section VI.
Software is present in many spheres of our society such
as online banking system, internet shopping, and social net- II. R ELATED WORK
work services. Software is getting more sophisticated due to A. Emerging idea of defining software-test-architecture
complex requirements and software integration. Such complex Nishi et al. proposed a test architecture design [12], [13].
software has a huge number of functions and the layers of They argue that software-test-architecture is used for the inside
the functions [1]. International architecture standards [2], [3] of test suites, which mainly consist of groups of test cases
defines architecture descriptions and processes for analyzing such as parameters of combinatorial tests, test conditions,
complex software. International standard on requirement en- test levels and test types. Hagar proposed a definition of
gineering [4] includes definitions of the requirement process software-test-architecture [14], [15]. He referred to the def-
that can be applied to complex software. initions of phrases: architecture, software architecture, and
Software testing is the major activity of system and software software testing in the vocabulary of software engineering
engineering. International standards of software testing [5]–[8] and standards such as the syllabus of International Software
have been published in 2013 and are widely used in industry Testing Qualification Board (ISTQB) [16]. He came to a
and academia [9]. However, these testing standards are difficult conclusion that software-test-architecture is both the process
to use for the analysis of testing in the case of complex and the product(s) of planning, designing, and constructing
software. The number of requirements and functions is so tests and supporting test structures.
huge, it is impossible for a human to analyze the importance
of testing. The testing standards do not provide guidance B. Software testing standards
on how to analyze the importance of the requirements and ISO/IEC/IEEE 29119 Software Testing standards, part 1
functions. We propose a method of testing analysis for com- through 4 ( 29119-1, 2, 3, and 4 ) define concepts and
plex software using the knowledge of software architecture definitions of software testing, test processes, test documen-
and requirements engineering in international standards. We tation and test techniques [5]–[8]. Standard 29119-1 defines
employ a psychological premise, the so called Miller’s law, some terms that are used in this paper as follows. Test level

978-1-7281-1075-2/20/$31.00 ©2020 IEEE 241

DOI 10.1109/ICSTW50294.2020.00049
Authorized licensed use limited to: Federation University Australia. Downloaded on April 21,2024 at 13:01:30 UTC from IEEE Xplore. Restrictions apply.
  
 of structure based metrics that measure the attribute in (1), and
   (3) the degree to which a system or a component has a design
 
  or implementation that is difficult to understand and verify.
 
 Aiguier et al. proposed a mathematical denotation is proposed
 
    
 
for the notion of complex software systems [17]. Complex
systems are described in a recursive way as an interconnection
 
  
 
of subsystems. These definitions imply difficulties for a human
 
   
  to analyze the requirements or the design of the software in
software engineering that includes software testing, due to the
         complexity of the software. Therefore, software architecture
   
has been studied to enable the analysis of requirements and
the design for such complex software.
Fig. 1. Test context diagram. Edited Figure 1 in [5].
D. Software architecture standards
International standard [2] defines architecture descriptions
is a specific instantiation of a test process; for examples, as fundamental concepts or properties of a system in its
component test level, integration test level, system test level, environment embodied in its elements, relationships, and in
and acceptance test level. Test type is a group of testing ac- the principles of its design and evolution. Architecture descrip-
tivities that are focused on specific quality characteristics, for tions are work products of systems and software architecting.
example; security testing, functional testing, usability testing, The standard defines two important terms: architecture view
and performance testing. Standard 29119-2 defines software and architecture viewpoint. The architecture view is a work
testing processes that includes test planning, test designing, product expressing the architecture of a system from the
test execution, and test reporting. The test planning consists perspective of its specific system concerns. The architecture
of such tasks as understanding of the context and the soft- viewpoint is a work product establishing the conventions for
ware testing requirements, organizing test plan development, the construction, interpretation and use of architecture views
identifying and analyzing risks, identifying risk mitigation to frame specific system concerns. The standard mentions
approaches, designing test strategies, determining staffing and an architect who constructs views of the system-of-interest
scheduling, recording test plans, and gaining consensus on and integrates these views within an architecture description.
test plan. However, the test planning process does not include Another international standard [3] defines the architecture
any analysis of the system-under-test (SUT). If the SUT is process as consisting of three tasks such as I) architecture con-
a complex software, it has a huge number of components ceptualization: characterize the problem space and determine
and an enormous number of relations between them. If we suitable solutions that address stakeholder concerns, achieve
analyze such complex software according to the software architecture objectives, and meet relevant requirements, II)
testing standards for planning software testing, the results of architecture evaluation: determine the extent to which one or
the analysis will also have a huge number of testing patterns more architectures meet their objectives, address stakeholder
or cases. Therefore, planning a project involving complex concerns, and meet relevant requirements, and III) architecture
software testing is impossible, or even if it were possible, elaboration: describe or document an architecture in a suffi-
such a plan would be infeasible. Standard 29119-3 defines test ciently complete and correct manner for the intended uses of
documents, and Standard 29119-4 defines test techniques. The the architecture.
test techniques include, for instance, equivalence partitioning, The international standards for software testing mentioned
boundary value analysis, and branch condition combination above do not cover standards for architecture processes.
testing. These techniques have low level techniques to analyze Hence, we propose a method for software-test-architecture.
the components in detail, but they do not cover high level
analysis such as evaluating complex software in software III. P ROPOSED METHOD FOR
testing, designing complex software testing, and determining SOFTWARE - TEST- ARCHITECTURE
test levels and types. Figure 1 shows a test context diagram in Figure 2 shows the proposed method for developing
29119-1 that has relations between the test plan, test strategy, software-test-architecture based on the international standards
test techniques, test level, and test types. Examples of test for architecture [2], [3]. The method consists of conceptual-
types are security testing, functional testing, usability testing, ization, evaluation and elaboration processes. These processes
and performance testing. are performed iteratively and repeatedly to determine problems
and solutions, evaluate the solutions, and elaborate on them.
C. Complex software The proposed method has three core processes: software-
International standard [1] defines the complexity of software test-architecture conceptualization, software-test-architecture
as: (1) the degree to which a system’s design or code is evaluation, and software-test-architecture elaboration.
difficult to understand because of numerous components or Software-test-architecture conceptualization process is to
relationships among components, (2) pertaining to any of a set “ characterize the problem space and determine suitable

242

Authorized licensed use limited to: Federation University Australia. Downloaded on April 21,2024 at 13:01:30 UTC from IEEE Xplore. Restrictions apply.
     
    "$ 
# "!
  
 

   
    #"
  
 
  !"%
# "!
 
 

   
# "!
 " 
# "!
 
"!
# "!
    
       
    ! !" "!
   
    

   
    
   "$ 
!%!"
"" #"!
   
 # "  "

Fig. 2. Method of software-test-architecting. The method consists of three

processes: Software-test-architecture conceptualization, architecture evalua- Fig. 4. Example of software requirements specification.
tion, and architecture elaboration. The processes are performed iteratively and Edited Figure 8 in [4].
repeatedly.
     TABLE I
E XAMPLES OF TESTING VIEWS GROUPING TEST TYPES .

     
 
Test type / Funct- Usab- Inter- Regre- Perfor- Secu- ...
  Testing view ional ility face ssion mance rity
     
 
Functional x x x ...


   


  Usability x x ...
Performance x x ...
  Interface x x x ...
Database x x x ...

  Note: ‘ x ’ means that the view includes the test type.



 

Fig. 3. Conceptualize software-test-architecture. • Analyze the requirements of the system-under-test with

testing views.
• Develop selected test types as a software test architecture.
solutions that address stakeholder concerns, achieve These steps are not always executed in order, but they are
architecture objectives, and meet relevant requirements ” executed iteratively. When we analyze the software-under-
[3]. The conceptualization process has tasks defined in testing, the test types and levels should be considered [6], [16].
software testing that include a) problem space definition: The number of combinations between the test types and test
define the boundaries of software to be tested, b) architecture levels in complex systems often exceeds dozens easily. The
objective: determine the most important factor costs, delivery working memory of a human brain can hold seven objects
quality, security, and performance, c) critical success criteria: on average; therefore, it is extremely difficult to consider the
determine key metrics of software testing, and d) solution trade-off of the test types and levels in this case. Hence,
space definition: determine the importance and the ways of to reduce the number under consideration, we employ the
testing test levels and types such as platform, tools, and data. testing view that groups test types according with the software
The software-test-architecture evaluation process objective is architecture standard [2] and defines the contents of the testing
to “ determine the extent to which one or more architectures view based on software requirement specifications in Figure 4
meet their objectives, address stakeholder concerns, and that has defined in [4]. The number of the requirement types
meet relevant requirements ” [3]. The evaluation process is limited, so when we select one upper requirement type, the
has tasks that we define which include a) evaluation results: number of the leaf requirements types does not exceed seven,
compare some patterns of the importance of the test level e.g. when we select product requirements, the requirement
and types with the ways of testing and determine which types are usability, performance, space, dependability, and
one achieves the objective, b) evaluation findings: inform security. Table I shows an example of the testing views that
about the findings of the software testing while evaluating, group the test types. The proposed method uses these testing
and c) recommendations: inform about the recommendations views to analyze the requirements instead of test types.
for improvement of the software testing architecture. The
objective of the software-test-architecture elaboration process IV. C ASE STUDY
is to revise the software-test-architecture. We define the We apply the proposed method to an existing complex
task as a) architecture views: elaborate test levels and types system as a desktop study. We choose“ bridging resources and
according to the evaluation results, and b) architecture agencies in large-scale emergency management ” (BRIDGE)
descriptions: describe test levels and types. system [11] as the complex system. Figure 5 shows the
Figure 3 shows the detail of software-test-architecture con- system context of the BRIDGE system. The BRIDGE system
ceptualization process. The process includes the following aims to bridge different resources and agencies in large-
steps: scale emergency rescue management and to supply a strong

243

Authorized licensed use limited to: Federation University Australia. Downloaded on April 21,2024 at 13:01:30 UTC from IEEE Xplore. Restrictions apply.
  # "#

## TABLE II
" I NITIAL SERVICES IN BRIDGE SYSTEM .
E DITED A PPENDIX A. IN [19]
(
## "#

"#$"$  #$"$ Main services Sub-services

""#

 " '     $"  Resource Management Resources
"# "$ 
&$ $ Available resources
Remote Resource Allocation
 %$  Resource allocation service
$# 
%#

Delegate Resource
##
#$" #%#"
Where is ‘ my ’ resources
$' "
$ Query Person/resource
 #!$
Task management Direct resources
$

 
$
Knowledge Management Tips of the trade
"&
Routines

$#  $$ 
$ 
Information Management Information Timeline
$' "
"
#$

 "$  $# #$ "
Last heard of
Timestamp
Identity
Distribution
(!"$# % Workflow WF Generator
 %
WF Composition
Agent instantiation and configuration
WF Monitoring
Weather Forecast
Wind
Rain
Fig. 5. BRIDGE system of systems context. Edited Figure 1 in [24]. Environment Monitoring Humidity
Temperature
Vibration
Oxygen level
Toxic level
decision support to hundreds of professionals from various Victims Where should I go next
organizations. The development project had started on 1 April Nearest safe area
Health Monitoring Vital signs
2011 and continued for 48 months. The total cost of the Triage (PPHDT) Portable Patient Health Data Transfer
project was e18,075,144. Gao et al. analyzed the architecture Incident Information Forecast number of injured in a certain area
Incident details
complexity of the BRIDGE system and described it as “ a Security System Protection (‘ Red button ’)
Map Geography
model-based automated support system as well as a dynamic Building
Danger
workflow composition and communication support system. White spots
This system consists of several components, including front- Plot on map
Location Positioning
ends or applications, back-ends and configuration tools, col- Coordinates
Remote (Device) Control Re‐Direction
laborating systems, information repositories and networking Remote Control of public Web camera
tools. ” [18]. Log Services Forget/destroy data
Search logs
The services of the BRIDGE system were defined during the Activity logging
architecture workshops based on requirements analysis [19]. Event logging
Forget/destroy
Table II shows the main and sub services in the BRIDGE data
Risk Management Provide risk warnings
system. There are 27 main services and 93 sub services. We International Aspects Language
apply our proposed method to analyze the testing of services Organisation
Culture
as requirements. Testing in the BRIDGE project has been Transformation Conversion (MM)
Information tailoring
reported in related works [20]–[23]. Al-Akkad et al. reported Site Information Retrieval External information retrieval
on the design and evaluation of an ad-hoc light weight S.O.S Building information
Land surveying
system for the BRIDGE system [21]. Table III shows the actual Information Sharing datAgent
Alert/Alarm/Notification Sensor alert
results of system requirements in the BRIDGE system. These Notify ICE
requirements correspond to the evaluation (testing) results, so Alarm button
Personal alerting
we call them actual-test-views. Change monitoring
Track and Trace Remember this
We analyze the services and select testing views for each Checked spaces
service. This analysis requires experiences and knowledges Track
Geo-tag
to determine which testing view is best for which services. Network and Communication Online
However, the number of the testing views is small enough Opportunistic networking
QoS
to consider the trade-off between of the testing views with Connect/Disconnect
Availability
conditions such as project duration, costs, and testing skills. Service
Table IV shows the results of testing views for the services in Social Media Social Respond
Filter false rumors
the BRIDGE system. The numbers of the testing views that Twitter
Religious counselling for victims?
should be used in requirements indicates the importance of Social media
testing for the system. Thus, the testing views in this case study Crowd sourcing
Access to social media
in order of importance are functionality, interface, usability, Expert Localization Services Request advice
Find expert
database, and performance. We call this list proposed-test- Modelling Services Create model
views. As described in section III, Table IV is the result of Access model archive

244

Authorized licensed use limited to: Federation University Australia. Downloaded on April 21,2024 at 13:01:30 UTC from IEEE Xplore. Restrictions apply.
 TABLE III analysis. Proposed-test-views can be used in test planning
ACTUAL RESULTS : S ET OF SYSTEM REQUIREMENTS IN [21]. T HESE
REQUIREMENTS CORRESPOND TO EVALUATION ( TESTING ) RESULTS .
for the importance of test types. Test planning process in
the international testing standards includes understanding the
ID Description context activity. In the activity, a task to understand the context
R1 Operate under harsh conditions in terms of temperature, and the software testing requirements shall be obtained to
vapor, or vibration.
R2 Operate despite environmental impacts that weaken commu-
support the preparation of the test plan with applicable product
nication signals or hamper sensor reading. documentation such as system requirement specifications for
R3 Work reliably and accurately as possible. test types [6]. Our proposed method improves this activity for
R4 Only augment users in their actions with the physical world, complex system.
but not to distract them.
R5 Use established technologies to exploit opportunities, e.g.
everyday services as Twitter or commercial OSs. VI. C ONCLUSION
R6 Enable quick construction of networks requiring no cumber-
some configuration efforts. In this paper, we proposed a method of complex software
R7 Operate fault-tolerant to cope with limitations, such as inter- testing analysis by using definitions of software architecture in
mittent disconnections.
R8 Facilitate distribution of data to co-existing networks.
international standards. We applied the method to a complex
R9 Support dynamic re-configurations of networks as needs system as a desktop study and discussed about the result. As
rapidly change in the aftermath of a crisis. future work, we would like to apply the proposed method to
actual projects. The results of the actual project will help us
to expand our idea to more practical uses.
TABLE IV
C ASE STUDY RESULTS : A NALYSIS OF TESTING WITH TESTING VIEWS FOR
THE BRIDGE SYSTEM SERVICES R EFERENCES
Testing views Functio- Usability Perform- Interface Database [1] “Iso/iec/ieee international standard - systems and software engineering–
nal ance vocabulary,” ISO/IEC/IEEE 24765:2017(E), pp. 1–541, Aug 2017.
BRIDGE services
[2] “Iso/iec/ieee systems and software engineering – architecture descrip-
Resource management x
Task management x x
tion,” ISO/IEC/IEEE 42010:2011(E) (Revision of ISO/IEC 42010:2007
Knowledge management x x and IEEE Std 1471-2000), pp. 1–46, Dec 2011.
Information management x x [3] “Iso/iec/ieee international standard - software, systems and enterprise –
Workflow x x architecture processes,” ISO/IEC/IEEE 42020:2019(E), pp. 1–126, July
Weather x x x 2019.
Environment monitoring x x
Victims x x x [4] “Iso/iec/ieee international standard - systems and software engineering
Health monitoring x x – life cycle processes – requirements engineering,” ISO/IEC/IEEE
Triage x x 29148:2018(E), pp. 1–104, Nov 2018.
Incident information x x x [5] “Iso/iec/ieee international standard - software and systems engineering –
Security x
software testing –part 1:concepts and definitions,” ISO/IEC/IEEE 29119-
Map x x x
Location x 1:2013(E), pp. 1–64, Sep. 2013.
Remote (Device) control x x [6] “Iso/iec/ieee international standard - software and systems engineer-
Log services x x ing –software testing –part 2:test processes,” ISO/IEC/IEEE 29119-
Risk management x 2:2013(E), pp. 1–68, Sep. 2013.
International aspects x x
Transformation x
[7] “Iso/iec/ieee international standard - software and systems engineering
Site Information retrieval x x – software testing –part 3: Test documentation,” ISO/IEC/IEEE 29119-
Information sharing 3:2013(E), pp. 1–138, Sep. 2013.
Alert/Alarm/Notification x x x [8] “Iso/iec/ieee international standard - software and systems engineering–
Track and Trace x x x software testing–part 4: Test techniques,” ISO/IEC/IEEE 29119-4:2015,
Network and communication x x
Social media x x x
pp. 1–149, Dec 2015.
Expert localization services x x [9] S. Condor, C. Garcı́a, and A. Dávila, “Adoption of iso/iec 29119-
Modelling services x x 2 software testing practices: An exploratory analysis in organizations
Total number of ’x’ 23 8 2 12 8 in lima, perú,” in 2016 International Conference on Software Process
Note: ‘ x ’ means that services should use the view Improvement (CIMPS), Oct 2016, pp. 1–8.
[10] G. A. Miller, “The magical number seven, plus or minus two: Some
limits on our capacity for processing information,” The Psychological
Review, vol. 63, no. 2, pp. 81–97, March 1956. [Online]. Available:
the iterative analysis of requirements with testing views. We http://www.musanim.com/miller1956/
[11] B. project. (2016) About bridge. bridge at aglance. 21st Dec. 2019
can decide the details of test types for planning the testing accessed. [Online]. Available: http://www.bridgeproject.eu/en/about-
according to the importance. bridge
[12] Y. Nishi, T. Katayama, and S. Yoshizawa, “Combinatorial test architec-
V. D ISCUSSION ture design using viewpoint diagram,” in 2013 IEEE Sixth International
Conference on Software Testing, Verification and Validation Workshops,
In this section, we discuss about the actual-test-views and March 2013, pp. 295–300.
the proposed-test-views. The actual-test-views are described [13] Y. Nishi, “Difference in quality of test architecture between service
providers and subcontractors,” in 2016 IEEE Ninth International Con-
in natural language, so we need to break them down into ference on Software Testing, Verification and Validation Workshops
concrete test types when we plan testing for the system. (ICSTW), April 2016, pp. 14–16.
For example, “ R1:Operate under harsh conditions in terms [14] J. D. Hagar, “Defining the phrase ”software test architecture” emerging
idea,” in 2017 IEEE International Conference on Software Testing,
of temperature, vapor, or vibration. ” can be broken down Verification and Validation Workshops (ICSTW), March 2017, pp. 313–
into test types of Functional and Interface by the testing 316.

245

Authorized licensed use limited to: Federation University Australia. Downloaded on April 21,2024 at 13:01:30 UTC from IEEE Xplore. Restrictions apply.
[15] ——, “Software test architectures and advanced support environments
for iot,” in 2018 IEEE International Conference on Software Testing,
Verification and Validation Workshops (ICSTW), April 2018, pp. 252–
256.
[16] I. S. T. Q. Board, Certified Tester Foundation Level Syllabus Version
2018 V3.1, K. Olsen, T. Parveen, and R. Black, Eds. ISTQB, 2018.
[17] M. Aiguier, P. L. Gall, and M. Mabrouki, “A formal definition of
complex software,” in 2008 The Third International Conference on
Software Engineering Advances, Oct 2008, pp. 415–420.
[18] L. Gao, M. Warnier, S. van Splunter, L. Chen, and F. M. T. Brazier,
“Architectural complexity analysis for large-scale emergency rescue
management systems: A preliminary study,” in 2015 International
Conference on Complex Systems Engineering (ICCSE), Nov 2015, pp.
1–6.
[19] B. project., “D04.2: Functional view on the bridge archi-
tecture,” 2013, 21st Dec. 2019 accessed. [Online]. Available:
http://www.bridgeproject.eu/en/bridge-results/deliverables/d042
[20] C. Raffelsberger and H. Hellwagner, “Evaluation of manet routing
protocols in a realistic emergency response scenario,” in Proceedings of
the 10th International Workshop on Intelligent Solutions in Embedded
Systems, July 2012, pp. 88–92.
[21] A. Al-Akkad, L. Ramirez, A. Boden, D. Randall, and A. Zimmermann,
“Help beacons: Design and evaluation of an ad-hoc lightweight s.o.s.
system for smartphones,” Conference on Human Factors in Computing
Systems - Proceedings, 04 2014.
[22] I. M. Haugstveit, E. L. Rake, and A. W. Eide, “Practitioner-centered,
long-term testing of an ict-based triage system for emergency manage-
ment,” in ISCRAM, 2015.
[23] A. Al-Akkad, Working around disruptions of network infrastructures:
Mobile Ad-Hoc systems for resilient communication in disasters, 01
2016.
[24] B. project., “D08.2: Integrated and quality assured bridge
platform,” 2015, 21st Dec. 2019 accessed. [Online]. Available:
http://www.bridgeproject.eu/en/bridge-results/deliverables/d082

246

Authorized licensed use limited to: Federation University Australia. Downloaded on April 21,2024 at 13:01:30 UTC from IEEE Xplore. Restrictions apply.