Available online at www.sciencedirect.

com
Available online at www.sciencedirect.com
Available online at www.sciencedirect.com

ScienceDirect
Procedia Computer Science 00 (2022) 000–000
Procedia
Procedia Computer
Computer Science
Science 21400 (2022)
(2022) 000–000
236–247 www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia

9th International Conference on Information Technology and Quantitative Management

9th International Conference on Information Technology and Quantitative Management
Bibliometric
Bibliometric study
study on
on multi-criteria
multi-criteria decision-making
decision-making methods
methods
applied to life cycle management of defense systems
applied to life cycle management of defense systems
Romullo Girardia,∗, Juraci Ferreira Galdinoaa , Paulo César Pellandaaa , José Adalberto
Romullo Girardia,∗, Juraci Ferreira
França Galdino
Juniorb , Paulo César Pellanda
a , José Adalberto
b , Marcos dos Santosa
a
França Junior , Marcos dos Santos
Military Institute of Engineering (IME), Praça General Tibúrcio, 80, Urca, Rio de Janeiro, 22290-270, Brazil
a
b Innovation andMilitary Institute
Technology Agencyof Engineering (IME),
of the Brazilian Praça
Army GeneralAvenida
(AGITEC), Tibúrcio,
das80, Urca, Rio
Américas, de Janeiro,
28.705, 22290-270,
Guaratiba, Rio deBrazil
Janeiro, 23020-470,
b Innovation and Technology Agency of the Brazilian Army (AGITEC), Avenida das Américas, 28.705, Guaratiba, Rio de Janeiro, 23020-470,
Brazil
Brazil

Abstract
Abstract
Complex environments, conflicting criteria, uncertainties, and inaccurate information are some characteristics of many nowadays
Complex
real-worldenvironments,
decision-making conflicting
problems. criteria, uncertainties, and
The Multi-Criteria inaccurate information
Decision-Making (MCDM)are some characteristics
methodology of many
is one of the nowadays
most used, per-
real-world decision-making problems. The Multi-Criteria Decision-Making (MCDM) methodology is one
haps because it drives decision-making processes to be more rational, efficient, and effective. Thus, the Armed Forces of several of the most used, per-
haps
countriesbecause
haveitused
drives
MCDMdecision-making
methods in theprocesses to beManagement
Life Cycle more rational, efficient,Systems
of Defense and effective. Thus,tothe
(LCM-DS) Armed
deal Forces ofsystems’
with military several
countries
complexity, have used MCDM
breadth, methods However,
and sensitivity. in the Lifedespite
Cycle Management
the rich and of Defense
vast Systems
literature (LCM-DS)
available to deal
on several MCDMwith military
methodssystems’
applied
complexity, breadth, and sensitivity. However, despite the rich and vast literature available
to LCM-DS problems, there is a lack of a recent bibliometric study on this relevant subject. This paper presents on several MCDM methods applied
an analysis of
atoquantitative
LCM-DS problems,
approach there
basedisona lack of a recent
bibliometric bibliometric
metrics to evaluatestudy
the on this relevant
application subject.methods
of MCDM This paper presents anByanalysis
in LCM-DS. of
using the
aScopus
quantitative approach
and Web baseddatabases
of Science on bibliometric metrics this
as references, to evaluate the application
study focuses of MCDM methods
on the identification in LCM-DS.
of the following By related
aspects using the
to
Scopus and Web
the research in thisoffield:
Science databases
distribution ofas references,
scientific this studymost
publications, focuses on the
relevant identification
sources, of the
interaction following
dynamics aspects
among theserelated to
sources,
the research in this field: distribution of scientific publications, most relevant sources, interaction dynamics
most prominent subjects in recent research (trend topics), and application of MCDM methods in LCM-DS. Regarding the latter, among these sources,
moststudy
this prominent subjects
identified in recent Hierarchy
the Analytic research (trend topics),
Process (AHP) andasapplication
the most used of MCDM
methodmethods in LCM-DS.
in the LCM-DS sinceRegarding the latter,
almost a quarter of
this study identified
the analyzed the Analytic
works applied Hierarchy
it in the original, Process
adapted,(AHP) as the
or hybrid most used
(combined method
with in the LCM-DS
other methods) format.since almost
Finally, a quarter
our results of
show
the analyzed works applied it in the original, adapted, or hybrid (combined with other methods) format.
that MCDM methods are commonly applied to LCM-DS problems in all life cycle stages, mainly in development, utilization, and Finally, our results show
that MCDM methods are commonly applied to LCM-DS problems in all life cycle stages, mainly in development, utilization, and
support.
support.
© 2022
© 2022 The
The Authors.
Authors. Published
Published by
by Elsevier
Elsevier B.V.
B.V.
© 2022
This is The Authors. Published by Elsevier B.V.
This is an
an open
open access
access article
article under
under the
the CC
CC BY-NC-ND
BY-NC-ND license
license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
(https://creativecommons.org/licenses/by-nc-nd/4.0)
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee
Peer-review under responsibility of the scientific committee ofof the 9th International
the 9th International Conference
Conference on Information Technology
on Information Technology and
and
Peer-review
Quantitative under
Quantitative responsibility of the scientific committee of the 9th International Conference on Information Technology and
Management.
Management
Quantitative Management.
Keywords: Operational Research; Systems Engineering; Multi-Criteria Decision-Making (MCDM); Life Cycle Management (LCM); Defense
Keywords: Operational
Systems; Defence Research;
Systems; MilitarySystems
Systems.Engineering; Multi-Criteria Decision-Making (MCDM); Life Cycle Management (LCM); Defense
Systems; Defence Systems; Military Systems.

∗ Corresponding author.
∗ Corresponding
E-mail address:author.
[email protected]
E-mail address: [email protected]

1877-0509 © 2022 The Authors. Published by Elsevier B.V.

1877-0509
1877-0509 © 2022
© 2022 The Authors.
Thearticle
Authors. Published
Published by
by Elsevier B.V.
Elsevier B.V.
This
This isisan
anopen
openaccess
access under
article the CC
under the BY-NC-ND
CC BY-NC-ND licenselicense
(http://creativecommons.org/licenses/by-nc-nd/4.0/)
(https://creativecommons.org/licenses/by-nc-nd/4.0)
This is an open
Peer-review underaccess article under the CC BY-NC-ND licenseof(http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review underresponsibility of the
responsibility scientific
of the committee
scientific committee theof9ththe
International Conference
9th International on Information
Conference TechnologyTechnology
on Information and Quantitative
and
Peer-review under
Management.
Quantitative responsibility of the scientific committee of the 9th International Conference on Information Technology and Quantitative
Management
Management.
10.1016/j.procs.2022.11.171
 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247 237
2 Girardi et al. / Procedia Computer Science 00 (2022) 000–000

1. Introduction

Complex environments, conflicting criteria, uncertainties, and inaccurate information are characteristics of many
decision problems present in the real world. In this context, the Multi-Criteria Decision-Making (MCDM) methodol-
ogy makes the decision-making processes more rational, efficient, and effective [1, 2].
MCDM is one of the fastest-growing Operational Research (OR) areas in the last two decades, and the decision
support methodologies that has been mostly used in operational management to support the quality of the decision-
making process [3]. It follows that MCDM methods have been widely applied in the military context, due to its
complexity, breadth, and sensitivity [4].
As for complexity, it is noteworthy that military systems are usually integrated by several customized components
and developed in a network of diverse actors, characteristics that are typical of Complex Product Systems (CoPS) [5].
Prior the development of military systems, decision makers are increasingly facing a growing number of technological
alternatives to achieve their planned defense capabilities, which requires the optimization of their acquisition and
selection processes [6].
This situation justifies the efforts of several Armed Forces around the world in establishing a life cycle model for
their defense product management processes [7], with a well-defined stage structure associated with decision-making
milestones and transition criteria [8]. Hence, MCDM methods become a highly relevant tool in the decision-making
process [9], making their systematic use in military systems crucial in reducing costs and optimizing the acquisition
of defense capabilities [10].
Although several MCDM applications in the management of military systems are available in scientific article
databases, so far, the authors are unaware of studies that perform a bibliometric analysis of these applications. There-
fore, to fill this gap in the specialized literature, this paper presents a bibliometric study on MCDM methods applied
to Life Cycle Management of Defense Systems (LCM-DS). In this sense, this research aims to answer the following
research questions:

• How are publications distributed (over time and by publication type)?

• What are the most relevant sources (publication vehicles, articles, references, authors, institutions, and coun-
tries)?
• What is the interaction dynamics among these sources?
• What are the most prominent subjects in recent research (trend topics)?
• What is the most applied MCDM method in LCM-DS?
• At what stage of the life cycle are MCDM methods used most?

With this analysis, this paper provides an overview of this topic with state-of-the-art scientific research, that indi-
cates the trend about MCDM applied to defense systems, helping managers and researchers inserted in the defense
field to select the most suitable methods and sources according to their particular contexts.
Based on the proposed research questions, the manuscript is structured as follows: Section 2 details the methodol-
ogy used; Section 3 presents and analyzes the results obtained in the bibliometric study; and Section 4 concludes with
the final considerations.

2. Methodology

To evaluate the application of MCDM methods in LCM-DS and answer the research questions, we follow a system-
atic literature review, as proposed by Thomé et al. (2016) [11], adopting a quantitative approach based on bibliometric
analysis [12]. We, therefore, apply the following five steps:

1. Planning: conception of the bibliometric research scope from the definition of the following search strategy:
Databases: Scopus and Web of Science (WoS), as suggested by Mongeon and Paul-Hus (2016) [13].
Search parameters in the Scopus database: TITLE-ABS-KEY((“multicriteria” OR “multi-criteria” OR
“multiple criteria” OR “multiple-criteria” OR “MCDA” OR “MCDM”) AND (“military” OR “defense” OR
238 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247
Girardi et al. / Procedia Computer Science 00 (2022) 000–000 3

“defence” OR “navy” OR “army” OR “air force”) AND (“system*” OR “life cycle” OR “life-cycle” OR “life-
cycle” OR “management”)).
Search parameters in the WoS database: TS = ((“multicriteria” OR “multi-criteria” OR “multiple cri-
teria” OR “multiple-criteria” OR “MCDA” OR “MCDM”) AND (“military” OR “defense” OR “defence” OR
“navy” OR “army” OR “air force”) AND (“system*” OR “life cycle” OR “life-cycle” OR “lifecycle” OR
“management”)).
2. Search: after defining the strategy, the search returned 476 documents from the Scopus database and 244 from
the WoS database, totaling 720 documents.
3. Refinement: the initial search result was filtered to exclude documents that were not directly related to the
application of MCDM methods in LCM-DS. Hence, 32 items from the initial search were removed from the
scope of the study, leaving 688 documents.
4. Synthesis: after refinement, the data from the 688 documents obtained from the Scopus and WoS databases
were merged using the software bibliometrix, an R-tool for comprehensive science mapping analysis [14]. The
synthesis removed 172 redundancies, leaving 516 documents.
5. Results & Analysis: after the data synthesis, results were generated with the support of the software biblioshiny
[15]. The results and their analysis are presented in Section 3.

The five steps of the methodology are summarized in Figure 1.

Fig. 1. Summary of the methodology used in the research.

3. Results & Analysis

Based on the research questions, the results, and their corresponding analysis, were segmented into five parts:
distribution of scientific publications, most relevant sources, interaction dynamics among sources, most prominent
subjects in recent research (trend topics), and application of MCDM methods in LCM-DS.
 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247 239
4 Girardi et al. / Procedia Computer Science 00 (2022) 000–000

3.1. Distribution of scientific publications

Concerning time-line distribution, the research obtained works from 1980 to nowadays. Disregarding the current
year, Figure 2 identifies an increase in the number of publications over the last few years. The dotted curve (quadratic
polynomial) shows a growing trend in the application of MCDM methods in the context of LCM-DS.

Fig. 2. Distribution of scientific publications over time.

As for the type of publication, Table 1 shows the distribution of the documents obtained in the research by typology
of scientific publications. It is possible to perceive a predominance of articles published in journals.

Table 1. Distribution of documents by publication type.

Publication type Number of documents Percentage

Journal article 285 55,23%

Conference article 187 36,24%
Review 33 6,40%
Book chapter 11 2,13%

3.2. Most relevant sources

The following aspects were analyzed to identify the most relevant sources: primary publication vehicles; most
influential works; high-impact authors, institutions, and countries applying MCDM methods in the context of LCM-
DS.
Regarding publication vehicles, Table 2 highlights the ten most relevant journals in this field of research based on
the h-index. This bibliometric metric combines productivity (number of articles) with impact (number of citations)
[16]. The relationship between the number of citations, the number of articles, and the number of years (NC/NA/NY)
was used as a tiebreaker. This relation between citations, publications, and time is widely used in computing the
impact factor of scientific publication vehicles [17].
240 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247
Girardi et al. / Procedia Computer Science 00 (2022) 000–000 5

Table 2. Most relevant publication vehicles.

Publication vehicle H-index Number of Number of Number of NC/NA/NY

citations (NC) articles (NA) years (NY)

Expert Systems with Applications 7 1120 7 15 (since 2007) 10,67

European Journal of Operational Research 7 897 9 34 (since 1988) 2,93
Fuzzy Sets and Systems 5 686 5 28 (since 1994) 4,90
Systems Engineering 5 123 5 20 (since 2002) 1,23
Lecture Notes in Computer Science 5 82 9 15 (since 2007) 0,61
Applied Soft Computing Journal 4 78 4 15 (since 2007) 1,30
System Engineering Theory and Practice 4 32 4 14 (since 2008) 0,57
Defence Science Journal 4 33 6 13 (since 2009) 0,42
Computers and Industrial Engineering 3 108 3 12 (since 2010) 3,00
International Journal of Production Research 3 124 3 14 (since 2008) 2,95

Concerning the most relevant studies, Table 3 brings together the ten articles in the research scope with the high-
est rates of citations (global citations) per year. Global citations mean the total number of documents indexed in a
bibliographic database that cited a paper (in the context of this research, Scopus and WoS databases) [18].

Table 3. Most relevant studies (global citations).

Reference Title Number of Number of NC/NY

citations (NC) years (NY)

Daǧdeviren et al. (2009) [19] Weapon selection using the AHP and TOPSIS methods 523 14 37,36
under fuzzy environment
Sennaroglu and Varlik Celebi A military airport location selection by AHP integrated 121 5 24,20
(2018) [20] PROMETHEE and VIKOR methods
Wang and Chang (2007) [21] Application of TOPSIS in evaluating initial training aircraft 361 16 22,56
under a fuzzy environment
Sipahi and Timor (2010) [22] The analytic hierarchy process and analytic network process: 269 13 20,69
an overview of applications
Yu and Lai (2011) [23] A distance-based group decision-making methodology for 230 12 19,17
multi-person multi-criteria emergency decision support
Cheng and Lin (2002) [24] Evaluating the best main battle tank using fuzzy decision 353 21 16,81
theory with linguistic criteria evaluation
Cheng (1997) [25] Evaluating naval tactical missile systems by fuzzy AHP 372 26 14,31
based on the grade value of membership function
Gigović et al. (2016) [26] The combination of expert judgment and GIS-MAIRCA 77 7 11,00
analysis for the selection of sites for ammunition depots
Moreira et al. (2021) [27] PROMETHEE-SAPEVO-M1 a hybrid approach based on 21 2 10,50
ordinal and cardinal inputs: Multi-criteria evaluation of
helicopters to support Brazilian navy operations
Bisdikian et al. (2013) [28] On the quality and value of information in sensor networks 99 10 9,90

Still, regarding relevant works, Table 4 presents the ten cited references in the scope of this research with the highest
citation rates (local citations) per year. Local citations mean the total appearances of a document in the reference lists
of articles contained in a research collection (in the context of this research, the collection of 516 articles) [18].
 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247 241
6 Girardi et al. / Procedia Computer Science 00 (2022) 000–000

Table 4. Most relevant references (local citations).

Reference Title Number of Number of NC/NY

citations (NC) years (NY)

Saaty (2008) [29] Decision making with the analytic hierarchy process 11 14 0,79
Saaty (1980) [30] The analytic hierarchy process 28 42 0,67
Huang et al. (2011) [31] Multi-criteria decision analysis in environmental sciences: 7 11 0,64
Ten years of applications and trends
Saaty (1990) [32] How to make a decision: the analytic hierarchy process 12 32 0,38
Vaidya and Kumar (2006) [33] Analytic hierarchy process: An overview of applications 6 16 0,38
Velasquez and Hester (2013) [34] An analysis of multi-criteria decision-making methods 3 9 0,33
Belton and Stewart (2002) [35] Multiple criteria decision analysis: an integrated approach 6 20 0,30
Karvetski et al. (2011) [36] Integration of decision analysis and scenario planning for 3 11 0,27
coastal engineering and climate change
Cheng and Mon (1994) [37] Evaluating weapon system by analytical hierarchy process 7 28 0,25
based on fuzzy scales
Montibeller et al. (2006) [38] Combining scenario planning and multi-criteria decision 4 16 0,25
analysis in practice

Table 5 lists the ten most prominent authors. Following a similar approach to that used with publication vehicles,
the analysis was based on the h-index and the relationship between the number of citations, articles, and years of
activity (NC/NA/NY) was used as a tiebreaker. Based on the publication period column, one can identify that only
J. Lambert, I. Linkov, M. dos Santos, C. Gomes, Y. Chen, and J. Sánchez-Lozano remain active in the research field
(considering authors with publications in the last five years).

Table 5. Most relevant authors.

Author Country H-index Number of Number of Publication Number of NC/NA/NY
citations (NC) articles (NA) period years (NY)

Cheng, C. China 8 1433 9 1994-2007 14 11,37

Lambert, J. USA 6 202 10 2010-2017 8 2,53
Linkov, I. USA 6 152 14 2007-2017 11 0,99
dos Santos, M. Brazil 5 82 8 2020-2021 2 5,13
Gomes, C. Brazil 5 82 9 2020-2021 2 4,56
Chen, Y. China 5 162 7 2009-2018 10 2,31
Mon, D. China 4 533 4 1994-1995 2 66,63
Stakhiv, E. USA 4 218 4 1992-1992 1 54,50
Sánchez-Lozano, J. Spain 4 45 4 2017-2020 4 2,81
Rose-Pehrsson, S. USA 4 123 6 2000-2010 11 1,86

Table 6 scores the ten most influential research centers. Following a similar approach to that used with publication
vehicles and authors, the analysis was based on the h-index and the relationship between the number of citations,
the number of articles, and the number of years of activity (NC/NA/NY) was used as a tiebreaker. Based on the
publication period column, one can identify that only King Saud University does not remain active in the research
field (considering institutions with publications in the last five years).
242 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247
Girardi et al. / Procedia Computer Science 00 (2022) 000–000 7

Table 6. Most relevant institutions.

Institution Country H-index Number of Number of Publication Number of NC/NA/NY
citations (NC) articles (NA) period years (NY)

National University of Defence China 6 124 10 2008-2019 12 1,03

Technology
University of Virginia USA 6 189 15 1980-2021 42 0,30
Fluminense Federal University Brazil 5 65 9 2020-2021 2 3,61
University of Defence Serbia 5 161 9 2000-2021 22 0,81
Engineer Research and Development USA 5 90 10 1988-2017 30 0,30
Center
Military Institute of Engineering Brazil 4 61 7 2012-2021 10 0,87
Tsinghua University China 4 48 5 2008-2019 12 0,80
Gazi University Turkey 3 552 6 2002-2020 19 4,84
King Saud University Saudi Arabia 3 30 8 2010-2016 7 0,54
Istanbul University Turkey 2 187 6 2010-2021 12 2,60

As for the most influential countries, Figure 3 illustrates the role of the USA, China, Turkey, Canada, Brazil, Italy,
UK, Australia, Iran, and Serbia as the most productive nations in the research area. The result reflects the trend pointed
out by the analysis of the relevance of authors and institutions (Tables 5 and 6, respectively), where it was already
noticed the highlight of the first two countries on the list (USA and China).

Fig. 3. Relevance of countries in terms of publications.

3.3. Interaction dynamics among sources

Considering the authors as the primary agents in the interaction dynamics among the sources, Figure 4 presents the
main research clusters related to the application of MCDM methods in the context of LCM-DS.
 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247 243
8 Girardi et al. / Procedia Computer Science 00 (2022) 000–000

Fig. 4. Main research clusters.

Figure 4 highlights twelve research clusters: four in the USA, three in China, one in Turkey, one in Canada, one
in Brazil, one in Australia, and one in Saudi Arabia. The green rectangles show the publication period of the clusters
which remain active in the research field (considering groups with publications in the last five years). The follow-
ing researchers lead these six active research groups: James Lambert (University of Virginia, Charlottesville, USA);
Yingwu Chen (National University of Defense Technology, Changsha, China); Lei Chen (Tsinghua University, Bei-
jing, China); Cevriye Gencer (Gazi University, Ankara, Turkey); Carlos Francisco Gomes (Fluminense Federal Uni-
versity, Niterói, Brazil), Minh Nguyen (Defense Science and Technology Group, Edinburgh, Australia), and Abdullah
Alghamdi (King Saud University, Riyadh, Saudi Arabia).
It is noteworthy that this result reflects the trend pointed out by the analysis of the relevance of the sources presented
in Section 3.2, considering that the prominent clusters bring together authors, institutions, and/or countries of high
impact in research related to the application of MCDM methods in the context of LCM-DS.

3.4. Most prominent subjects in recent research (trend topics)

After a global analysis of the frequency of the words presented in the title, abstract, and keywords of the papers
in the scope of the study, Figure 5 shows the most prominent terms in recent research (trend topics). A growth trend
related to subjects associated with risk management and computing can be observed (emphasizing on network security,
information systems, and artificial intelligence).

Fig. 5. Trend topics.

244 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247
Girardi et al. / Procedia Computer Science 00 (2022) 000–000 9

3.5. Application of MCDM methods in LCM-DS

After analyzing MCDM applied to LCM-DS, the following twenty-two methods were found in the articles col-
lection: AHP, ANP, Borda, Condorcet, DEMATEL, ELECTRE, FITradeoff, FUCOM, Fuzzy Sets, MAUT, MAVT,
PAMSSEM, PROMETHEE, PrOPPAGA, RIM, SAPEVO, SAW, SMART, THOR, TODIM, TOPSIS, and VIKOR.
Table 7 presents a ranking with the percentage of incidence of these methods in the papers contained in the re-
search scope. The normalized incidence column was inserted in the table because some articles generically applied
the MCDM methodology. The normalization considered the percentage calculation only within the set of papers with
explicitly defined MCDM methods.
The survey identified the AHP (Analytic Hierarchy Process) method as the most used, considering that almost a
quarter (23,26%) of the analyzed studies applied it in the original, adapted, or hybrid (combined with other methods)
format. Thomas Saaty proposed the AHP method in the ’70s. The methodology seeks to structure a decision problem
simply and rationally through a hierarchy of three levels: goal, criteria, and alternatives. In this context, the available
alternatives (third level) are analyzed in light of a set of criteria (second level) to achieve the proposed goal (first level)
[30].

Table 7. MCDM methods applied to LCM-DS.

MCDM Number of Incidence Normalized MCDM Number of Incidence Normalized

method applications (%) incidence (%) method applications (%) incidence (%)

AHP 120 23,26 39,34 THOR 3 0,58 0,98

TOPSIS 50 9,69 16,39 FITradeoff 2 0,39 0,66
Fuzzy Sets 42 8,14 13,77 RIM 2 0,39 0,66
ANP 25 4,84 8,20 SMART 2 0,39 0,66
ELECTRE 16 3,10 5,25 Borda 1 0,19 0,33
PROMETHEE 10 1,94 3,28 Condorcet 1 0,19 0,33
VIKOR 9 1,74 2,95 MAVT 1 0,19 0,33
DEMATEL 7 1,36 2,30 PAMSSEM 1 0,19 0,33
SAPEVO 4 0,78 1,31 PrOPPAGA 1 0,19 0,33
FUCOM 3 0,58 0,98 SAW 1 0,19 0,33
MAUT 3 0,58 0,98 TODIM 1 0,19 0,33

Still, in the context of LCM-DS, Figure 6 shows how the application of MCDM methods is allocated throughout
the life cycle stages of defense systems. This graphic clearly shows that MCDM methods are generally applied in all
life cycle stages, with greater emphasis on the phases of development, utilization, and support.

Fig. 6. Application of MCDM methods through the life cycle stages of defense systems.
 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247 245
10 Girardi et al. / Procedia Computer Science 00 (2022) 000–000

It is worth emphasizing that the study considered the aggregation of the ISO/IEC/IEEE 15288 (2015) [39] and
Blanchard and Fabrycky (2011) [40] models. The ISO 2015 standard proposes six stages for the life cycle of a system:
concept, development, production, utilization, support, and retirement [39]. Blanchard and Fabrycky (2011) establish
a step before the concept stage of a system, the pre-acquisition phase. At this point, the identification and prioritization
of high-level needs of an organization take place, together with the analysis of the technical and economic feasibility of
these needs [40]. In the military context, the pre-acquisition phase is usually supported by Capability-Based Planning
[41], an approach used by several Armed Forces, such as the USA [42], China [43], United Kingdom [44], and Brazil
[45].
The development stage is a period full of complex decisions, such as the physical specification of the system (sub-
systems, assemblies, and components), the definition of strategies for the solution implementation, and the discussion
of approaches for the critical technologies maturation [39, 40, 46]. This perception justifies the high incidence (24%)
of the application of MCDM methods in the development stage of the defense systems life cycle.
It is also possible to identify aspects that justify the high use of MCDM methods in the utilization and support stages
(21% and 24%, respectively). These two steps occur in parallel throughout the life cycle of a defense system [39] and
also involve critical decisions, such as evaluating the best way to employ the system or to combine it with other
systems and analyzing means of support to ensure reliability, availability, and maintainability to a systemic capability
[8]. In addition, it is noteworthy that the utilization and support phases consume the most significant amount of time
and resources throughout the life cycle of a system [47].
Finally, it is worth highlighting that the fourth place attributed to the retirement stage (13%) shows how aspects
related to the environment have gained relevance in decision-making processes. In line with the increase in regulations
and awareness of environmental issues, the life cycle of any system should dedicate special attention to the retirement
phase, evaluating how the system’s disposal will happen [8].

4. Conclusion

This paper presented a quantitative approach study based on bibliometric metrics to evaluate the application of
MCDM methods in LCM-DS. Therefore, concerning the Scopus and Web of Science databases, this research was
structured in five phases - planning, search, refinement, synthesis, and results/analysis - to answer the research ques-
tions related to the following aspects: distribution of scientific publications, most relevant sources, interaction dy-
namics among these sources, most prominent subjects in recent research (trend topics), and application of MCDM
methods in LCM-DS.
After the presentation and analysis of the results, we suggest that the bibliometric study was able to satisfactorily
answer the proposed research questions.
As a limitation of the proposed method, it can be pointed out that the study is restricted to academic literature,
specifically the Scopus and Web of Science databases. Other academic databases and patent literature may also be
considered to improve results.
Finally, for future studies, two suggestions are indicated:

• Carry out a broader study, contemplating other academic bases and/or patent literature.
• As the AHP stood out in the scope of the research, a bibliometric study focused on this method is suggested to
identify its applications, adaptations, and combinations with other methods in the context of LCM-DS.

References

[1] Munier, N., Hontoria, E., and Jiménez-Sáez, F. (2019). Strategic Approach in Multi-Criteria Decision Making: A Practical Guide for Complex
Scenarios (1st ed.). Springer.
[2] Pereira, F. de C., Costa, H. G., and Pereira, V. (2017). Patent filings versus articles published: A review of the literature in the context of
Multicriteria Decision Aid. In World Patent Information (Vol. 50). https://doi.org/10.1016/j.wpi.2017.07.003
[3] Aires, R. F. de F., and Ferreira, L. (2018). The rank reversal problem in multi-criteria decision making: A literature review. Pesquisa Opera-
cional, 38(2). https://doi.org/10.1590/0101-7438.2018.038.02.0331
[4] Hamurcu, M., and Eren, T. (2020). Selection of Unmanned Aerial Vehicles by Using Multicriteria Decision-Making for Defence. Journal of
Mathematics, 2020. https://doi.org/10.1155/2020/4308756
 246 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247
Girardi et al. / Procedia Computer Science 00 (2022) 000–000 11

[5] Hobday, M. (1998). Product complexity, innovation and industrial organisation. Research Policy, 26(6), 689–710.
https://doi.org/10.1016/S0048-7333(97)00044-9
[6] Galdino, J. F., and Schons, D. L. (2022). Maquiavel e a importância do poder militar nacional. Coleção Meira Mattos, 16(56), 369–384.
https://doi.org/10.52781/cmm.a077
[7] Lin, S., Hou, Y., Ni, L., Zhang, F., and Wang, Z. (2022). Research on Equipment Lifecycle Management Based on Consor-
tium Blockchain. 2022 14th International Conference on Measuring Technology and Mechatronics Automation, ICMTMA 2022.
https://doi.org/10.1109/ICMTMA54903.2022.00113
[8] Walden, D., Roedler, G., Forsberg, K., Hamelin, R., and Shortell, T. (2015). INCOSE Systems Engineering Handbook, 4th edition.
[9] Sánchez-Lozano, J. M., and Rodrı́guez, O. N. (2020). Application of Fuzzy Reference Ideal Method (FRIM) to the military advanced training
aircraft selection. Applied Soft Computing Journal, 88. https://doi.org/10.1016/j.asoc.2020.106061
[10] Van Hoan, P., and Ha, Y. (2020). ARAS-fucom approach for VPAF fighter aircraft selection. Decision Science Letters, 10(1).
https://doi.org/10.5267/j.dsl.2020.10.004
[11] Thomé, A. M. T., Scavarda, L. F., and Scavarda, A. J. (2016). Conducting systematic literature review in operations management. In Production
Planning and Control (Vol. 27, Issue 5). https://doi.org/10.1080/09537287.2015.1129464
[12] Cauchick-Miguel, P. A., Fleury, A., Mello, C. H. P., Nakano, D. N., Lima, E. P. de, Turrioni, J. B., Ho, L. L., Morabito, R., Costa, S. E. G. da,
Martins, R. A., Sousa, R., and Pureza, V. (2018). Metodologia de Pesquisa em Engenharia de Produção e Gestão de Operações (3rd ed.). LTC.
[13] Mongeon, P., and Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: a comparative analysis. Scientometrics, 106(1).
https://doi.org/10.1007/s11192-015-1765-5
[14] Aria, M., and Cuccurullo, C. (2017). bibliometrix: An R-tool for comprehensive science mapping analysis. Journal of Informetrics, 11(4).
https://doi.org/10.1016/j.joi.2017.08.007
[15] Bibliometrix.org. (2022). Biblioshiny: the shiny app for bibliometrix. https://www.bibliometrix.org/home/index.php/layout/biblioshiny
[16] Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., and Herrera, F. (2009). h-Index: A review focused in its variants, computation and standard-
ization for different scientific fields. Journal of Informetrics, 3(4). https://doi.org/10.1016/j.joi.2009.04.001
[17] Guo, X., Li, X., and Yu, Y. (2021). Publication delay adjusted impact factor: The effect of publication delay of articles on journal impact factor.
Journal of Informetrics, 15(1), 101100. https://doi.org/10.1016/J.JOI.2020.101100
[18] Yang, J., and Liu, Z. (2022). The effect of citation behaviour on knowledge diffusion and intellectual structure. Journal of Informetrics, 16(1).
https://doi.org/10.1016/j.joi.2021.101225
[19] Daǧdeviren, M., Yavuz, S., and Kilinç, N. (2009). Weapon selection using the AHP and TOPSIS methods under fuzzy environment. Expert
Systems with Applications, 36(4). https://doi.org/10.1016/j.eswa.2008.10.016
[20] Sennaroglu, B., and Varlik Celebi, G. (2018). A military airport location selection by AHP integrated PROMETHEE and VIKOR methods.
Transportation Research Part D: Transport and Environment, 59. https://doi.org/10.1016/j.trd.2017.12.022
[21] Wang, T. C., and Chang, T. H. (2007). Application of TOPSIS in evaluating initial training aircraft under a fuzzy environment. Expert Systems
with Applications, 33(4). https://doi.org/10.1016/j.eswa.2006.07.003
[22] Sipahi, S., and Timor, M. (2010). The analytic hierarchy process and analytic network process: An overview of applications. Management
Decision, 48(5). https://doi.org/10.1108/00251741011043920
[23] Yu, L., and Lai, K. K. (2011). A distance-based group decision-making methodology for multi-person multi-criteria emergency decision
support. Decision Support Systems, 51(2). https://doi.org/10.1016/j.dss.2010.11.024
[24] Cheng, C. H., and Lin, Y. (2002). Evaluating the best main battle tank using fuzzy decision theory with linguistic criteria evaluation. European
Journal of Operational Research, 142(1). https://doi.org/10.1016/S0377-2217(01)00280-6
[25] Cheng, C. H. (1997). Evaluating naval tactical missile systems by fuzzy AHP based on the grade value of membership function. European
Journal of Operational Research, 96(2). https://doi.org/10.1016/S0377-2217(96)00026-4
[26] Gigović, L., Pamuĉar, D., Bajić, Z., and Milićević, M. (2016). The combination of expert judgment and GIS-MAIRCA analysis for the selection
of sites for ammunition depots. Sustainability (Switzerland), 8(4). https://doi.org/10.3390/su8040372
[27] Moreira, M. Â. L., De Costa, I. P. A., Pereira, M. T., Dos Santos, M., Gomes, C. F. S., and Muradas, F. M. (2021). Promethee-sapevo-m1 a
hybrid approach based on ordinal and cardinal inputs: Multi-criteria evaluation of helicopters to support Brazilian navy operations. Algorithms,
14(5). https://doi.org/10.3390/a14050140
[28] Bisdikian, C., Kaplan, L. M., and Srivastava, M. B. (2013). On the quality and value of information in sensor networks. ACM Transactions on
Sensor Networks, 9(4). https://doi.org/10.1145/2489253.2489265
[29] Saaty, T. L. (2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences, 1(1).
[30] Saaty, T. L. (1980). The Analytic Hierarchy Process. McGraw-Hill.
[31] Huang, I. B., Keisler, J., and Linkov, I. (2011). Multi-criteria decision analysis in environmental sciences: Ten years of applications and trends.
Science of The Total Environment, 409(19), 3578–3594. https://doi.org/10.1016/J.SCITOTENV.2011.06.022
[32] Saaty, T. L. (1990). How to make a decision: The analytic hierarchy process. European Journal of Operational Research, 48(1).
https://doi.org/10.1016/0377-2217(90)90057-I
[33] Vaidya, O. S., and Kumar, S. (2006). Analytic hierarchy process: An overview of applications. European Journal of Operational Research,
169(1). https://doi.org/10.1016/j.ejor.2004.04.028
[34] Velasquez, M., and Hester, P. (2013). An analysis of multi-criteria decision making methods. International Journal of Operations Research,
10(2).
[35] Belton, V., and Stewart, T. J. (2002). Multiple Criteria Decision Analysis. In Multiple Criteria Decision Analysis. https://doi.org/10.1007/978-
1-4615-1495-4
[36] Karvetski, C. W., Lambert, J. H., Keisler, J. M., and Linkov, I. (2011). Integration of decision analysis and scenario planning for
coastal engineering and climate change. IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans, 41(1).
 Romullo Girardi et al. / Procedia Computer Science 214 (2022) 236–247 247
12 Girardi et al. / Procedia Computer Science 00 (2022) 000–000

https://doi.org/10.1109/TSMCA.2010.2055154
[37] Cheng, C. H., and Mon, D. L. (1994). Evaluating weapon system by Analytical Hierarchy Process based on fuzzy scales. Fuzzy Sets and
Systems, 63(1). https://doi.org/10.1016/0165-0114(94)90140-6
[38] Montibeller, G., Gummer, H., and Tumidei, D. (2006). Combining scenario planning and multi-criteria decision analysis in practice. Journal
of Multi-Criteria Decision Analysis, 14(1–3). https://doi.org/10.1002/mcda.403
[39] ISO. (2015). ISO/IEC/IEEE 15288 - Systems and software engineering - System life cycle processes.
[40] Blanchard, B., and Fabrycky, W. (2011). Systems Engineering and Analysis (5th ed.). Prentice Hall.
[41] Thaba, J. M. (2020). Technology support for military capability based acquisition. 26th International Association for Management of Technol-
ogy Conference, IAMOT 2017.
[42] United States of America. (2018). CJCSI 5123.01H - Charter of the Joint Requirements Oversight Council (JROC) and Implementation of the
Joint Capabilities Integration And Development System (JCIDS). U.S. Department of Defense.
[43] Barton, J. (2021). China’s PLA Modernization through the DOTMLPF-P Lens. TRADOC Mad Scientist Laboratory.
https://madsciblog.tradoc.army.mil/330-chinas-pla-modernization-through-the-dotmlpf-p-lens/
[44] United Kingdom. (2019). Knowledge in Defence (KiD). UK Ministry of Defence.
[45] Brasil. (2019). EB20-MF-10.102 - Manual de Fundamentos da Doutrina Militar Terrestre. Exército Brasileiro.
[46] ISO. (2010). ISO/IEC TR 24748-1 - Systems and software engineering - Life cycle management.
[47] Adcock, R., Wells, B., Jackson, S., Singer, J., and Hybertson, D. (2021). Guide to the Systems Engineering Body of Knowledge (SEBoK),
version 2.5 (Issue October).