See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/343535421

Multicriteria Model for Evaluation of Outsourcing Services by Logistics

Operators

Chapter · August 2020

DOI: 10.1007/978-3-030-51965-0_49

CITATIONS READS

4 159

3 authors, including:

Marcelo Crispim Salazar Plácido Rogério Pinheiro

Universidade de Fortaleza Universidade de Fortaleza
2 PUBLICATIONS 6 CITATIONS 256 PUBLICATIONS 3,320 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Plácido Rogério Pinheiro on 21 August 2020.

The user has requested enhancement of the downloaded file.

 Multicriteria Model for Evaluation of Outsourcing
Services By Logistics Operators

Marcelo Crispim Salazar1, Placido Rogério Pinheiro2, Isaura Cardoso Cavalcante de Castro 3
1
Salazar Advisory and Management Consulting, Imbituba, Brazil
[email protected]
2
University of Fortaleza, Fortaleza, Brazil
[email protected]
3
IVIA, Fortaleza, Brazil
[email protected]

Abstract. Logistics has a critical role in the competitiveness of enterprises,

in order that their costs have significant participation in the pricing of
products. In this context, outsourcing of services becomes a significant ally
of organizations increasingly, with the execution of the activities under the
responsibility of the Logistics Operator, a company specialized in providing
these services. However, in the quest for better operating performance and
financial, to align the expectations of the Logistics Operator Shippers and
their customers, it is necessary. Therefore, it is necessary to develop a study
proposing a multi-criteria model for evaluation of outsourcing services
provided by Logistics Operators. As the first phase of the research, the aim
of this study is the identification of criteria and sub-criteria considered
essential for analysis. With the participation of specialists and application of
methodologies Brainstorming and Web-Delphi, it was possible to perform
construction value tree needed to structure the next phase of the multi-criteria
analysis decision support model.

Keywords: Logistic Operator; Multi-Criteria Analysis; Web-Delphi.

1 INTRODUCTION

The generation of better economic and financial results of an enterprise is a factor

that guarantees its existence, and the profit the result of reasoned management to apply
the search for improved performance in all areas of the company. [22] points out that,
for the company to obtain business success, you need to reduce costs and at the same
time, be able to offer excellent performance, offering best relationship between cost and
benefit to their customers.
Accordingly, the logistics become essential shows the composition of the costs of
the products, causing the level of effectiveness is evident in handling, storing materials,
and inventory information from respective streams.
Together with the coordinated and integrated management, the supply chain plays a
crucial role in the strategy of companies [6]. However, most of them have difficulties
in finding internal solutions to their logistical problems, and outsourcing known as
outsourcing services Logistics Operators, have become increasingly appropriate. Thus,
shippers’ customers, who are the owners of the cargo, can direct their forces to their
core business.
Efficient logistics is not linked only to trucks and storage of cargo, but also the
management of the best mode of operation "end to end" Supplies, using the best possible
practices. Due to the high cost of logistics in the composition of product prices,
companies need improvements in this area to win higher profits, enabling gain a
competitive advantage on competitors. An LO acting effectively can meet the needs of
the business as a service provider, offering transportation, warehousing and distribution,
making it essential in product pricing of the composition to the final consumer.
According to the Logistics Center, Supply Chain and Infrastructure ABOL -
Brazilian Association of Logistics Operators, one LO should systematize information,
develop storage strategies, internal and transport drives, using its assets or third parties
to perform activities in the chain of supply and distribution companies, generating better
conditions in the processes and reduction in costs. [15]
Furthermore to [15], the services provided by LO combine physical activities,
warehousing and transportation management merchants and activities. As Shippers
Customers face greater complexity in their operations amounts to the presence of the
Logistic Operators (LOs), offering outsourcing services in logistics, integrating the
supply chain, from incoming raw material to the exit finished products in industries.
The Logistics Operators, LOs or 3PLs - Third Party Logistics as they are known
worldwide, need to improve the management of the services offered. Therefore, it is
necessary to understand how their customers, the shippers, who hire logistics services,
evaluate their operators, enabling scale the current performance and identify
opportunities for improvement, creating conditions for operators observe where they
can become more competitive, based on market expectations.
This paper presents the stages of structuring the multi-criteria model for decision-
making, where through brainstorming methodologies reconciled to Web-Delphi
research, it can achieve the goal of the study, identifying the main criteria and sub-
performing. Moreover, the construction value tree needed for structuring a multi-criteria
analysis model to support the decision to evaluate the outsourcing of services provided
by Logistics Operators, from the perspective of customers Shippers.

2 THEORETICAL REFERENCE

Logistics is continually evolving since the second world war, where it already had
vital importance on the strategies in the battles. Currently, it is critical to the success of
organizations because of its direct influence on the degree of customer satisfaction.
Logistics is of utmost importance for organizations and, before that, there is the figure
of the Logistics Operator, specialized in the provision of logistics services within steps
of the supply chain. To [18], the Logistics Operator (in English known as third-party
logistics, 3PL or logistics providers) is a juridical company that provides outsourced
logistics activities services, performing functions which may include all or part of the
logistics process of a company that owns the product, which is known as shipper client.
Talks [31] that outsourcing is a management process by which some activities are
passed on to third parties with which it establishes a partnership, being the only
company focused on tasks related mainly to the business in which it operates.
According to [26], outsourcing is a management technique which companies have
resorted in order to increase their productivity and competitiveness. It is a business
strategy in order to focus efforts and intelligence in the core business, leaving the other
activities in charge of partners specialized in specific functions, producing better results
and bringing real profits to both parties. Stress [33] the practice of outsourcing services
in logistics activities allows companies that hire the LO to save on capital investment,
reducing financial risks. Therefore, they need not have machinery, equipment, vehicles
and distribution centres, channelling their resources to their end activities. According to
[33] despite the widely recognized importance of logistics outsourcing in many cases,
it does not result in satisfactory performance. In this context, it is clear that the
evaluation of outsourcing services provided by Logistics Operators, proposed in this
study is relevant in view, which can direct the use of best management practices, aimed
at the continuous search for perpetuity companies, Operators Shippers and logistics
customers.

2.1 METHODOLOGY MULTICRITERIA

2.1.1 Support multi-criteria decision
When it comes to advanced methods of decision support should be seen the
foundations of decision theory, that support the MCDA methodology (multiple criteria
decision analysis). Making decisions involving ordinary activities in daily life, become
more complicated because it is necessary to solve problems, taking into account the
various points of views, with specific forms of evaluation that can generate; as a result,
a set of alternative solutions may consider multiple criteria conflicting with each other
[13] and [21].
In this line of reasoning [1, 4] confirms that a problem of the multi-criteria decision
before two actions alternatives to choose from, the choice is driven by the desire to meet
multiple objectives, often conflicting.
According to [3], the decision making in private and public organizations, depend on
human activity where the value judgment of the managers play a crucial role. Thus, the
integration of information technology to human decisions through the use of techniques
and decision support tools, it is appropriate to answer the central question of how
numerically represent this value judgment.
Support multi-criteria decision is a dynamic area of knowledge and research, aimed
at helping policymakers and negotiators, assisting them in structuring problems, making
it possible to expand the argument and expand the capacity of learning and
understanding [11].
2.1.2 Multi-criteria decision support concepts
The methodology support multi-criteria decision is aimed to assess complex
situations, from a set of quantitative and qualitative indicators different, including
financial indicators, physical, input, process, products and results that can be used to
support the decision as a set of analytical techniques [32]. In [14], advanced decision
methods to evaluate multiple possible choices on the various decision criteria to select,
sort, classify and describe a set of alternatives for more complex decision making.
Ratified [13] the advanced methods for decision support tools are used to select, sort,
classify and describe, in detail, the alternatives that will base the decision making [13],
or may also analyze and evaluate life situations that as a whole, they are the result of
past decisions. Furthermore, this research will be evaluated by outsourcing services
provided by Logistics Operators.
To [13], the advanced methods of decision support can be used in different areas of
knowledge because of the ability to work with quantitative and qualitative aspects.
2.1.3 Stages of the decision support process
According to [4], there are three main stages of the decision support process: 1)
Structure: deals with the formulation of the problem and identification of goals and
make a qualitative analysis of the options and their impacts; 2) Assessment: is designed
for one model decision to be made, the amounts of the criteria are defined according to
the information gathered in the design phase; 3) Recommendation: apply the model in
the evaluation phase using the weighting of the criteria suggested for decision support,
they are subsequently carried out sensitivity analyzes and robustness. The study of
reference for later problems is necessary to structure the multi-criteria model to be used.
According to [12], there are four types of problems:
1. Description of the Problem (P.δ) - a description is made of alternatives;
2. Problematic classification (P.β) - is carried out a classification of alternatives;
3. Sorting (P.γ) - is generated ordering of the alternatives;
4. Problematic of choice (P.α) - is chosen as the best alternative or alternatives best.
Moreover, in the [28] confirms that the design phase is significant in the construction
of the multi-criteria model of decision support, according to the author at this stage we
define the problem, the actors, identification and implementation of the elements,
whether they are less relevant or more impactful in the evaluation process.

3 RESEARCH METHODOLOGY

Concerning objectives, research is presented as descriptive and exploratory as part

because, according to [10], the descriptive involves using standard techniques of data
collection: questionnaires and systematic observation. Thus, by lifting, describes the
characteristics of a particular phenomenon or establishing relationships between
variables. Exploratory research involves literature review and interviews with people
who have practical experience with the researched problem. As for the procedures,
research has bibliographical. Nature, the research can be classified as applied, as will
be developed directly with the logistics operators who provide outsourcing services to
customers Shippers, in order to resolve the problem of evaluation. The brainstorming
technique for lifting qualitative information is a tool associated with creativity and used
in the planning phase of research, which attempts to answer a question problem.
In order to provide further consistency to the data collection phase, after the
brainstorming technique of application of four experts, will use the Delphi method as a
research technique, through the opinions of six experts. So, it can get more consistency
and consistency of information, creating greater credibility to the survey.
The Delphi technique began to be disseminated in the 50s, the Mathematical Olaf
Helmer and Norman Dalker, researchers at the US Rand Corporation, Santa Monica,
California, to help the US Air Force during the Cold War.
According to [17], the Delphi method is defined as "a method for structuring a
process of collective communication so that it is effective in allowing a group of
individuals, as a whole, to deal with a complex problem."
Besides, [7] highlight that Delphi method aims to obtain the most reliable consensus
of a group of experts, by applying a series of intensive questionnaires interspersed with
controlled feedback of views. So, from this methodological development, Delphi is
considered a systematic technique to collect the opinions, scientifically valid, expert, on
a particular subject. [8]
The method aims to facilitate and improve decision making made by a group of
experts, without face to face interaction [25]. The same author points out that this
method is a set of questionnaires that are answered sequentially and individually per
participant. The information is summarized on the responses of the group the previous
questionnaires, establishing a kind of dialogue between the participants, which creates
conditions to go building a collective response gradually. On the other hand, this is done
by establishing three necessary conditions: the anonymity of the respondents, the
statistical representation of the distribution of results and feedback of the group
responses to subsequent rounds re [20]
According to [23], “The goal was to obtain the consensus of expert opinion as
objectively as possible, through a series of questions."
Thus, experts, throughout the different rounds of questionnaires, will present their
opinions, will compare with the group, go arguing and defending their positions, and at
the same time are open to reconsider and change them before the arguments of other
colleagues or the general trend of the group [19].
Also [19] emphasize another potential Delphi: the identification of knowledge, skills
or abilities of a particular professional.
The implementation process of Delphi is accomplished in a few steps, as shown in
Fig. 1:

Figure 1 - A general scheme for implementing the method Delphi three rounds
Source: [19]

According to the illustration, the process of implementation of the Delphi method,

after the choice of experts, takes place through the construction and implementation of
the first questionnaire. Following is made a quantitative and qualitative analysis of the
responses. Afterwards, send the second questionnaire, followed by a new reflection and
having no closer to consensus, continues the process.
With the advent of the Internet came the so-called Web-Delphi or e-Delphi, where,
through technological platforms, it can send the questionnaires and streamline the entire
research process, allowing participants to respond at their own pace, without
geographical barriers [34].

3.1 DATA PHASE

This research was structured in three data collection tools: bibliographic research,
interviews and questionnaires. According to [16], the literature is intended to put the
researcher in touch with everything that has been produced on the subject.

3.2 STRUCTURE OF PHASE

The design phase began with interviews applied to experts, as ratifies [24]. The
purpose of the search through the interview aims to explore the experiences of the actors
engaged in the organizational context.
Step 1 - Interviews with experts -Interviews were conducted with four professional
specialists working in coordination positions, management and direction of companies
providing logistics services. Using the brainstorming technique participants were free
to treat the logistics issue. First, we evaluated the concepts related to the Logistics
Operator selection treated in the literature and used by professionals. They were then
addressed some points of facilities and difficulties in order to categorize the criteria and
sub-criteria are considered more relevant and appropriate in the process of evaluation
of outsourcing services provided by Logistics Operators. In this first stage, some
opinions were different, given the complexity and range of primary and peripheral
services involved in this type of service. Thus, the classification of developed criteria
and sub-criteria have been validated, classified, and the most suitable ordered by other
experts through the Web-Delphi methodology, according to the next step.
Step 2 - Questionnaire - Web-Delphi (First round) - To ensure greater consistency
to the categorization of the most important criteria and sub-criteria, we applied a
questionnaire directed to a group of six specialists working both in logistics operators
like us Customers Shippers, who drew up ranking criteria and sub-criteria.
Step 3 - Questionnaire - Web-Delphi (Second round) - In the second round of Web-
Delphi after the evaluation and analysis of the first round answers, according to the
opinion of each participant, aiming the nearest search for consensus, provided an
opportunity is to reclassification and ordering individually, in compliance with the
compiled responses group, which allowed the definition of the most important criteria
and sub-criteria for the evaluation of outsourcing services provided by Logistics
Operators.

3.3 APPLICATION STAGE

For the creation of the evaluation model of outsourcing services provided by
Logistics Operators chose the methodology Multiple Criteria Decision Analysis
(MCDA), considering that this objective analysis clarifies conflicting criteria, enabling
process more rational decisions, justifiable and explainable and can handle mixed data
set of quantitative and qualitative [5]. With the criteria and duly validated sub-criteria
and considered the most important by experts, a questionnaire online (Google Forms /
SurveyMonkey) will be applied, with the participation of over thirty professionals in
the logistics area, only the shippers, the main customers of logistics operators. The
results of the surveys will be treated by applying the approach Macbeth, that allows the
transformation of qualitative assessments into quantitative. The analysis will be the
basis for setting the M-MACBETH software in the creation of the judgment matrix of
each of the criteria studied.
4 STRUCTURE OF THE EVALUATION MODEL

The model developed to evaluate outsourcing services, more specifically, to assess

the significant facilities and difficulties facing customers, shippers, owners of cargo
handled, followed the premises, dictates and epistemological foundations proposed by
[9] and [32]. Based on the model cited by the authors, which has as one of the pillars of
the multi-criteria approach to support decision (MCDA), reported by [29] and [30],
constructivism is adopted as central paradigm and relies on two objectives: assist in the
process of choice, ordering or ranking of alternatives and provide the decision-maker
an instrument that can help you solve decision problems, where many criteria, often
conflicting, must be considered.
4.1 Identification Context Deciding
In cargo movements processes, there are several players, among them: shippers, road
carriers, rail carriers, cargo agents, port operators, wharfage teams, rail-road terminals,
ports, airports and Logistics Operators (LOs), and the last cited cameo holders because
they offer a range of services outsourcing for various business segments.
This work has the participation of logistics operators and shippers of cargo, and they
need to move their goods fashion more effective and efficient possible. For this to
happen, the real understanding of both parties is necessary, both service providers as
the contractor of the main points that deserve alignment expectations. In this context, it
is salutary extreme to identify the points of greatest vulnerability in the processes so
that they can act with the proposition properly targeted management improvements.
Shippers, as well as selecting the logistics operators, need to evaluate the services
through multi-criteria analysis methodology to support their decisions as well as
evidence on which aspects need improvement.

4.2 Construction of the Model

For the construction of the evaluation model, [9] highlight the following steps:
Defining the problem of labelling, identification of the actors involved in the evaluation
process, identification of assessment criteria (criteria), the amount of tree construction,
construction of the descriptors, construction of the Value Functions (VF) and
determining the replacement rates (weights).

4.2.1 Definition of label

Taking into account that this study aims to evaluate the outsourcing of services
provided by Logistics Operators, from the perspective of customers Shippers, the label
set to the model were as follows:

• Of outsourcing services assessment provided by Logistics Operators

4.2.2 Identification of actors and evaluation elements

In constructing the model, some actors involved directly and indirectly in decision-
making, being classified according to [9], in two groups acted and actors. The actors
acted are not directly participating in the decision-making process; however, they are
affected by the decisions and may put pressure on the players. Stakeholders are
constituted by three types of actors: policymakers, representatives, facilitators. The
basis for the evaluation process is supported by a series of steps to identify the
evaluation elements [9].
As mentioning [9], identification of Primary Elements Evaluation (PEE) is required
for further construction of the cognitive map, enabling the identification Fundamental
Points of View (FPVs), and from these are derived the criteria for evaluation.
Identifying steps of the primary elements of evaluations (PEEs) and the development of
cognitive map have been replaced in the search for the application of brainstorming and
Web-Delphi. It is important to note that there are criteria for selection; however, these
selected logistics operators are not evaluated with multiple criteria methodology as the
model developed proposes. Thus, the study makes it possible to achieve the objective,
through which the results achieved will be possible to bring about a better alignment
between the Customer expectations Shipper and logistics operator.
By analyzing the brainstorming and answers rounds Web-Delphi, we came up to six (6)
criteria and 26 (twenty-six) sub-criteria most relevant in the opinion of experts to
evaluate the outsourcing services provided by Logistics Operators.
The evaluation process was divided into six main areas, defined in the survey, as the
Fundamental Point of View. According to [1, 2], is called a family of FPVs, used as a
pillar in the initial structuring of the multi-criteria evaluation model:
FPV 1 - Operational Performance;
FPV 2 - Service Quality;
FPV 3 - Information Technology;
FPV 4 - costs;
FPV 5 - Relationship;
FPV 6 - Risk Management.

After defining the FPVs, due to decomposition was conducted by creating the views
of elementary points (FPVs), both the basic structure of the multi-criteria evaluation
model [27]. It is noteworthy that the decomposition of FPVs (criteria) and EPVs (sub-
criteria) were duly validated by the participating stakeholders Web-Delphi research.
4.2.3 Construction of the value tree
With the definition of the basic structure of the evaluation model, it was possible to
create a value tree, through the arborescent diagram, comprising the following listed
components, duly represented in Figure 2:
 EPV 1.1 Malfunctions

EPV 1.2 Delivery On Time

FPV 1 EPV 1.3 Time Request Cycle

Operational
performance EPV 1.4 Operating area

EPV 1.5 Control deliveries

enttrgasentregaadorias
EPV 1.6 Programming deliveries

EPV 2.2 Storage

EPV 2.1 Transport
FPV 2
Quality of
Services
operational EPV 2.3 Inventory Management

EPV 2.4 Capatazia

EPV 3.1 Tracking load

FPV 3
Information EPV 3.2 Security Information
Technology
EPV 3.3 System Integration

Outsourcin
EPV 4.1 Reduction Costs
g of services
assessment
provided by FPV 4 EPV 4.2 Cost of Services
costs
Logistics
Operators EPV 4.3 Costs of Transport

EPV 5.1 commercial Clarifications

FIG. 2 - value EPV 5.2 Feedback

tree5 of multi-criteria evaluation modelCustomer Service
FPV
sSERVICEcomerciais
Relationship
EPV 5.3 Flexibility

EPV 5.4 Pro team activity

EPV 6.1 Infrastructure Fleet

FPV 6 EPV 6.2 Infrastructure Facilities

Risk
management EPV 6.3 Financial Stability

EPV 6.4 Resolution contingencies

EPV 6.5 Insurance charges

 After completion of the steps of building the "physical structure" of the multi-criteria
evaluation model, the following steps are the structure of the internal aspects of the
model of mensuration nature of potential actions to be evaluated. Following the [27]
model, which highlights the theoretical and epistemological foundations, such aspects
are the descriptors, value functions and replacement rates, to be developed in the next
stages of the research.

5 FINAL CONSIDERATIONS
This study achieved its initial objective because it had the essential participation of
experts, representatives of Shippers Customers and Logistics Operators through
research methodologies brainstorming and Web-Delphi contributed positively to
identify the criteria and sub-criteria, enabling the construction of the value tree for the
future through multi-criteria model, evaluate outsourcing services provided by Logistics
Operators. Through income generated by the research will be possible to apply
management techniques well-founded and directed the real expectations cleared the
logistics tracking, which will provide conditions for more significant gains, the whole
chain of actors involved.

Acknowledgments
Plácido Rogério Pinheiro is grateful to the National Council for Scientific and
Technological Development (CNPq) for the contribution received to develop this
project.

REFERENCES

1. Capistrano, JR; Pinheiro, Placido R; Silva Júnior, JG; Sousa Nogueira, P. Structuring a
Multicriteria Model to Optimize the Profitability of a Clinical Laboratory. In: Silhavy R.,
Silhavy P., Prokopova Z. (Org.). Advances in Intelligent Systems and Computing. 1ed.:
Springer International Publishing, p. 41-51, 2019.
2. Bana and Costa, CA (1992). Structuration, Construction Exploitation et d'un Modèle
Multicritèrie d'Aide à la Décision. Doctoral thesis. Technical University of Lisbon. Lisbon.
3. Bana and Costa, CA; Cutting, JM; Vansnick, JC. MACBETH. International Journal of
Information Technology and Decision Making, Vol. 11, no. 2, p. 359-387, 2012.
4. Vasconcelos, M. F.; Pinheiro, P. R.; Simao Filho, M.. A Multicriteria Model Applied to the
Choice of a Competitive Strategy for the Printed Newspaper. In: Radek Silhavy; Petr Silhavy
Zdenka Prokopova. (Org.). Cybernetics Approaches in Intelligent Systems. 1ed.: Springer,
2017, v. 1, p. 206-215.
5. Belton, S.; Stewart, TS Decision Multiple Criteria Analysis. An Integrated Approach. Kluwer
Academic Publishers Massachusetts, In 2002.
6. Bottani, E.; Rizzi, A. Fuzzy TOPSIS methodology to support the outsourcing of logistics
services. Supply Chain Management: An International Journal, Vol. 11, no. 4, p. 294-308,
2006.
7. Dalkey, N; Helmer, O. An experimental application of the Delphi method to the use of
experts. Management Science, v.9, n. 3, p. 458-467, 1963.
8. Dalkey, N.C; The Delphi method. An experimental study of group opinion. Santa Monica:
Rand Corporation, 1969.
9. Ensslin, L.; Montibeller Neto, G.; Noronha, SM Decision Support - the methodology for
structuring problems and multi-criteria evaluation of alternatives. Florianópolis: Insular,
2001.
10. GIL, Antonio Carlos. How to design research projects. São Paulo: Atlas, 2007
11. Simão Filho, M; Pinheiro, Placido R; Albuquerque, AB. Task Allocation in Distributed
Software Development Aided by Verbal Decision Analysis. In: Radek Silhavy; Roman
Senkerik; Zuzana Kominkova Oplatkova; Petr Silhavy; Zdenka Prokopova. (Org.). Advances
in Intelligent Systems and Computing. 1ed.Switzerland: Springer International Publishing, v.
465, p. 127-137, 2016
12. Lima, KKF; Pinheiro, Placido R; Simão Filho, M. Handing a Hybrid Multicriteria Model for
Choosing Specialists to Analyze Application Management Service Tickets. In: Anna Visvizi
and Miltiades Lytras. (Org.). Springer Proceedings in Complexity. 1ed.: Springer
International Publishing, p. 369-375, 2019.
13. Gomes, LFAM Decision Theory. (Collection Debates Administration). Thomson Learning,
São Paulo, 2007.
 14. Huang, IB; Keisler, J.; Linkov I. Multi-Criteria Decision Analysis in environmental sciences:
Ten years of applications and trends. Science of the total environment, v. 409, 19, p. 3578-
3594, 2011.
15. KPMG Transaction Services Ltd. and Forensic & Mattos Filho, Veiga Filho, Marrey Jr. and
Quiroga Advogados, with the participation of Prof. PhD Paulo Resende, the Dom Cabral
(FDC). Logistics Operators: sector overview, regulatory, technical and operational aspects,
in 2015.
16. Lakatos, MS; Marconi, M. A. Research techniques: planning and execution of research,
sampling and research techniques, processing, analysis and interpretation of data. 3. ed. São
Paulo: Atlas, 1996.
17. Linstone, HA, & Turoff, M. The Delphi method: Techniques and Applications. Addison
Wesley Newark, NJ: New Jersey Institute of Technology. Access in March 2015,
<https://web.njit.edu/~turoff/pubs/delphibook/index.html>. 2002.
18. Luna, MMM. The process of outsourcing of logistic activities. In: Management Symposium
of Production, Logistics and International Operations - SIMPOI, 9, São Paulo: FGV-EAESP,
2006.
19. Marques, Joan Brás Balcony; FREITAS, Denise. Delphi Method: Characterization And
Potential Research In Education. n. 29, No. 2 May / Aug. 2018.
20. Martino, Joseph P. Technological Forecasting for Decision Making. 3. Ed. New York:
Mcgraw-Hill Inc., 1993.
21. Morais, DC Multi-Criteria Modelling Group for Strategic Planning Control Losses In The
Water Supply. 148 F. Thesis (Phd) - Post-Graduate Program in Production Engineering,
Federal University of Pernambuco, Recife, 2006.
22. Moura, B. Logistics: Concepts and Trends. Atlantic Center, 2006
23. Munaretto, Lorimar Francisco; Correa, Luiz Hamilton; CUNHA, Júlio Araújo Carneiro Da.
A Study on The Delphi Method of Characteristics And The Focal Group As Techniques To
Obtain Data On Exploratory Research. Rev. Adm. UFSM, Santa Maria, v. 6, No. 1, p. 09-24,
jan./mar. 2013
24. Oliveira, Vm; Martins, MF; Vasconcelos, ACF Interviews "in-depth" in qualitative research
in management: theoretical and methodological ways: Symposium of Production
Management, Logistics and International Operations, 2012, São Paulo. São Paulo: FGV,
2012.
25. Osborne, C. S. Ratcliffe, M. Millar, R., & Duschl, A. What "Ideas-about-Science" should be
taught in school Science. A Delphi study of the community expert. Journal of Research in
teaching Science, 40 (7), 692-720, 2003.
26. Pinto, Marta Manuela Sousa. Outsourcing as The Company's Strategy - The Case MD Group.
81f, Dissertation (Marketing and Communication Master) - School of Education of Coimbra,
Coimbra, 2019.
27. Quirino, MG Incorporation of Subordination Relations In The Order Matrix - Roberts In
MCDA When Axioms Of Asymmetry And Transitivity Negative Are Violated. 211F. Thesis
(Doctorate in Production Engineering. Federal University of Santa Catarina), 2002.
28. Reichert, LJ Evaluation of organic potato production systems on family farms: an application
of multi-criteria decision support methodology (MCDA). 346 f. Thesis (PhD) - Post-Graduate
Program in Family Agricultural Production Systems, Faculty of Agronomy Eliseu Maciel,
Federal University of Pelotas, Pelotas, 2012.
29. Roy, B. Decision Science or Science Decision-Aid? European Journal Of Operational
Research. North-Holland, V. 66, P. 184-203. 1993
30. Roy, B. Methodology for Multicriteria Decision Aiding. Kluwer Academic Publishers,
Boston. 1996.
31. Silva, Luiz Marcel Chagas. Mota, Cesar Farias Marcelo. Outsourcing the Logistics Sector in
Industries. Magazine Contribuiciones Las Ciencias Sociales.2019. Available At:
<Https://Www.Eumed.Net/Rev/Cccss/2019/07/Terceirizacao-Setor-Logistico.Html>.
Accessed On 02 November 2019.
32. Silveira JR., Aldery. Methodology to evaluate the operating conditions of cargo
transportation by cabotage in Brazil, from the perspective of owners. 234 f. Thesis (Doctorate
in Transport). The University of Brasilia. Technology College. Brasilia, 2016.
33. Vasiliauskas, AV, G. Jakubauskas, G. Principle and Benefits of third party logistics approach
When managing the logistics supply chain, transport, 22: 2, 68-72, 2007.
34. Vieria, Ana Cl; Oliveira, Monica D.; Bana and Costa, Carlos A. Improving knowledge
building process within the multi-criteria decision analysis: The Collaborative Modeling
Value frame. Elsevier - Omega, p. 6-12, 2019.
35. Zhu, W., NG. CHS, Wang, Z., Zhao, X. The role of outsourcing process management in
logistics Improving the effectiveness of outsourcing. Elsevier - International Journal of
Production Economics. v. 188, p. 29-40, 2017.

View publication stats