Reliability Engineering and System Safety 177 (2018) 12–23

Contents lists available at ScienceDirect

Reliability Engineering and System Safety

journal homepage: www.elsevier.com/locate/ress

Condition-based maintenance of naval propulsion systems: Data analysis

with minimal feedback
Francesca Cipollini a, Luca Oneto a, Andrea Coraddu b,∗, Alan John Murphy c, Davide Anguita a
a
DIBRIS - University of Genova, Via Opera Pia 13, Genova, I-16145, Italy
b
Naval Architecture, Ocean & Marine Engineering Strathclyde University, Glasgow, G1 1XW, UK
c
Marine, Offshore and Subsea Technology Group, School of Engineering Newcastle University, Newcastle upon Tyne, NE1 7RU, UK

a r t i c l e i n f o a b s t r a c t

Keywords: The maintenance of the several components of a Ship Propulsion Systems is an onerous activity, which need to be
Data analysis efficiently programmed by a shipbuilding company in order to save time and money. The replacement policies
Naval propulsion systems of these components can be planned in a Condition-Based fashion, by predicting their decay state and thus
Condition-based maintenance
proceed to substitution only when really needed. In this paper, authors propose several Data Analysis supervised
Supervised learning
and unsupervised techniques for the Condition-Based Maintenance of a vessel, characterised by a combined
Unsupervised learning
Novelty detection diesel-electric and gas propulsion plant. In particular, this analysis considers a scenario where the collection of
Minimal feedback. vast amounts of labelled data containing the decay state of the components is unfeasible. In fact, the collection
of labelled data requires a drydocking of the ship and the intervention of expert operators, which is usually
an infrequent event. As a result, authors focus on methods which could allow only a minimal feedback from
naval specialists, thus simplifying the dataset collection phase. Confidentiality constraints with the Navy require
authors to use a real-data validated simulator and the dataset has been published for free use through the OpenML
repository.

1. Introduction tenance policies can be divided into two main categories [13,14]: Cor-
rective (CM), and Preventive (PM).
Maintenance is one of the most critical tasks to be designed and CM has been for many years the only way of performing mainte-
programmed by any product-selling company [1–4]. As a fact, every nance, by replacing a component only after its breakdown, thus com-
complex system is designed assigning it a specific life-cycle, which is promising the overall system availability and causing exceptional costs
influenced by different factors such as the raw materials adopted, the and loss in incomes [15]. In PM, instead, a component is replaced when
estimated working hours, and the environmental conditions [5,6]. Nev- it reaches the end of its life cycle before a possible breakdown. One of
ertheless, this time-to-live information is inevitably inaccurate as it is the traditional ways to perform PM is to predetermine a conservative av-
impossible to predict at design phase the exact working conditions in erage estimation of the component time-to-live adopting the experience
which the system will operate [7,8]. In any case, the decay of the sys- gained with all the components belonging to a specific class [16]. Sim-
tem components will require at some point of time to be repaired or ilarly to CM, this particular type of PM, usually called Predetermined
replaced, thus leading to the system halt to perform some maintenance Maintenance (PRM), can bring unnecessary costs, if the replaced com-
tasks [9]. This is the reason why an efficient maintenance program can ponent could have been used more than originally forecast. Moreover,
be time and costs saving since replacing a malfunctioning component PRM does not guarantee to limit the number of faults in a fleet, since
after it has failed during service, results in multiple downsides for the a breakdown could still happen before the replacement takes place. In
system owner company. this case, there is a trade-off between the number of breakdowns and the
The Shipbuilding industry is particularly affected by this problem, lifetime estimation of the components, which is not easy to reach since
as a ship breakdown necessarily requires a drydocking, and retrieving the actual ship usage can be very different from ship to ship. Neverthe-
a stricken vessel offshore is not a trivial task [10,11]. A correct main- less, Condition-Based Maintenance (CBM) can be considered as another
tenance program ensures that a ship works as it was designed, with the way of performing PM, which aims at reducing both the costs of CM
desired level performances, without impacting the service [12]. Main- and PRM by relying on the exact decay state of each component and

∗
Corresponding Author.
E-mail addresses: [email protected] (F. Cipollini), [email protected] (L. Oneto), [email protected] (A. Coraddu),
[email protected] (A.J. Murphy), [email protected] (D. Anguita).

https://doi.org/10.1016/j.ress.2018.04.015
Received 25 August 2017; Received in revised form 20 December 2017; Accepted 18 April 2018
Available online 19 April 2018
0951-8320/© 2018 Elsevier Ltd. All rights reserved.
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

then by efficiently planning its maintenance [17,18]. Note that, condi- out comparing different state-of-the-art methodologies such as Kernel
tion monitoring and failure prediction are two different concepts which Methods [36], Neural Network [37], Gaussian Processes [38], Similar-
are somehow strictly related. In fact, a failure of a component is pre- ity Based Method [39], and Ensemble Methods [40]. These binary clas-
dictable only if it is preceded by a decay in its performance or in the sification techniques are adopted to predict if the efficiency coefficient
performance of some related component [19,20]. is above or below a certain threshold defined by the accepted loss in ef-
Since, in most cases, the decay state of each component cannot be ficiency of the NPS components. Secondly, the same problem has been
tracked with a sensor, CBM requires a model able to predict it based on tackled with another state-of-the-art approach which, in principle, does
other sensors available. In fact, the decay state cannot be easily mea- not need any labelled sample since it searches for novel behaviour in
sured without an interruption of service or a drydock of the ship, situa- the data though a novelty detection algorithms [41,42]. Results show
tion which is usually avoided. To overcome this problem, the available that with just a few labelled samples it is possible to fine tune this last
on-board sensors can be used to collect a huge amount of real-time data methodology to achieve satisfying performances.
which can be stored into historical datasets and adopted in order to This work is the natural continuation of Coraddu et al. [31], where
formulate a statistical Data-Driven Model (DDM) to predict the exact authors presented a dataset published trough the University of California
components decay [21]. In fact, DDMs exploit advanced statistical tech- Irvine (UCI) website of data coming from a simulator of a Frigate, char-
niques to build models directly based on the significant amount of data acterised by a COmbined Diesel ELectric And Gas (CODLAG) propulsion
produced by the logging and monitoring apparatus, without requiring plant. A similar simulator was adopted in this study, characterised by a
any a priori knowledge of the underlining physical phenomena [22– higher amount of decaying components, and it will be published through
24]. Considering the estimated state of decay, it is possible to schedule the OpenML dataset repository [43].
each component’s replacement before failures occur, maximising its life The paper is organised as follows. Section 2 reports a general de-
cycle, according to the time required for each maintenance [11]. As a re- scription of the vessel, the numerical model, and the degradation phe-
sult, the additional costs of CM and PRM can be replaced with the lower nomena. Section 3 presents a description of the dataset extracted from
ones of equipping the propulsion system with sensors and by collecting, the numerical simulator and published through OpenML. Section 4 re-
storing, and analysing these data for the purpose of creating effective ports the proposed DDMs. Results of the DDMs tested on the proposed
predictive DDMs [10,11]. data are reported in Section 5 with conclusions in Section 6.
In this paper, authors address the problem of building effective DDMs
to predict the main components decay state in a Naval Propulsion Sys-
2. Naval propulsion system
tem (NPS) for CBM purposes. In particular, the decay of a vessel Gas
Turbine (GT), Gas Turbine Compressor (GTC), Hull (HLL) and Propeller
2.1. Vessel description
(PRP) is estimated. Many examples of Data Analysis (DA) techniques ap-
plied to different CBM problems can be found in literature [25]. Among
In this work authors focus on a Frigate, characterised by a COD-
other, Support Vector Machines [26], Hidden Markov Models [27] and
LAG NPS, widespread detailed in [31]. In particular, the GT mechani-
Kalman filter [28] are the most frequently used. Examples of DA ap-
cally drives the two Controllable Pitch Propellers (CPP) through a cross-
proaches applied to the marine industry can be found in [29], where
connected gearbox (GB). Besides, each shaft has its electric propulsion
a standard Neural Network approach is used to improve monitoring of
motor (EPM) mounted on the two shaft-lines. Two clutches between the
Gas Turbines, while Kernel based methods are applied in [30,31]. In
GB and the two EPM and another clutch between the GT and the GB as-
[32,33] image processing techniques are adopted for hull condition as-
sure the possibility of using two different type of prime movers, i.e. EPM
sessment. In [34] the engine and propeller state is predicted adopting an
and GT. Finally, the electric power is provided by four diesel generators
Artificial Neural Network. A complete overview can be found in [35].
(DG). This particular GB arrangement, allows the vessel to operate un-
In particular, this work can be seen as the continuation of Coraddu
der different propulsive configurations to achieve the requirements of
et al. [31], where a similar approach was attempted adopting a smaller
the vessel’s mission profile. The vessel is characterised by the follow-
amount of decayed NPS components and supervised Machine Learning
ing mission profiles: Anti-Submarine Warfare (ASW), General-Purpose
(ML) regression models in order to predict their exact decay. Neverthe-
(GEP) and Anti-Aircraft Warfare (AAW). In particular, for the ASW pro-
less, in [31] it was proven that a significant amount of historical data
file, the EPMs are prime movers while the GT is disconnected through
needed to be collected, together with the actual state of decay of each
the clutches. Under the GEP mission profile, the GT is the prime mover
component. In the end, this approach resulted not feasible in a real-
while the EPMs are working as shaft generators. Finally, for the AAW
world scenario where the labelling process requires the intervention of
mission profile both the GT and the EPM are the prime movers. In this
an experienced operator and, in some cases, to stop the vessel or even
work, only the GT operating conditions have been taken into account.
to put the ship in a dry dock.
As a result, authors here propose a different approach where col-
lecting labeled samples is an easier task that can be performed by less 2.2. Model description
experienced operators since the raw information about the decay is re-
quested and it can be retrieved without impacting the ship activities. In this work, authors consider an NPS numerical model developed in
Specifically, two approaches are here proposed and compared. First, the Matlab® Simulink® software environment within many years of re-
authors build virtual sensors able to continuously estimate the need for search [44]. The numerical model is composed of several modules each
replacement of the components based on other sensors measurements one representing a single propulsion component such as the hull, the
which are indirectly influenced by this decay. Then authors try to per- main engines, the propellers, the rudders, the GB, and the control sys-
form the same analysis, in conditions where only few labelled samples tem. In the previous literature, authors presented a model that considers
are present, by adopting a DDM which require a limited amount of in- the GT and GTC decay performance [31]. The model is now further im-
formation to achieve satisfying performance. To the best knowledge of proved to take into account the performance decay of the HLL and PRP,
the authors, the novelty of the proposed work relies on its ability of and is now readily to undertake a holistic approach in addressing the
building a model whose accuracy is comparable with the state-of-the- performance decay by accounting the important components as follows:
art supervised learning techniques, adopting only an extremely limited
number of labelled samples. 1. Gas Turbine (GT);
For this reason, firstly authors performed a traditional classification 2. Gas Turbine Compressor (GTC);
analysis where the target is to estimate the label state of the components 3. Hull (HLL);
described with an efficiency coefficient. The analysis has been carried 4. Propeller (PRP).

13
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

3. From data to condition-based maintenance

3.1. Dataset creation

In this work, authors will use the real-data validated complex numer-
ical simulator of a Navy frigate described in Section 2 to build a realistic
set of data for designing and test purposes of DDMs. This dataset will
be released for the use to the research community on the widespread
well-known dataset repository OpenML [43]. Currently, it can be down-
loaded from https://cbm-anomaly-detection.smartlab.ws.
The NPS model input parameters are provided in [53] however, for
clarity in this paper they are repeated verbatim here as follows:
• Speed: this parameter is controlled by the control lever. The latter
can only assume a finite number of positions lpi with i ∈ {0, ⋅⋅⋅, 9},
which in turn correspond to a finite set of possible configurations for
Fig. 1. Correction factors for thrust reduction, kKt , and torque increase, kKq .
fuel flow and blade position. Each set point is designed to reach a
desired speed vi with i ∈ {0, ⋅⋅⋅, 9}:

𝑣𝑖 = 3 ∗ 𝑙𝑝𝑖 [Knots], ∀𝑖 ∈ {0, ⋯ , 9}. (1)

Note that, if the transients is not taken into account, lpi and vi are
deterministically related by a linear law. In the presented analysis
the transients between different speeds have been not considered.
• As reported in Section 2.3, the PRP thrust and torque decay limit
over two years of operations are:

𝑘𝐾𝑡 ∈ [0.9, 1.0], 𝑘𝐾𝑞 ∈ [1.0, 1.1] (2)

kKt and kKq are respectively the components which define the decay
of the torque and the thrust provided by the propeller in time. They
are linearly correlated, since as the first decay of a certain quantity,
the latter decay of the same quantity (1 − 𝑘𝐾𝑡 = 𝑘𝐾𝑞 − 1). For this
reason only kKt will be analysed, considering the linear dependency
between the two variables.
Fig. 2. Hull resistance increase behaviour over time and speed.
• The HLL decay has been modelled according to the available liter-
ature [52] as described in Section 2.3. The decay limits over two
2.3. Degradation model years of operations are:

𝑘𝐻 ∈ [1, 1.2] (3)

2.3.1. GTC and GT degradation model
As reported in [45,46], fouling of the GTC increases the specific fuel • GTC decay:
consumption and the temperature of the exhaust gas. In agreement with
𝑘𝑀𝑐 ∈ [0.95, 1.0] (4)
previous work, [31], the effect of the fouling is simulated by reducing
the numerical values of the airflow rate Mc and of the isentropic effi- • GT decay:
ciency 𝜂 c with a reduction factor kMc and 𝐾𝜂𝑐 . The detailed description
𝑘𝑀𝑡 ∈ [0.975, 1.0] (5)
of the GTC and GT degradation model is provided in [31,47]. The au-
thors applied a reduction factor, kMT , to the GT flow rate to represent The performance decay functions described in Section 2.3 have been
the effect of fouling. empirically derived as functions of the time variable solely. The real
degradation behaviour of the physical asset should be defined through
2.3.2. PRP degradation model specific functions able to express the time dependency, the mutual in-
An increase in the roughness of the blade surface is the primary cause teractions between the subsystems and the real operational profile. To
of marine PRP performance degradation [48]. This is caused by accre- overcome this issue, authors considered each possible combination of
tions of marine organisms to the metal, alloy erosion and corrosion, GTC, GT, HLL, and PRP decay status based on the described functions,
or combinations of these elements. The PRP decay status has been mod- and sampled the range of decays with a uniform grid characterised by a
elled by increasing the torque coefficient (Kq ) and by reducing the thrust degree of precision sufficient to have a proper granularity of representa-
coefficient (Kt ). The correction factors used for thrust reduction, kKt , and tion. Given the above premises, the evolution of the system between two
torque increase, kKq , have been derived from Atlar et al. [49] and Wan important dry dock maintenance for HLL and PRP can be exhaustively
et al. [50] and reported in Fig. 1. and realistically explored by simulating all its possible decayed states,
as all the components are decaying the same time.
2.3.3. HLL degradation model The space of possible states is described via the following tuple:
The main factors that affect HLL performance are the shape, the coat- ( )
𝑙𝑝, 𝑘𝐾𝑡 , 𝑘𝐾𝑞 , 𝑘𝐻, 𝑘𝑀𝑐 , 𝑘𝑀𝑡 𝑖 , 𝑖 ∈ {1, ⋯ , 455, 625} (6)
ing used and the extent of fouling [51]. The resistance increase due to
fouling has been modelled utilising a correction factor kH based on the since:
information reported in [52]. Using the towing tank data provided by
the ship owner the resistance has been evaluated after one and two years 𝑙𝑝 ∈  𝑙𝑝 = {0, 3, 6, ⋯ , 27}, (7)
of vessel operations in agreement with the data available in the litera-
ture [47]. In Fig. 2 the behaviour of the HLL resistance referred to the
trial condition (𝑅𝑡 ∕𝑅𝑡𝑟𝑒𝑓 ) over the time and speed is reported. 𝑘𝐾𝑡 ∈  𝑘𝐾𝑡 = {0.9, 0.9 + 0.1∕14, 0.09 + 0.2∕14, ⋯ , 1.0}, (8)

14
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

Table 1 predict the actual decay coefficient based on the automation data com-
Measured values available from the continuous ing from the sensors installed on-board [22]. Unfortunately, this ap-
monitoring system. proach cannot be adopted in a real operational scenario. While the sen-
# Variable name Unit sors’ data coming from the automation system are easy to collect, the
information regarding the associated state of decay is not so easy to
1 Lever position []
2 Vessel speed [knots]
retrieve. In fact, to circumvent this challenge to prove DDMs authors
3 GT shaft torque [kN m] exploited a numerical model for gathering all the information and build
4 GT speed [rpm] the dataset presented in Section 3.1. In practice, instead, retrieving the
5 PRP thrust (starboard) [N] state of decay of the different NPS components requires the interven-
6 PRP thrust (port) [N]
tion of an experienced operator and, in some cases, to stop the vessel
7 Shaft torque (port) [kN m]
8 Shaft speed (port) [rpm] or even to put the ship in a dry dock. Moreover, data-driven regression
9 Shaft torque (starboard) [kN m] models require a huge amount of historical data and therefore a long
10 Shaft speed (starboard) [rpm] acquisition time.
11 HP GT exit temperature [o C]
Based on these considerations, authors decided to build simplified
12 Gas generator speed [rpm]
13 Fuel flow (mf) [kg/s]
DDMs able to detect if the component state of decay is above or be-
14 TIC control signal [%] low a certain threshold. These thresholds represent the accepted loss
15 GTC outlet air pressure [bar] in efficiency of the NPS components and the consequently sustainable
16 GTC outlet air temperature [o C] costs of keeping a less performing vessel operative. This approach rep-
17 External pressure [bar]
resents an abstraction of the problem which allows a more practical
18 HP GT exit pressure [bar]
19 TCS TIC control signal [] collection of the state of decay of the component. In fact, instead of re-
20 Thrust coefficient (starboard) [] quiring the precise state of decay, this approach only requires detecting
21 PRP speed (starboard) [rps] if the decay state of the components is acceptable or not. Consequently,
22 Thrust coefficient (port) []
the collection of these data can be performed by less experienced op-
23 PRP speed (port) [rps]
24 PRP torque (port) [kN m]
erators since raw information about the decay is requested and can be
25 PRP torque (starboard) [kN m] retrieved without impacting the ship activities. This new problem can
be straightforwardly mapped into a classical multi-output binary classi-
fication problem [22] where the aim is to predict if the decay state of
an NPS component is acceptable or not based on the automation data
coming from the sensors installed on-board.
𝑘𝐾𝑞 = 2 − 𝑘𝐾𝑡 , (9) The data described in Section 3.1 can be easily exploited to tackle
this new problem as well. In fact, by thresholding kKt , kH, kMc , and kMt
the corresponding binary valued state of decay of the NPS components
are obtained. In other words, if the efficiency coefficients are above or
𝑘𝐻 ∈  𝑘𝐻 = {1.0, 1.0 + 0.2∕14, 1.0 + 0.4∕14, ⋯ , 1.2}, (10)
below a defined threshold, based on the accepted loss in efficiency of
the NPS components, they will be tagged as “decayed” or “not decayed”.
Thresholds were fixed according to the least affordable value of decay
𝑘𝑀𝑐 ∈  𝑘𝑀𝑐 = {0.95, 0.95 + 0.05∕14, 0.95 + 0.1∕14, ⋯ , 1.0}, (11) of the single component. Defining these thresholds is not a trivial task.
Authors approach is to define the maximum level of inefficiency that
the operator or the shipowner is willing to tolerate before taking action
𝑘𝑀𝑡 ∈  𝑘𝑀𝑡 = {0.975, 0.975 + 0.025∕14, 0.975 + 0.05∕14, ⋯ , 1.0}. (12) and re-establish the efficiency of the system. The authors considered two
years as a typical time frame between two important dry dock mainte-
Note that the total number of samples 455, 625 is the result of making
nance for HLL and PRP.
a simulation for each possible combination of decay status (15 values
The HLL and PRP thresholds have been defined considering one year
for GTC, 15 for GT, 15 for HLL, and 15 PRP) and speed (9 values). Once
of operation. The proposed limits are just an example of the possible
these quantities are fixed, the numerical model is run until the steady
selection that is possible to setup to implement a CBM framework
state is reached. Then, the model is able to provide all the quantities {
reported in Table 1. These subsets of models outputs are the same quan- [0.9 − 0.95) decayed
𝑘𝐾𝑡 (13)
tities that the automation system installed on-board can acquire and [0.95 − 1] not decayed
store.
{
The simulator was run on a server equipped with four Intel® Xeon® (1.1 − 1.2] decayed
𝑘𝐻 (14)
CPU E5-4620 2.2 GHz, 128GB of RAM, 120GB SSD disk, and Matlab® [1 − 1.1] not decayed
R2016a.
As for GT and GTC, an effective time service of 2000 h per year
is considered as a reasonable operating time for these vessel types. In
3.2. Condition-based maintenance agreement with these observations authors defined the following thresh-
olds based on the knowledge of the time domain decay functions:
This study aims at estimating the four decay variables described in {
[0.95 − 0.98) decayed
the previous section, adopting different DA techniques. This section re- 𝑘𝑀𝑐 (15)
[0.98 − 1] not decayed
ports how the data generated can be used to create effective predictive
DDMs for the CBM of an NPS. {
[0.975 − 0.99) decayed
The data described in Section 3.1 contain two sets of information: 𝑘𝑀𝑡 (16)
[0.99 − 1] not decayed
one regarding the quantities that the automation system installed on-
board can acquire and store and the other one regarding the associated Results will show that estimating if the decay state is acceptable or
state of decay (efficiency coefficient) of the different NPS components not, instead of estimating its specific state, remarkably reduces the num-
(GT, GTC, HLL, and PRP). ber of samples required to find accurate DDMs. However, this quantity is
This problem could have been straightforwardly mapped into a clas- still too large with respect to what can be collected in a real operational
sical multi-output regression problem, as in [31], where the aim is to scenario.

15
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

To solve this issue, authors tried to look at the same problem from When both xi and yi with i ∈ {1, ⋅⋅⋅, n} are available, the problems
another perspective. Specifically, it is reasonable to state that, for the is named supervised and consequently supervised ML technique are
vast majority of the time, NPS components of the ships operate in an adopted [22]. Classification is one of the most popular examples of su-
acceptable state of decay. Consequently, most of the sensor data col- pervised ML problems [36]. In classification, the output space is com-
lected by the automation system represent ordinary operating condi- posed of a finite set of c possibilities  ∈ {𝐶1 , ⋯ , 𝐶𝑐 }. Binary classifica-
tions corresponding to a reasonable decay state of the NPS components tion is a particular example of classification problem where  ∈ {±1}.
(GT, GTC, HLL, and PRP). Just very few times during the ship lifetime When just xi with i ∈ {1, ⋅⋅⋅, n} are available, which means that the
it happens that it has to operate with over-decayed components. If, for associated element of the output space yi with i ∈ {1, ⋅⋅⋅, n} is not explic-
some reasons, one or more NPS components decay too fast, the cor- itly known, it has to be assumed that “similar” xi are associated with
responding automation data measurements will deviate from their ex- “similar” yi where the concept of similarity is something that needs to
pected behaviour. This new problem can be straightforwardly mapped be defined based on 𝜇. In this last case, the ML problems are called
into a classical outlier (novelty) detection problem [36,42,54] where the unsupervised, and consequently, unsupervised ML techniques need to
aim is to detect unexpected behaviour in the sensor data collected by the be adopted [55]. Anomaly (novelty, outlier) detection is a common
automation system which may correspond to an over-decayed state of example of unsupervised learning problem where the unknown 𝑦 ∈ 
an NPS components. This method does not require to know either the can assume only two possible values: −1 for “non-anomaly” and +1 for
actual state of decay of the components, as a regression task would do, “anomaly” [36].
or the less detailed information about “decayed” or “not decayed”, as in The error that f commits in approximating 𝜇 is measured with ref-
the binary classification framework. In this case, the method just needs erence to a loss function 𝓁 ∶  ×  ×  → [0, ∞). Obviously, the error
the sensor data collected by the automation system (see Table 1) with- that f commits over 𝑛 , is optimistically biased since 𝑛 has been
out any supervision or feedback from the operator. These kinds of DDMs used, together with  , for building f itself. For this reason, another
try to build a model of the “usual” operational profile of the ship and set of fresh data, composed of m samples and called test set 𝑚 =
automatically detect if the sensor data collected by the automation sys- {(𝒙𝑡1 , 𝑦𝑡1 ), ⋯ , (𝒙𝑡𝑚 , 𝑦𝑡𝑚 )}, needs to be exploited. Note that, 𝒙𝑡𝑖 ∈  and 𝑦𝑡𝑖 ∈
tem are “deviating too much” from the established behaviour. In our  with i ∈ {1, ⋅⋅⋅, m}, and the association of 𝑦𝑡𝑖 to 𝒙𝑡𝑖 is again made based
context “usual” means that the efficiency of GT, GTC, HLL, and PRP are on 𝜇. Moreover, both for supervised and unsupervised problems 𝑚 must
in the acceptable range while “deviating too much” means that they are contain both 𝒙𝑡𝑖 ∈  and 𝑦𝑡𝑖 ∈  with i ∈ {1, ⋅⋅⋅, m} to estimate the error
not in the acceptable range, according to Eqs. (13)–(16). of f, while, for unsupervised learning problems, yi with i ∈ {1, ⋅⋅⋅, n} in
As for the binary classification framework, the data described in 𝑛 is unknown.
Section 3.1 can be easily exploited to tackle this problem as well. In
fact, it is just necessary to keep the data corresponding to an acceptable 4.1. Measuring the error
decay state with respect to kKt , kH, kMc , and kMt and in accordance with
Eqs. (13)–(16). Finally, for testing and tuning the DDMs, it is possible to In this work, many state-of-the-art ML techniques will be tested and
use just a few samples of the dataset corresponding to an unacceptable their performances will be compared to understand what is the most
decay state. Note that these are the only samples which are costly to suited solution for building CBM DDMs for NPS.
retrieve since they are the only ones that require the intervention of ex- In order to perform this analysis, authors have to define different
pert operators. Results will show that with just very few samples ( ≈ 10) measures of error, also called indexes of performance, able to well char-
of decayed state of the vessel, it is possible to obtain effective DDMs for acterize the quality of the different CBM DDMs for NPS. Once f has been
CBM of NPS. chosen based on 𝑛 , it is possible to use the fresh set of data 𝑚 in or-
As a final remark, authors would like to recall that each navy frigate der to compute its error based on different losses. The choice of the loss
is characterised by different mission profiles (AAW, ASW, and GEP) as strongly depends on the problem under examination [56].
described in Section 2.1. Each mission profile is characterised by a par- In the classification framework, the most natural choice as loss func-
ticular use of the ship in terms of speed. CBM DDMs for NPS do not need tion is the Hard loss one, which counts the number of misclassified sam-
to estimate the state of decay of the NPS components for all the possible ples 𝓁𝐻 (𝑓 (𝒙), 𝑦) = [𝑓 (𝒙) ≠ 𝑦]. Note that the Iverson bracket notation is
mission profiles. In fact, the vessel operates at a cruise speed (which is exploited. In this work, only binary classification problems are inves-
approximately ≈ 15 knots) while the time spent by the vessel at different tigated, then the Hard loss function can be expressed as 𝓁𝐻 (𝑓 (𝒙), 𝑦) =
speeds is negligible. For this reason, authors conduct the same analysis 1 − 𝑦𝑓 (𝒙)∕2. Moreover, this measure will be also used for the anomaly
described in the previous paragraph by setting 𝑙𝑝 = 15. Results will show detection problems since, also in this case, a binary output is considered
that this simplification will further reduce the amount of historical data (non-anomaly or anomaly).
needed to build effective CBM DDMs for NPS. Based on the Hard loss it is possible to define different indexes of
performance [57]:
4. Machine learning
• the Average Misclassifications Rate (AMR) is the mean number of
∑
In this section, authors will present the ML techniques adopted in misclassified samples: AMR = 𝑚1 𝑚 𝑡 𝑡
𝑖=1 𝓁𝐻 (𝑓 (𝒙𝑖 ), 𝑦𝑖 );
• the Confusion Matrix, which measures four different quantities:
order to build the CBM DDMs for NPS described in Section 2, based on ∑
the data outlined in Section 3. - TN = 100∕𝑚 𝑚 𝑡 𝑡 𝑡
𝑖=1 [𝑓 (𝒙𝑖 = 𝑦𝑖 ∧ 𝑦𝑖 = −1] which is the percentage of
Let authors consider an input space  ⊆ ℝ𝑑 and an output space . true negative;
∑
Note that, for what concerns this paper,  takes into account the dif- - TP = 100∕𝑚 𝑚 𝑡 𝑡 𝑡
𝑖=1 [𝑓 (𝒙𝑖 = 𝑦𝑖 ∧ 𝑦𝑖 = +1] which is the percentage of
ferent sensors measurements, also called features, reported in Table 1, true positive;
∑
while the output space  depends on the particular problem identi- - FN = 100∕𝑚 𝑚 𝑡 𝑡 𝑡
𝑖=1 [𝑓 (𝒙𝑖 ≠ 𝑦𝑖 ∧ 𝑦𝑖 = −1] which is the percentage of
fied in Section 3.2. ML techniques aim at estimating the unknown rule false negative;
∑
𝜇 ∶  →  which associates an element 𝑦 ∈  to an element 𝒙 ∈ . - FP = 100∕𝑚 𝑚 𝑡 𝑡 𝑡
𝑖=1 [𝑓 (𝒙𝑖 ≠ 𝑦𝑖 ∧ 𝑦𝑖 = +1] which is the percentage of
Note that, in general, 𝜇 can be non-deterministic. An ML technique false positive.
estimates 𝜇 through a learning algorithm 𝒜 ∶ 𝑛 ×  → ℎ, character-
ized by its set of hyperparameters , which maps a series of exam- 4.2. Machine learning techniques
ples of the input/output relation contained in a dataset of n samples
{( ) ( )}
𝑛 ∶ 𝒙1 , 𝑦1 , ⋯ , 𝒙𝑛 , 𝑦𝑛 into a function 𝑓 ∶  →  chosen in a set of In this section, authors will present the supervised and unsupervised
possible ones  . learning algorithms exploited in this paper for building CBM DDMs for

16
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

NPS. Moreover, authors will show how to tune their performances by In fact, this algorithm defines the probability distribution of the output
tuning their hyperparameters during the so-called Model Selection (MS) values as a sum of Gaussians whose variance is fixed according to the
phase [58–60]. Finally, authors will also check for possible spurious training data. The hyperparameter of the GP  GP is the parameter which
correlation in the data by performing the Feature Selection (FS) phase governs the Gaussians width h1 .
[61–65]. In fact, once f is built based on the different learning algorithm LMs ML techniques are learning method in which the definition of
and has been confirmed to be a sufficiently accurate representation of f is delayed until f(x) needs to be computed [39]. LMs approximate 𝜇
𝜇, it can be interesting to investigate how the model f is affected by the locally with respect to x. K-Nearest Neighbors (KNN) is one of the most
different features that have been exploited to build f itself during the popular LM due to its implementation simplicity and effectiveness [88].
feature ranking procedure [61]. As authors will describe later, for some The hyperparameter of the KNN  KNN is the number of neighbors of x
algorithms, the feature ranking procedure is a by-product of the learning to be considered h1 .
process itself and allows to simply check the physical plausibility of f.
4.2.2. Unsupervised learning algorithms for anomaly detection
4.2.1. Supervised classification learning algorithms Similarly to their supervised counterpart, also unsupervised ML
Supervised ML techniques can be grouped into different families, ac- methods can be divided into different families. Since this work deals
cording to the space of function  from which the learning algorithm only with anomaly detection problems, authors will recall the most
chooses the particular f, the approximation of 𝜇, based on the avail- known and effective techniques for solving these problems according
able data 𝑛 . In fact, techniques belonging to the same family, share an to Swersky et al. [42]. In particular Swersky et al. [42] shows that two
affine  . Among the several possible ML families, authors choose the anomaly detection methods based on SVM and KNN respectively, are
state-of-the-art ones which are commonly adopted in real-world appli- the top choices in this context.
cation, and, in each family, the best performing techniques are selected. In particular One-Class SVM (OCSVM) is a boundary-based anomaly
In particular, Neural Networks (NNs), Kernel Methods (KMs), Ensemble detection method, inspired by SVM, which enclose the inlier class in
Methods (EMs), Bayesian Methods (BMs), and Lazy Methods (LMs) are a minimum volume hypersphere by minimizing a Tikhonov regular-
adopted. ization problem, similar to the one reported for SVM framework. Like
NNs are ML techniques which combine together many simple mod- traditional SVMs, OCSVM can also be extended to non-linearly trans-
els of a human brain neuron, called perceptrons [37], in order to build formed spaces using the “Kernel trick” for distances. The hyperparame-
a complex network. The neurons are organized in stacked layers con- ters OCSVM  OCSVM are the same as the ones of SVM.
nected together by weights that are learned based on the available data The Global KNN (GKNN), inspired by the KNN, has been originally
via backpropagation [66]. The hyperparameters of an NN  NN are the introduced as an unsupervised distance-based outlier detection method
number of layers h1 and the number of neurons for each layer h2, i with [42,89]. The hyperparameter GKNN  GKNN is the same as the one of
i ∈ {1, ⋅⋅⋅, h1 }. Note that it is assumed that NN with only one hidden KNN.
layer has ℎ1 = 1. If the architecture of the NN consists of only one hid-
4.2.3. Model selection
den layer, it is called shallow (SNN) [67,68], while, if multiple layers
MS deals with the problem of tuning the hyperparameters of each
are staked together, the architecture is defined as deep (DNN) [69–71].
learning algorithm [60]. Several methods exist for MS purpose: re-
Extreme Learning Machines (ELMs) are a particular kind of SNN, where
sampling methods, like the well-known k-Fold Cross Validation (KCV)
the weights of the first layer are randomly chosen, while the ones of the
[58] or the nonparametric Bootstrap (BTS) approach [90,91], which
output layers are computed according to the Regularized Least Squares
represent the state-of-the-art MS approaches when targeting real-world
(RLS) principle [72–74]. The hyperparameters of the ELM  ELM are the
applications. Resampling methods rely on a simple idea: the original
number of neurons of the hidden layer, h1 , and the RLS regularization
dataset 𝑛 is resampled once or many (nr ) times, with or without
hyperparameter h2 [75].
replacement, to build two independent datasets called training, and
KMs are a family of ML techniques which exploits the “Kernel trick”
validation sets, respectively 𝑟𝑙 and 𝑣𝑟 , with r ∈ {1, ⋅⋅⋅, nr }. Note that
for distances in order to extend linear techniques to solve non-linear
𝑟𝑙 ∩ 𝑣𝑟 = ⊘, 𝑟𝑙 ∪ 𝑣𝑟 = 𝑛 . Then, to select the best combination the hy-
problems [76,77]. In the case of classification, KMs select the function f
perparameters  in a set of possible ones ℌ = {1 , 2 , ⋯} for the al-
minimizing the trade-off between the sum of the accuracy over the data,
gorithm 𝒜 or, in other words, to perform the MS phase, the following
namely the empirical error, and the solution complexity, namely the reg-
procedure has to be applied:
ularization term [76,78,79]. The most effective KM techniques are: Ker-
𝑛𝑟
nelized Regularized Least Squares (KRLS), and Support Vector Machines 1 ∑1 ∑
∗ ∶ min 𝓁(𝒜,𝑟 (𝒙𝑖 ), 𝑦𝑖 ), (17)
(SVMs). The hyperparameters of the KRLS  KRLS are: the kernel, which ∈ℌ 𝑛𝑟 𝑟=1 𝑣 (𝒙𝑖 ,𝑦𝑖 )∈𝑣𝑟
𝑙

in this paper is fixed to Gaussian Kernel for the reasons described in

where 𝒜,𝑟 is a model built with the algorithm 𝒜 with its set of hyper-
[80,81], its hyperparameter h1 and the regularization hyperparameter 𝑙

h2 . SVM, instead, is a classification method, which roots in the Statisti- parameters  and with the data 𝑟𝑙 . Since the data in 𝑟𝑙 are independent
cal Learning Theory [22] and differs from the KRLS mainly because of from the ones in 𝑣𝑟 , the idea is that  ∗ should be the set of hyperpa-
its particular loss function [56]. The hyperparameters of the SVM are rameters which allows to achieve a small error on a data set that is
the same as the one of the KRLS. independent from the training set.
EMs ML techniques rely on the fact that combining the output of Note that, for the anomaly detection problem, the algorithms do not
several classifiers results in a much better performance than using any need any label in 𝑟𝑙 , consequently authors just need the labelled data
one of them alone [40,82]. Random Forest (RF) [40] and Random Ro- for 𝑣𝑟 .
tation Ensembles (RRF) [83], two popular state-of-the-art and widely If 𝑟 = 1, if l and v are aprioristically set such that 𝑛 = 𝑙 + 𝑣, and if
adopted methods, combine many decision trees in order to obtain effec- the resample procedure is performed without replacement, the hold out
tive predictors which have limited hyperparameter sensitivity and high method is obtained [60]. For implementing the complete KCV, instead,
( )(𝑛− 𝑛 )
numerical robustness [84,85]. Both RF and RRF have hidden hyperpa- it is needed to set 𝑟 ≤ 𝑘𝑛 𝑘 , 𝑙 = (𝑘 − 2) 𝑛 , 𝑣 = 𝑛 , and 𝑡 = 𝑛 and the
𝑘 𝑘 𝑘
𝑘
rameters which are arbitrarily fixed in this work because of their limited resampling must be done without replacement [58,60,92]. Finally, for
effect [86]. implementing the BTS, 𝑙 = 𝑛 and 𝑟𝑙 must be sampled with replacement
BMs are ML techniques where, instead of choosing a particular 𝑓 ∈  from 𝑛 , while 𝑣𝑟 and 𝑡 𝑟 are sampled without replacement from the
a distribution for choosing 𝑓 ∈  is defined [87]. Gaussian Processes sample of 𝑛 that have not been sampled in 𝑟𝑙 [60,90]. Note that for
( )
(GP) learning algorithm is a popular BM [38] which employs a collection the BTS procedure 𝑟 ≤ 2𝑛𝑛−1 . In this paper the BTS is exploited because
of Gaussians in order to compute the posterior distribution of the f(x). it represents the state-of-the-art approach [60,90].

17
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

4.3. Feature selection 7. GP: the set of hyperparameters is  𝐺𝑃 = {ℎ1 } and authors chose it
in ℌGP = {100 , 100.3 , ⋯ , 103 };
Once the CBM NPS models are built and have been confirmed to
When RF is exploited, also the FS phase is performed to understand
be sufficiently accurate representation of the real decays of the com-
how the data-driven model combines the different features in order to
ponents, it can be interesting to investigate how these models are af-
predict the decay state of each component.
fected by the different features used in the model identification phase
Similarly to the supervised learning task, in the unsupervised case
(see Table 1).
different dimensions of the training set were considered n ∈ {1500,
In DA this procedure is called FS or Feature Ranking [61–65]. This
2000, 3000, 4000} and the MS procedure was performed as follows:
process allows detecting if the importance of those features, that are
known to be relevant from a physical perspective, is appropriately de- 1. OCSVM: the set of hyperparameters is  𝑂𝐶𝑆𝑉 𝑀 = {ℎ1 , ℎ2 } and au-
scribed by the different CBM NPS models. The failure of the statistical thors chose it in ℌOCSVM = {10−4 , 10−3.7 , ⋯ , 103 } × {10−4 , 10−3.8 , ⋯ ,
model to properly account for the relevant features might indicate poor 10−1.0 };
quality in the measurements or spurious correlations. FS therefore rep- 2. GKNN: the set of hyperparameters is  𝐺𝐾𝑁𝑁 = {ℎ1 } and authors
resents an important step of model verification, since it should generate chose it in ℌGKNN = {1, 3, 7, 13, 27, 51};
consistent results with the available knowledge of the physical system The 𝑣𝑟 cardinality was varied v ∈ {10, 20, 30} with linear step, in
under exam. order to test the possibility of building an efficient model with a few
In addition to its use for classification purposes, the EMs can also be labelled samples. Note that, also in this case, the BTS MS procedure is
used to perform a very stable FS procedure. The procedure is a com- adopted with 𝑟 = 1000 and that the labels are only needed in 𝑣𝑟 and not
bination of EMs, together with the permutation test [93], in order to in 𝑟𝑙 as described in Section 4.2.3.
perform the selection and the ranking of the features. In details, for ev- The performances of each model are measured according to the met-
ery tree, two quantities are computed: the first one is the error on the rics described in Section 4.1. Each experiment was performed 10 times
out-of-bag samples as they are used during prediction, while the second in order to obtain statistical relevant result, and the t-student 95% confi-
one is the error on the out-of-bag samples after a random permutation dence interval is reported when space in the table was available without
of the values of variable j. These two values are then subtracted and compromising their readability.
the average of the result over all the trees in the ensemble is the raw For SNN and DNN the Python Keras library [94] has been exploited.
importance score for variable j (mean decrease in accuracy). This pro- For ELM, SVM, KRLS, KNN, and GKNN a custom R implementation has
cedure was adopted since it can be easily carried out during the main been developed. For RF the R package of Liaw and Wiener [95] has
prediction process inexpensively. been exploited. For RFE the implementation of Blaser and Fryzlewicz
[83] has been exploited. For GP the R package of Zeileis et al. [96] has
been exploited. For OCSVM the R package of Meyer et al. [97] has been
5. Results exploited.

In this section, authors report the results obtained by the different 5.1. CLASS-PROB
methods applied to the CBM of the main components of an NPS, in clas-
sification, and novelty detection frameworks, as described in Section 2, In this section, the results on the CLASS-PROB are reported. In Fig. 3
based on the data described in Section 3. the AMR of the models learned with the different algorithms is reported,
As described in Section 4, the first problem is a classification one when varying n and for the four main NPS components. In Fig. 4 the
where the label of the decay parameters needs to be estimated (CLASS- AMR of the DNN (the best performing model) is reported, when varying
PROB). Secondly, authors attempted to solve the same problem in an n and for the four main NPS components.
unsupervised fashion by modelling the problem as a novelty detection From the different tables and figures it is possible to observe that:
one in order to further reduce the necessity of labelled data (ANOMALY-
• the larger is n the better performances are achieved by the learned
PROB).
models (see Fig. 3) and the models learned with ELM, SNN, and
The dataset considered in Section 3.2 was divided into training and
especially DNN generally show the best performances (see Fig. 3);
test set, respectively 𝑛 and 𝑚 , as reported in Section 4. Moreover,
• as expected, to achieve a reasonable AMR a smaller number of sam-
different dimensions of the training set n ∈ {10, 24, 55, 130, 307, 722,
ples is needed with respect to a regression-based approach not fea-
1700, 4000} were considered. These dimensions were derived by con-
sible in practice.
sidering 8 values on a logarithmic scale from 10 to 4000, to analyse the
behaviour of the predictive models in different conditions. In Fig. 5 the FS phase for the four main NPS components is re-
For each supervised classification ML technique, the BTS MS proce- ported. Taking into account the problem P15, for each feature reported
dure was performed with 𝑟 = 1000, as described in Section 4.2.3. Here- in Table 1, the mean decrease in accuracy as described in Section 4.3 is
below, the list of hyperparameters tested during the MS, with their re- reported. From Fig. 5, it is possible to note that the RF model can ad-
spective intervals, is reported: equately account for the relevant features as the outcome is consistent
with the available knowle.g. [53]. According to Fig. 5, several features
1. DNN: the set of hyperparameters is  𝐷𝑁𝑁 = {ℎ1 , ℎ2,1 , ⋯ , ℎ2,ℎ1 } and are always necessary to forecast the decay state of each component. As
authors chose it in ℌDNN = {1, 3, 5, 7, 10} × {10, 101.2 ⋯ , 103 } × ⋯ × far as the PRP component is concerned, the thrust features (5 and 6), the
{10, 101.2 ⋯ , 103 }; shaft torque features (7 and 9) and the PRP torque features (24 and 25)
2. SNN: the set of hyperparameters is  𝑆𝑁𝑁 = {ℎ1 } and authors chose have high predictive power. As expected, for the HLL component the
it in ℌSNN = {1, 3, 5, 7, 10}; thrust coefficients features (20 and 22) have the most significant pre-
3. ELM: the set of hyperparameters is  𝐸𝐿𝑀 = {ℎ1 , ℎ2 } and authors dictive power. When it comes to the GTC component, the features de-
chose it in ℌELM = {10, 101.2 , ⋯ , 103 } × {10−2 , 10−1.5 ⋯ , 102 }; scribing the thermodynamic process have the highest predictive power,
4. SVM: the set of hyperparameters is  𝑆𝑉 𝑀 = {ℎ1 , ℎ2 } and authors nominally GTC outlet air temperature, External pressure, and HP GT
chose it in ℌSVM = {10−2 , 10−1.4 , ⋯ , 103 } × {10−2 , 10−1.4 ⋯ , 103 }; exit temperature features (16, 17 and 11). Finally, for the GT compo-
5. KRLS: the set of hyperparameters is  𝐾𝑅𝐿𝑆 = {ℎ1 , ℎ2 } and authors nent prediction, several features are necessary, also this case is in line
chose it in ℌKRLS = {10−2 , 10−1.4 , ⋯ , 103 } × {10−2 , 10−1.4 , ⋯ , 103 }; with engineering state-of-the-art knowledge [53]. These results indicate
6. KNN: the set of hyperparameters is  𝐾𝑁𝑁 = {ℎ1 } and authors chose that, from a data driven perspective, the decay state of each component
it in ℌKNN = {1, 3, 7, 13, 27, 51}; influences different phases of the NPS behaviour.

18
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

PRP HLL
0.55 0.5
KRLS KRLS
0.5 ELM 0.45 ELM
KNN KNN
0.45 RF RF
0.4
GP GP
0.4 SVM SVM
0.35
DNN DNN
0.35 RFR RFR
SNN 0.3 SNN
AMR

AMR
0.3
0.25
0.25
0.2
0.2

0.15
0.15

0.1 0.1

0.05 0.05
0 50 100 150 200 250 300 350 400 0 50 100 150 200 250 300 350 400
n n

GTC GT
0.5 0.5
KRLS KRLS
0.45 ELM 0.45 ELM
KNN KNN
0.4 RF RF
0.4
GP GP
0.35 SVM SVM
0.35
DNN DNN
0.3 RFR RFR
SNN 0.3 SNN
AMR

AMR

0.25
0.25
0.2
0.2
0.15

0.15
0.1

0.05 0.1

0 0.05
0 50 100 150 200 250 300 350 400 0 50 100 150 200 250 300 350 400
n n

Fig. 3. CLASS-PROB: AMR of the models learned with the different algorithms when varying n and for the four main NPS components.

Fig. 5 clearly shows that the interaction between the main compo-
0.55
nents cannot be easily modelled with a physical approach, considering
PRP
0.5 HLL the large number of variables that affect the final behaviour of each
GTC component. Instead, DDMs, by making use of these variables, can out-
0.45 GT perform physical models as they have the capability to take into account
all the available sensors measurements to build effective and accurate
0.4 predictors as reported in Fig. 3.
0.35
AMR

5.2. ANOMALY-PROB
0.3
In this section the results on the ANOMALY-PROB are reported. In
0.25 Table 2, for PRP, HLL, GTC, and GT, the AMR of the models learned
with the different algorithms (OCSVM and GKNN) is reported, when the
0.2
number of unlabelled samples in the learning set is 𝑙 = 4000 and when
0.15
varying the number of labelled samples in the validation set v ∈ {10,
20, 30} (half positively and half negatively labelled). In Table 3, respec-
0.1 tively for PRP, HLL, GTC, and GT, the AMR of the models learned with
0 50 100 150 200 250 300 350 400 the different algorithms is reported, when 𝑣 = 30 and when l ∈ {1500,
n 2000, 3000, 4000}. In Table 4, for PRP, HLL, GTC, and GT, the differ-
ent indexes of performances (AMR, TP, TN, FP, and FN) of the models
Fig. 4. CLASS-PROB: AMR of the models learned with DNN when varying n for
the four main NPS components.
learned with the different algorithms are reported when 𝑛 = 4000 and
𝑣 = 30.

19
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

P15:PRP P15:HLL
0.12 0.18

0.16
0.1
0.14
Mean Decrease in Accuracy

Mean Decrease in Accuracy

0.08 0.12

0.1
0.06
0.08

0.04 0.06

0.04
0.02
0.02

0 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
feature feature

P15:GTC P15:GT
0.2 0.14

0.18
0.12
0.16
Mean Decrease in Accuracy

Mean Decrease in Accuracy

0.14 0.1

0.12
0.08
0.1
0.06
0.08

0.06 0.04

0.04
0.02
0.02

0 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
feature feature

Fig. 5. CLASS-PROB: FS performed with RF for the four main NPS components.

Table 2 Table 3
ANOMALY-PROB: AMR of the models learned with the different algo- ANOMALY-PROB: AMR of the models learned with the different algorithms
rithms (OCSVM and GKNN) when 𝑙 = 4000 and v ∈ {10, 20, 30} and for (OCSVM and GKNN) when l ∈ {1500, 2000, 3000, 4000} and 𝑣 = 30 and
the four main NPS components. for the four main NPS components.

v PRP HLL GTC GT n PRP HLL GTC GT

OCSVM 10 0.08 ± 0.08 0.07 ± 0.09 0.05 ± 0.07 0.11 ± 0.06 OCSVM 1500 0.09 ± 0.04 0.14 ± 0.09 0.11 ± 0.09 0.08 ± 0.03
20 0.08 ± 0.10 0.08 ± 0.07 0.10 ± 0.07 0.09 ± 0.06 2000 0.12 ± 0.10 0.11 ± 0.04 0.10 ± 0.04 0.11 ± 0.13
30 0.08 ± 0.07 0.12 ± 0.08 0.10 ± 0.07 0.09 ± 0.03 3000 0.08 ± 0.06 0.11 ± 0.04 0.13 ± 0.17 0.08 ± 0.05
GKNN 10 0.07 ± 0.07 0.07 ± 0.09 0.04 ± 0.07 0.10 ± 0.06 4000 0.08 ± 0.07 0.12 ± 0.08 0.10 ± 0.07 0.09 ± 0.03
20 0.08 ± 0.10 0.08 ± 0.07 0.08 ± 0.07 0.07 ± 0.06 GKNN 1500 0.09 ± 0.04 0.12 ± 0.09 0.10 ± 0.09 0.06 ± 0.03
30 0.08 ± 0.07 0.12 ± 0.08 0.08 ± 0.08 0.08 ± 0.03 2000 0.11 ± 0.11 0.12 ± 0.04 0.08 ± 0.04 0.08 ± 0.13
3000 0.07 ± 0.06 0.10 ± 0.04 0.12 ± 0.16 0.09 ± 0.04
4000 0.08 ± 0.07 0.12 ± 0.08 0.08 ± 0.08 0.08 ± 0.03

From the tables it is possible to observe that:

• in some cases, the mean value of the reported AMR increases instead
• both OCSVM and GKNN perform quite well on the problem and there of decreasing when the number of training samples increases; as a
is no clear winner; fact, this value is subject to statistical variation in the data whose
• changing l or v does not remarkably affect the performance of the variance in the results can be considered acceptable from a statistical
models; point of view. Note also that the number of validation samples is very
• with just few labelled samples, around 10, it is possible to obtain limited and this high variance is also justified by this fact;
satisfying accuracies and this is quite a remarkable result, since 10 • FP and FN rate are quite balanced and this is a further indication of
samples can be easily manually labelled by an expert operator; the quality of the result.

20
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

Table 4
Table A.5
ANOMALY-PROB: the different indexes of performances (AMR, TP, TN, FP, and
CLASS-PROB: AMR of the models learned with the different algorithms
FN) of the models learned with the different algorithms (OCSVM and GKNN)
(DNN, SNN, ELM, SVM, KRLS, KNN, and GP) when varying n for the four
when 𝑛 = 4000 and 𝑣 = 3 and for the four main NPS components.
main NPS components.
PRP HLL GTC GT
n PRP HLL GTC GT
OCSVM AMR 0.08 ± 0.07 0.12 ± 0.08 0.10 ± 0.07 0.09 ± 0.03
KRLS 10 0.50 ± 0.06 0.48 ± 0.06 0.46 ± 0.10 0.44 ± 0.09
TP 46.68 ± 8.88 45.29 ± 9.25 47.89 ± 3.17 46.59 ± 5.49
24 0.43 ± 0.10 0.43 ± 0.10 0.43 ± 0.07 0.43 ± 0.07
TN 45.47 ± 3.88 42.56 ± 7.54 42.42 ± 9.36 44.56 ± 4.50
55 0.29 ± 0.06 0.25 ± 0.07 0.37 ± 0.06 0.42 ± 0.08
FN 3.32 ± 8.88 4.71 ± 9.25 2.11 ± 3.17 3.41 ± 5.49
130 0.17 ± 0.05 0.17 ± 0.06 0.25 ± 0.12 0.32 ± 0.09
FP 4.53 ± 3.88 7.44 ± 7.54 7.58 ± 9.36 5.44 ± 4.50
307 0.13 ± 0.02 0.11 ± 0.02 0.13 ± 0.03 0.19 ± 0.03
GKNN AMR 0.08 ± 0.07 0.12 ± 0.08 0.08 ± 0.08 0.08 ± 0.03
722 0.10 ± 0.02 0.07 ± 0.02 0.08 ± 0.03 0.12 ± 0.03
TP 46.68 ± 8.66 45.89 ± 9.04 48.04 ± 3.16 47.42 ± 5.41
1700 0.06 ± 0.01 0.04 ± 0.02 0.05 ± 0.02 0.07 ± 0.02
TN 45.24 ± 3.86 42.30 ± 7.61 44.14 ± 9.55 44.49 ± 4.47
4000 0.03 ± 0.01 0.02 ± 0.01 0.04 ± 0.02 0.05 ± 0.01
FN 3.32 ± 8.84 4.11 ± 9.21 1.96 ± 3.13 2.58 ± 5.54
ELM 10 0.40 ± 0.22 0.44 ± 0.13 0.46 ± 0.16 0.45 ± 0.09
FP 4.76 ± 3.87 7.70 ± 7.54 5.86 ± 9.29 5.51 ± 4.56
24 0.28 ± 0.14 0.31 ± 0.18 0.31 ± 0.20 0.42 ± 0.16
55 0.19 ± 0.08 0.23 ± 0.09 0.16 ± 0.07 0.22 ± 0.07
130 0.12 ± 0.05 0.11 ± 0.05 0.08 ± 0.03 0.12 ± 0.03
307 0.09 ± 0.04 0.06 ± 0.03 0.06 ± 0.02 0.07 ± 0.01
6. Conclusions
722 0.05 ± 0.04 0.04 ± 0.02 0.05 ± 0.02 0.05 ± 0.02
1700 0.03 ± 0.01 0.03 ± 0.01 0.03 ± 0.01 0.03 ± 0.01
The maintenance of the several components of a Ship Propulsion Sys- 4000 0.01 ± 0.01 0.01 ± 0.01 0.01 ± 0.01 0.02 ± 0.02
tems is an onerous activity, which needs to be efficiently programmed KNN 10 0.50 ± 0.07 0.47 ± 0.06 0.46 ± 0.10 0.43 ± 0.08
by a shipbuilding company to save time and money. The replacement 24 0.46 ± 0.08 0.43 ± 0.15 0.43 ± 0.08 0.43 ± 0.07
55 0.29 ± 0.07 0.28 ± 0.12 0.39 ± 0.05 0.42 ± 0.08
policies of these components can be planned in a Condition-Based fash-
130 0.21 ± 0.05 0.21 ± 0.06 0.30 ± 0.03 0.40 ± 0.07
ion, by predicting their decay state and thus proceed to substitution 307 0.17 ± 0.02 0.15 ± 0.02 0.24 ± 0.04 0.33 ± 0.04
only when needed. In this paper, authors proposed several Data Analysis 722 0.15 ± 0.03 0.12 ± 0.02 0.18 ± 0.03 0.26 ± 0.02
supervised and unsupervised techniques for the Condition-Based Main- 1700 0.13 ± 0.02 0.10 ± 0.02 0.13 ± 0.02 0.20 ± 0.02
tenance of a naval vessel, characterised by a combined diesel-electric 4000 0.09 ± 0.01 0.07 ± 0.01 0.10 ± 0.02 0.15 ± 0.02
RF 10 0.47 ± 0.07 0.43 ± 0.12 0.47 ± 0.09 0.47 ± 0.07
and gas propulsion plant. The propulsion plant has been modelled us- 24 0.40 ± 0.12 0.36 ± 0.15 0.45 ± 0.07 0.45 ± 0.04
ing the state-of-the-art simulation techniques available in the literature 55 0.31 ± 0.08 0.25 ± 0.06 0.41 ± 0.04 0.45 ± 0.03
[44,47]. The dataset used to benchmark the proposed data-driven ap- 130 0.19 ± 0.04 0.19 ± 0.05 0.31 ± 0.06 0.38 ± 0.04
proaches has been created using a realistic simulator of Frigate validated 307 0.13 ± 0.04 0.12 ± 0.04 0.21 ± 0.05 0.30 ± 0.04
722 0.07 ± 0.02 0.07 ± 0.02 0.13 ± 0.01 0.21 ± 0.03
and fine-tuned during sea trials. The model has been designed to work
1700 0.04 ± 0.02 0.04 ± 0.01 0.09 ± 0.02 0.13 ± 0.02
in calm water scenario, and measurement uncertainties have not been 4000 0.02 ± 0.01 0.02 ± 0.01 0.06 ± 0.02 0.08 ± 0.02
taken into account. Within the mentioned limitations of the numerical GP 10 0.49 ± 0.05 0.47 ± 0.07 0.46 ± 0.07 0.44 ± 0.05
model, the authors are confident that the results shown are in line with 24 0.42 ± 0.11 0.40 ± 0.11 0.42 ± 0.07 0.45 ± 0.09
the real behaviour of the system. The authors considered the case in 55 0.29 ± 0.05 0.27 ± 0.05 0.37 ± 0.04 0.44 ± 0.07
130 0.20 ± 0.05 0.20 ± 0.03 0.30 ± 0.03 0.39 ± 0.08
which GTC, GT, HLL, and PRP NPS components decay at the same time, 307 0.17 ± 0.02 0.14 ± 0.03 0.22 ± 0.02 0.32 ± 0.06
to provide a realistic simulation environment. 722 0.13 ± 0.03 0.11 ± 0.03 0.15 ± 0.03 0.24 ± 0.04
The proposed analysis considered contexts where the collection of 1700 0.11 ± 0.02 0.07 ± 0.02 0.10 ± 0.03 0.17 ± 0.03
vast amounts of labelled data containing the exact decay state of the 4000 0.08 ± 0.01 0.04 ± 0.01 0.07 ± 0.02 0.11 ± 0.02
SVM 10 0.46 ± 0.06 0.46 ± 0.07 0.44 ± 0.11 0.43 ± 0.10
components is unfeasible. In fact, the collection of labelled data requires
24 0.40 ± 0.11 0.41 ± 0.11 0.41 ± 0.07 0.40 ± 0.08
a drydocking of the ship and the intervention of expert operators, which 55 0.27 ± 0.06 0.24 ± 0.07 0.35 ± 0.07 0.41 ± 0.09
is usually an infrequent event. As a result, authors focused on methods 130 0.17 ± 0.06 0.17 ± 0.07 0.23 ± 0.14 0.31 ± 0.10
which could allow only a minimal feedback from naval specialists, thus 307 0.13 ± 0.03 0.11 ± 0.02 0.12 ± 0.03 0.18 ± 0.04
simplifying the dataset collection phase. In particular, supervised Data 722 0.10 ± 0.02 0.07 ± 0.02 0.07 ± 0.03 0.11 ± 0.03
1700 0.06 ± 0.02 0.04 ± 0.02 0.05 ± 0.02 0.07 ± 0.02
Analysis techniques allowed to reach an average percentage error of 4000 0.03 ± 0.01 0.02 ± 0.01 0.03 ± 0.02 0.05 ± 0.01
1% ± 2% adopting 4000 labelled samples respectively for all PRP, HLL, DNN 10 0.35 ± 0.12 0.42 ± 0.08 0.42 ± 0.09 0.41 ± 0.05
GTC, and GT. On the other hand, the non-supervised techniques ex- 24 0.25 ± 0.08 0.28 ± 0.10 0.28 ± 0.11 0.37 ± 0.09
ploited could reach an average value of 8% ± 5% error adopting only 10 55 0.18 ± 0.05 0.20 ± 0.05 0.14 ± 0.04 0.20 ± 0.04
130 0.11 ± 0.03 0.09 ± 0.02 0.08 ± 0.02 0.11 ± 0.02
labelled data overall. To reach the same average error percentage, the
307 0.08 ± 0.02 0.06 ± 0.02 0.05 ± 0.01 0.06 ± 0.01
obtained supervised models required at least a number of labelled data 722 0.04 ± 0.02 0.04 ± 0.01 0.04 ± 0.01 0.04 ± 0.01
between 130 and 307, thus requiring a higher amount of information 1700 0.02 ± 0.01 0.02 ± 0.01 0.02 ± 0.01 0.03 ± 0.01
for learning a performing model. Clearly, supervised models could reach 4000 0.01 ± 0.00 0.01 ± 0.01 0.01 ± 0.00 0.02 ± 0.01
a lower error rate with respect to unsupervised ones (1% ± 2%), adopt- RFR 10 0.40 ± 0.04 0.36 ± 0.07 0.39 ± 0.05 0.39 ± 0.04
24 0.33 ± 0.06 0.30 ± 0.08 0.38 ± 0.04 0.38 ± 0.02
ing a higher amount of labelling training data, but such a difference in 55 0.26 ± 0.04 0.22 ± 0.03 0.35 ± 0.02 0.40 ± 0.02
accuracy is obtained through higher costs for dataset collection, which 130 0.16 ± 0.02 0.17 ± 0.03 0.27 ± 0.03 0.34 ± 0.02
cannot be sustained in most cases. 307 0.11 ± 0.02 0.11 ± 0.02 0.19 ± 0.03 0.26 ± 0.02
In conclusion, this study proved that it is possible to treat a 722 0.06 ± 0.01 0.06 ± 0.01 0.11 ± 0.01 0.19 ± 0.02
1700 0.04 ± 0.01 0.03 ± 0.01 0.07 ± 0.01 0.11 ± 0.01
Condition-Based Maintenance problem in an unsupervised fashion, with
4000 0.02 ± 0.01 0.02 ± 0.01 0.05 ± 0.01 0.07 ± 0.01
results close to the ones obtained with supervised techniques present in SNN 10 0.38 ± 0.12 0.43 ± 0.08 0.43 ± 0.09 0.42 ± 0.05
literature. These models can be adopted for real-time applications di- 24 0.25 ± 0.08 0.30 ± 0.10 0.29 ± 0.10 0.40 ± 0.09
rectly on-board, to easily and quickly identify maintenance necessities. 55 0.19 ± 0.05 0.22 ± 0.05 0.16 ± 0.04 0.21 ± 0.04
130 0.12 ± 0.03 0.10 ± 0.02 0.08 ± 0.02 0.11 ± 0.02
307 0.08 ± 0.02 0.06 ± 0.02 0.06 ± 0.01 0.07 ± 0.01
Appendix A. CLASS-PROB extended results 722 0.04 ± 0.02 0.04 ± 0.01 0.04 ± 0.01 0.04 ± 0.01
1700 0.02 ± 0.01 0.02 ± 0.01 0.03 ± 0.01 0.03 ± 0.01
In this section, the results for the CLASS-PROB, from which 4000 0.01 ± 0.00 0.01 ± 0.01 0.01 ± 0.00 0.02 ± 0.01
Figs. 3 and 4 were derived, are extensively reported. In Table A.5, re-

21
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

Table A.6 [7] Peng Y, Dong M, Zuo MJ. Current status of machine prognostics in condition-based
CLASS-PROB: the different indexes of performances (AMR, TP, TN, FP, and maintenance: a review. Int J Adv Manuf Technol 2010;50(1–4):297–313.
FN) of the models learned with the different algorithms (DNN, SNN, ELM, [8] Kyrtatos NP, Tzanos E, Coustas J, Vastarouhas D, Rizos E. Shipboard engine perfor-
mance assessment by comparing actual measured data to nominal values produced
SVM, KRLS, KNN, and GP) when n is the largest possible for the four main
by detailed engine simulations. In: Proceedings; 2010.
NPS components. [9] Goossens AJM, Basten RJI. Exploring maintenance policy selection using the an-
alytic hierarchy process; an application for naval ships. Reliab Eng Syst Saf
PRP HLL GTC GT
2015;142:31–41.
KRLS AMR 0.03 ± 0.01 0.02 ± 0.01 0.04 ± 0.01 0.05 ± 0.00 [10] Widodo A, Yang B. Support vector machine in machine condition monitoring and
TP 45.11 ± 1.10 45.68 ± 1.79 58.06 ± 1.67 57.29 ± 1.37 fault diagnosis. Mech Syst Signal Process 2007;21(6):2560–74.
[11] Mobley RK. An introduction to predictive maintenance. Butterworth-Heinemann;
TN 51.72 ± 1.02 52.33 ± 1.97 38.31 ± 1.45 37.77 ± 1.50
2002.
FN 1.64 ± 0.39 0.84 ± 0.28 1.96 ± 0.74 2.81 ± 0.55
[12] Management GBQ, Committee SS, Institution BS. British standard glossary of main-
FP 1.53 ± 0.47 1.15 ± 0.35 1.67 ± 0.49 2.13 ± 0.37
tenance management terms in terotechnology. British Standards Institution; 1984.
ELM AMR 0.01 ± 0.00 0.01 ± 0.01 0.01 ± 0.00 0.02 ± 0.01 [13] Budai-Balke G. Operations research models for scheduling railway infrastructure
TP 45.05 ± 1.87 46.32 ± 0.85 60.18 ± 1.14 57.51 ± 1.80 maintenance. Rozenberg Publishers; 2009.
TN 53.60 ± 1.67 52.22 ± 1.08 38.33 ± 1.23 40.20 ± 1.53 [14] Abu-Elanien AEB, Salama MMA. Asset management techniques for transformers.
FN 0.80 ± 0.31 0.64 ± 0.30 0.73 ± 0.24 1.94 ± 0.75 Electr Power Syst Res 2010;80(4):456–64.
FP 0.55 ± 0.22 0.82 ± 0.41 0.76 ± 0.31 0.35 ± 0.27 [15] Kothamasu R, Huang SH. Adaptive mamdani fuzzy model for condition-based main-
KNN AMR 0.09 ± 0.01 0.07 ± 0.01 0.10 ± 0.01 0.15 ± 0.01 tenance. Fuzzy Sets Syst 2007;158(24):2715–33.
TP 42.06 ± 0.63 43.10 ± 1.75 55.23 ± 1.68 52.65 ± 1.70 [16] Selvik JT, Aven T. A framework for reliability and risk centered maintenance. Reliab
TN 48.46 ± 1.03 49.56 ± 1.93 34.92 ± 1.53 31.96 ± 1.81 Eng Syst Saf 2011;96(2):324–31.
FN 4.69 ± 0.51 3.42 ± 0.53 4.79 ± 0.71 7.45 ± 0.68 [17] Mann L, Saxena A, Knapp GM. Statistical-based or condition-based preventive main-
FP 4.79 ± 0.59 3.92 ± 0.40 5.06 ± 0.79 7.94 ± 0.82 tenance? J Quality Maint Eng 1995;1(1):46–59.
[18] ISO BS. 13372: 2012, condition monitoring and diagnostics of machines-vocabulary.
RF AMR 0.02 ± 0.01 0.02 ± 0.01 0.06 ± 0.01 0.08 ± 0.01
British standards; 2004.
TP 45.81 ± 1.90 45.70 ± 1.47 57.17 ± 1.01 56.31 ± 1.53
[19] Ghasemi A, Yacout S, Ouali MS. Parameter estimation methods for condition-based
TN 52.03 ± 1.68 51.89 ± 1.26 37.31 ± 1.01 35.78 ± 1.61
maintenance with indirect observations. IEEE Trans Reliab 2010;59(2):426–39.
FN 1.23 ± 0.42 1.29 ± 0.37 2.52 ± 0.39 3.50 ± 0.77 [20] Jardine AKS, Banjevic D, Wiseman M, Buck S, Joseph T. Optimizing a mine haul
FP 0.93 ± 0.36 1.12 ± 0.48 3.00 ± 0.73 4.41 ± 0.73 truck wheel motors? condition monitoring program use of proportional hazards
GP AMR 0.08 ± 0.01 0.04 ± 0.01 0.07 ± 0.01 0.11 ± 0.01 modeling. J Quality Maint Eng 2001;7(4):286–302.
TP 42.33 ± 0.84 44.17 ± 1.94 57.43 ± 1.55 57.07 ± 1.51 [21] Linoff GS, Berry MJA. Data mining techniques: for marketing, sales, and customer
TN 49.99 ± 0.91 51.55 ± 1.90 35.71 ± 1.32 32.39 ± 1.45 relationship management. John Wiley & Sons; 2011.
FN 4.42 ± 0.67 2.35 ± 0.85 2.59 ± 0.74 3.03 ± 0.63 [22] Vapnik VN. Statistical learning theory. Wiley New York; 1998.
FP 3.26 ± 0.86 1.93 ± 0.51 4.27 ± 0.73 7.51 ± 1.03 [23] Györfi L, Kohler M, Krzyzak A, Walk H. A distribution-free theory of nonparametric
SVM AMR 0.03 ± 0.01 0.02 ± 0.01 0.03 ± 0.01 0.05 ± 0.00 regression. Springer Science & Business Media; 2006.
TP 45.15 ± 1.21 45.70 ± 1.95 58.16 ± 1.90 57.36 ± 1.48 [24] Newton LR. Data-logging in practical science: research and reality. Int J Sci Educ
TN 51.79 ± 1.13 52.43 ± 2.15 38.37 ± 1.57 37.95 ± 1.65 2000;22(12):1247–59.
FN 1.60 ± 0.42 0.82 ± 0.30 1.86 ± 0.83 2.74 ± 0.59 [25] Jardine AKS, Lin D, Banjevic D. A review on machinery diagnostics and prog-
nostics implementing condition-based maintenance. Mech Syst Signal Process
FP 1.46 ± 0.53 1.05 ± 0.39 1.61 ± 0.53 1.95 ± 0.39
2006;20(7):1483–510.
DNN AMR 0.01 ± 0.00 0.01 ± 0.01 0.01 ± 0.00 0.02 ± 0.01
[26] Poyhonen S, Jover P, Hyotyniemi H. Signal processing of vibrations for condition
TP 45.10 ± 2.08 46.40 ± 0.96 60.25 ± 1.20 57.65 ± 2.02 monitoring of an induction motor. Symposium on control, communications and sig-
TN 53.63 ± 1.83 52.30 ± 1.23 38.40 ± 1.38 40.24 ± 1.75 nal processing; 2004.
FN 0.75 ± 0.35 0.56 ± 0.34 0.66 ± 0.26 1.80 ± 0.85 [27] Bunks C, McCarthy D, Al-Ani T. Condition-based maintenance of machines using
FP 0.52 ± 0.25 0.74 ± 0.45 0.69 ± 0.34 0.31 ± 0.31 hidden markov models. Mech Syst Signal Process 2000;14(4):597–612.
RFR AMR 0.02 ± 0.01 0.02 ± 0.01 0.05 ± 0.01 0.07 ± 0.01 [28] Simani S, Fantuzzi C, Patton R. Model-based fault diagnosis in dynamic systems
TP 46.05 ± 2.12 45.96 ± 1.63 57.56 ± 1.08 56.68 ± 1.64 using identification techniques. Springer-Verlag London; 2003.
TN 52.21 ± 1.88 52.05 ± 1.43 37.70 ± 1.07 36.45 ± 1.72 [29] Palmé T, Breuhaus P, Assadi M, Klein A, Kim M. New alstom monitoring tools lever-
FN 0.99 ± 0.48 1.03 ± 0.39 2.13 ± 0.45 3.13 ± 0.88 aging artificial neural network technologies. Turbo expo: turbine technical confer-
FP 0.75 ± 0.40 0.96 ± 0.53 2.61 ± 0.83 3.74 ± 0.78 ence and exposition; 2011.
SNN AMR 0.01 ± 0.00 0.01 ± 0.01 0.01 ± 0.00 0.02 ± 0.01 [30] Singer RM, Gross KC, King RW. A pattern-recognition-based, fault-tolerant monitor-
TP 45.10 ± 2.00 46.33 ± 0.93 60.19 ± 1.25 57.56 ± 1.89 ing and diagnostic technique. Argonne national lab., Idaho Falls, ID (United States);
1995.
TN 53.64 ± 1.90 52.25 ± 1.17 38.36 ± 1.41 40.20 ± 1.63
[31] Coraddu A, Oneto L, Ghio A, Savio S, Anguita D, Figari M. Machine learning ap-
FN 0.75 ± 0.34 0.63 ± 0.32 0.72 ± 0.28 1.89 ± 0.79
proaches for improving condition-based maintenance of naval propulsion plants.
FP 0.51 ± 0.25 0.79 ± 0.47 0.73 ± 0.34 0.35 ± 0.29
Proc Inst Mech Eng Part M 2016;230(1):136–53.
[32] Akinfiev TS, Armada MA, Fernandez R. Nondestructive testing of the state of a ship’s
hull with an underwater robot. Russ J Nondestr Test 2008;44(9):626–33.
[33] Bagavathiappan S, Lahiri BB, Saravanan T, Philip J, Jayakumar T. Infrared thermog-
spectively for PRP, HLL, GTC, and GT, the AMR of the models learned raphy for condition monitoring-a review. Infrared Phys Technol 2013;60:35–55.
[34] Basurko OC, Uriondo Z. Condition-based maintenance for medium speed diesel en-
with the different algorithms (DNN, SNN, ELM, SVM, KRLS, KNN, and
gines used in vessels in operation. Appl Therm Eng 2015;80:404–12.
GP) is reported, when varying n. In Table A.6, instead, respectively for [35] Lazakis I, Dikis K, Michala AL, Theotokatos G. Advanced ship systems condition
PRP, HLL, GTC, and GT, the different indexes of performances (AMR, monitoring for enhanced inspection, maintenance and decision making in ship op-
TP, TN, FP, and FN) of the models learned with the different algorithms erations. Transp Res Procedia 2016;14:1679–88.
[36] Shawe-Taylor J, Cristianini N. Kernel methods for pattern analysis. Cambridge uni-
are reported, when n is the largest possible. versity press; 2004.
[37] Rosenblatt F. The perceptron: a probabilistic model for information storage and or-
ganization in the brain. Psychol Rev 1958;65(6):386.
References [38] Rasmussen CE. Gaussian processes for machine learning. Gaussian processes for ma-
chine learning; 2006.
[1] Ahmad R, Kamaruddin S. An overview of time-based and condition-based mainte- [39] Duch W. Similarity-based methods: a general framework for classification, approxi-
nance in industrial application. Comput Indus Eng 2012;63(1):135–49. mation and association. Control Cybern 2000;29(4).
[2] Pinjala SK, Pintelon L, Vereecke A. An empirical investigation on the relationship [40] Breiman L. Random forests. Mach Learn 2001;45(1):5–32.
between business and maintenance strategies. Int J Prod Econ 2006;104(1):214–29. [41] Markou M, Singh S. Novelty detection: a review. Signal Process
[3] Tsang AHC. Condition-based maintenance: tools and decision making. J Quality 2003;83(12):2481–97.
Maint Eng 1995;1(3):3–17. [42] Swersky L, Marques HO, Sander J, Campello RJ, Zimek A. On the evaluation of
[4] Dekker R, Scarf PA. On the impact of optimisation models in maintenance decision outlier detection and one-class classification methods. IEEE international conference
making: the state of the art. Reliab Eng Syst Saf 1998;60(2):111–19. on data science and advanced analytics; 2016.
[5] Takata S, Kirnura F, van Houten FAM, Westkamper E, Shpitalni M, Ceglarek D, [43] Vanschoren J, van Rijn JN, Bischl B, Torgo L. Openml: networked science in machine
et al. Maintenance: changing role in life cycle management. CIRP Ann-Manuf Tech- learning. SIGKDD Explor 2013;15(2):49–60.
nol 2004;53(2):643–55. [44] Altosole M, Benvenuto G, Figari M, Campora U. Real-time simulation of a cogag
[6] Frangopol D, Dan M, Bocchini D, Decò A, Kim D, Kwon D, et al. Integrated life-cycle naval ship propulsion system. J Eng Maritime Environ 2009;223(1):47–62.
framework for maintenance, monitoring, and reliability of naval ship structures. Nav [45] Tarabrin AP, Schurovsky VA, Bodrov AI, Stalder JP. An analysis of axial compressor
Eng J 2012;124(1):89–99. fouling and a blade cleaning method. J Turbomach 1998;120(2):256–61.

22
F. Cipollini et al. Reliability Engineering and System Safety 177 (2018) 12–23

[46] Meher-Homji CB, Focke AB, Wooldridge MB. Fouling of axial flow compressors [72] Huang G, Huang G-B, Song S, You K. Trends in extreme learning machines: a review.
- causes, effects, detection, and control. Eighteenth turbomachinery symposium; Neural Netw 2015;61:32–48.
1989. [73] Huang GB, Zhu QY, Siew CK. Extreme learning machine: theory and applications.
[47] Altosole M, Campora U, Martelli M, Figari M. Performance decay analysis of a marine Neurocomputing 2006;70(1):489–501.
gas turbine propulsion system. J Ship Res 2014;58(3):117–29. [74] Cambria E, Huang G-B. Extreme learning machines. IEEE Intell Syst
[48] Khor YS, Xiao Q. Cfd simulations of the effects of fouling and antifouling. Ocean Eng 2013;28(6):30–59.
2011;38(10):1065–79. [75] Huang G-B, Zhu Q-Y, Siew C-K. Extreme learning machine: a new learning scheme
[49] Atlar M, Glover EJ, Candries M, Mutton RJ, Anderson CD. The effect of a foul release of feedforward neural networks. IEEE international joint conference on neural net-
coating on propeller performance. Marine science and technology for environmental works; 2004.
sustainability; 2002. [76] Cristianini N, Shawe-Taylor J. An introduction to support vector machines and other
[50] Wan B, Nishikswa E, Uchida M. The experiment and numerical calculation of pro- kernel-based learning methods. Cambridge university press; 2000.
peller performance with surface roughness effects. J Kansai Soc Naval Architects, [77] Scholkopf B. The kernel trick for distances. Advances in neural information process-
Japan 2002;2002(238):49–54. ing systems; 2001.
[51] Lindholdt A, Dam-Johansen K, Olsen SM, Yebra DM, Kiil S. Effects of biofouling [78] Rifkin R, Yeo G, Poggio T. Regularized least-squares classification. Nato Sci Ser Sub
development on drag forces of hull coatings for ocean-going ships: a review. J Coat Ser III Comput Syst Sci 2003;190:131–54.
Technol Res 2015;12(3):315–444. [79] Lee WS, Bartlett PL, Williamson RC. The importance of convexity in learning with
[52] Hadler JB, Wilson CJ, Beal AL. Ship standardisation trial performance and correla- squared loss. IEEE Trans Inf Theory 1998;44(5):1974–80.
tion with model predictions. The society of naval architects and marine engineers; [80] Keerthi SS, Lin CJ. Asymptotic behaviors of support vector machines with gaussian
1962. kernel. Neural Comput 2003;15(7):1667–89.
[53] Cipollini F, Oneto L, Coraddu A, Murphy A, Anguita D. Condition-based main- [81] Oneto L, Ghio A, Ridella S, Anguita D. Support vector machines and strictly positive
tenance of naval propulsion systems with supervised data analysis. Ocean Eng definite kernel: The regularization hyperparameter is more important than the ker-
2018;149:268–78. nel hyperparameters. IEEE international joint conference on neural networks; 2015.
[54] Hawkins DM. Identification of outliers. Springer; 1980. [82] Germain P, Lacasse A, Laviolette F, Marchand M, Roy JF. Risk bounds for the ma-
[55] Hastie T, Tibshirani R, Friedman J. Unsupervised learning. The elements of statistical jority vote: from a pac-bayesian analysis to a learning algorithm. J Mach Learn Res
learning; 2009. 2015;16(1):787–860.
[56] Rosasco L, De Vito E, Caponnetto A, Piana M, Verri A. Are loss functions all the [83] Blaser R, Fryzlewicz P. Random rotation ensembles. J Mach Learn Res 2015;2:1–15.
same? Neural Comput 2004;16(5):1063–76. [84] Fernández-Delgado M, Cernadas E, Barro S, Amorim D. Do we need hundreds
[57] Powers DM. Evaluation: from precision, recall and f-measure to roc, informedness, of classifiers to solve real world classification problems. J Mach Learn Res
markedness and correlation. J Mach Learn Technol 2011;2(1):37–63. 2014;15(1):3133–81.
[58] Kohavi R, et al. A study of cross-validation and bootstrap for accuracy estimation [85] Wainberg M, Alipanahi B, Frey BJ. Are random forests truly the best classifiers? J
and model selection International joint conference on artificial intelligence; 1995. Mach Learn Res 2016;17(110):1–5.
[59] Bartlett PL, Boucheron S, Lugosi G. Model selection and error estimation. Mach Learn [86] Orlandi I, Oneto L, Anguita D. Random forests model selection. European sympo-
2002;48(1–3):85–113. sium on artificial neural networks, computational intelligence and machine learning;
[60] Anguita D, Ghio A, Oneto L, Ridella S. In-sample and out-of-sample model selection 2016.
and error estimation for support vector machines. IEEE Trans Neural Netw Learn [87] Gelman A, Carlin JB, Stern HS, Rubin DB. Bayesian data analysis, 2. Chapman and
Syst 2012;23(9):1390–406. Hall/CRC; 2014.
[61] Guyon I, Elisseeff A. An introduction to variable and feature selection. J Mach Learn [88] Cover T, Hart P. Nearest neighbor pattern classification. IEEE Trans Inf Theory
Res 2003;3:1157–82. 1967;13(1):21–7.
[62] Friedman J, Hastie T, Tibshirani R. The elements of statistical learning. Springer [89] Ramaswamy S, Rastogi R, Shim K. Efficient algorithms for mining outliers from large
series in statistics Springer, Berlin; 2001. data sets. ACM sigmod record; 2000.
[63] Chang YW, Lin CJ. Feature ranking using linear svm. WCCI causation and prediction [90] Efron B, Tibshirani RJ. An introduction to the bootstrap. CRC press; 1994.
challenge; 2008. [91] Anguita D, Boni A, Ridella S. Evaluating the generalization ability of support vector
[64] Yoon H, Yang K, Shahabi C. Feature subset selection and feature ranking for multi- machines through the bootstrap. Neural Process Lett 2000;11(1):51–8.
variate time series. IEEE Trans Knowl Data Eng 2005;17(9):1186–98. [92] Arlot S, Celisse A. A survey of cross-validation procedures for model selection. Stat
[65] Hong SJ. Use of contextual information for feature ranking and discretization. IEEE Surv 2010;4:40–79.
Trans Knowl Data Eng 1997;9(5):718–30. [93] Good P. Permutation tests: a practical guide to resampling methods for testing hy-
[66] Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagat- potheses. Springer Science & Business Media; 2013.
ing errors. Cognit Model 1988;5(3):1. [94] Chollet F. Keras. https://keras.io; 2015.
[67] Cybenko G. Approximation by superpositions of a sigmoidal function. Math Control [95] Liaw A, Wiener M. Classification and regression by randomforest. R News
Signals Syst (MCSS) 1989;2(4):303–14. 2002;2(3):18–22.
[68] Bishop CM. Neural networks for pattern recognition. Oxford University Press; 1995. [96] Zeileis A, Hornik K, Smola A, Karatzoglou A. Kernlab-an s4 package for kernel meth-
[69] Hinton GE, Osindero S, Teh YW. A fast learning algorithm for deep belief nets. Neural ods in r. J Stat Softw 2004;11(9):1–20.
Comput 2006;18(7):1527–54. [97] Meyer D., Dimitriadou E., Hornik K., Weingessel A., Leisch F. e1071: Misc functions
[70] Bengio Y. Learning deep architectures for ai. Found Trends® Mach Learn of the department of statistics, probability theory group (formerly: E1071), tu wien.
2009;2(1):1–127. R package; 2015. R package version 1.6–7.
[71] Bengio Y, Courville A, Vincent P. Representation learning: a review and new per-
spectives. IEEE Trans Pattern Anal Mach Intell 2013;35(8):1798–828.

23