This is the author's version of an article that has been published in this journal.

Changes were made to this version by the publisher prior to publication.

The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 1

A Hybrid Approach to Motion Prediction for Ship

Docking— Integration of a Neural Network Model
into the Ship Dynamic Model
Robert Skulstad, Guoyuan Li, Senior Member, IEEE, Thor I. Fossen, Fellow, IEEE, Bjørnar Vik,
and Houxiang Zhang, Senior Member, IEEE

Abstract—While automatic controllers are frequently used a vessel, including takeoff and landing of autonomous aerial
during transit operations and low-speed maneuvering of ships, vehicles and helicopters [1], crane operations [2] [3], and
ship operators typically perform docking maneuvers. This task missile launch [4]. These operations can be made safer and
is more or less challenging depending on factors such as local
environment disturbances, number of nearby vessels, and the more efficient by incorporating knowledge about future vessel
speed of the ship as it docks. This paper proposes a tool states.
for onboard support that offers position predictions based on Docking is the task of maneuvering the vessel to a fixed
an integration of a supervised machine learning (ML) model mooring location. On the path towards the dock the ves-
of the ship into the ship dynamic model. The ML model is
applied as a compensator of the unmodelled behaviour or sel operator must tackle challenges such as passing/nearby
inaccuracies from the dynamic model. The dynamic model vessels, compensating for forces induced on the vessel by
increases the amount of predetermined knowledge about how environment disturbances, and arriving at the dock location
the vessel is likely to move and thus reduces the black-box in a timely fashion. The latter is especially important for
factor typically experienced in purely data-driven predictors. A ferries or vessels transporting goods on a fixed route, where
prediction horizon of 30 seconds ahead of real time during
docking operations is examined. History data from the 29- keeping the time schedule is key. Although much effort has
meter coastal displacement ship RV (Research Vessel) Gunnerus been put into ship autonomy in recent years [5], docking is
is applied to validate the approach. Results show that the still a largely manual task performed by the vessel operator.
inclusion of the data-based ML model significantly improves Research in the field of ship motion prediction typically
the prediction accuracy. focuses on methods within one domain, e.g. dynamic- or
Index Terms—Ship motion prediction, supervised deep learn-
kinematic models and Kalman filters, machine learning (ML),
ing, onboard support deep learning or auto-regressive (AR) methods (see Section
II).
Dynamic models aim at describing the motion of the
I. I NTRODUCTION
vessel due to forces estimated by simplified representations
HIP motion prediction is a general term that incorporates
S many elements. These include the states in which to per-
form predictions – for example prediction of ship orientation,
of the vessel, including thruster effects and to some extent,
forces due to environmental disturbances. Simplifications
are necessary due to the lack of direct measurements of
position, or up/down motion – the temporal aspect (long, wave/current drift. Additionally, for docking applications,
medium and short) and the model that makes the prediction effects due to local wind fields, cushioning effects at the dock
of the states in the near future. Typically these predictions, and shallow water exist, which are not measured directly. The
which are based on time-series data, coincide with a specific true model is complex and nonlinear; thus a simplified model
application that could benefit from having information about is often used and discrepancies between the behaviour of the
future states of the vessel motion. Historically, research real ship and the dynamic model are expected. Kinematic
efforts have been focused on ship orientation and applications prediction models allow for translating motion measurements,
where safety or efficiency can be increased using predictions such as accelerations into predictions of position. However,
of these states. Mainly this is due to the abundance of they account for neither the effects of thruster commands nor
operations that are severely impacted by angular motions of the direct effect of wind forces. While many ML methods are
Robert Skulstad, Guoyuan Li and Houxiang Zhang are with the Depart-
well suited for representing nonlinear models they require
ment of Ocean Operations and Civil Engineering, Norwegian University of a substantial amount of sampled data to do so reliably. In
Science and Technology (NTNU), Aalesund, Norway addition, the inner connections in a ML model may not be
Thor I. Fossen is with the Department of Engineering Cybernetics, NTNU,
Trondheim, Norway
readily understandable.
Bjørnar Vik is with Kongsberg Maritime, Aalesund, Norway Examples where existing knowledge of the behaviour of

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 2

the ship is utilized in cooperation with data-based ML models A. Model-based motion prediction
are scarce. This study will therefore investigate the feasibility This section introduces predictors applying predetermined
of one such approach: making position predictions using a knowledge of how the vessel behaves when maneuvering.
dynamic model while in parallel, an ML model predicts the 1) Dynamic model: Triantafyllou et al. used a standard
position prediction error made by the dynamic model in order Kalman Filter (KF) to estimate and predict the motion states
to compensate for any unmodelled behaviour or inaccuracies. of the two decoupled motion groups heave-pitch and roll-
Including the measured wind velocity and direction as input sway-yaw [7]. They found that in order for the KF to
to the ML model, contributes significantly to the success of be successful, an accurate model of both vessel and sea
the proposed approach. While a Kalman filter could simulate state spectrum was required. For the latter requirement the
the dynamic model and derive predictions in a similar man- estimation of the modal frequency of the spectrum was key
ner, the ability to provide corrections to the dynamic model to the performance of the KF.
predictions, gained by learning from docking examples differ- Sutulo et al. [8] aimed at creating maneuvering models
ing in port location, sea state and wind conditions, would be (dynamic or kinematic) that could be inexpensive to evaluate,
lost. This work will focus on the prediction of position during and thus be used in tasks related to prediction and onboard
the docking operation of a regular displacement ship. Figure support. According to the authors this could make applica-
1 shows a picture of the ship. Currently, this is a manual task, tions such as model-based collision avoidance and onboard
relying on the ship operator to make appropriate and timely decision support for deciding control commands feasible due
corrections to actuators in order to safely dock the vessel. to the computational efficiency of the models.
During this operation the ship operator must make many 2) Motion density functions: Instead of using the equa-
choices due to changing environment factors, regulations tions of motions to model the dynamic behaviour of vessel
calling for proper interaction with nearby/crossing vessels, states, as Triantafyllou et al. had done, Sidar and Doolin
and the effects of applied actuator commands. To aid in constructed the linear KF using approximations of density
making these choices this paper proposes an onboard support functions of measured heave and pitch motions [9]. The
tool, which will provide the vessel operator with predictions density functions were obtained experimentally. This led to
of the vessel position. These predictions originate from the a KF of significantly lower dimension compared to the work
hybrid predictor and span 30 seconds into the future [6], of Triantafyllou et al. Measured heave and pitch time series
hereafter termed the prediction horizon. Key contributions of were assumed to be stationary, narrow band, and stochastic
this paper includes the construction of a hybrid model for for the duration of the prediction interval. The choice of a
prediction of the future motion of a ship, and the use of data KF as a tool for making predictions using the motion density
sampled onboard a coastal ship for training of the data-based functions was motivated by its ability to produce predictions
model as well as verification of the prediction performance. in real time.
The remainder of this paper is organized as follows. The approach to ship motion prediction taken by Nielsen
Section II presents previous research in this domain. Section et al. also relies on density functions of time series data [10].
III introduces the predictors and their architecture/parameters, By deriving the observed autocorrelation matrix for variables
Section IV gives results and describes the vessel and data largely dictated by the induced wave force, predictions of 15-
selected for training and testing of the predictors. Section V 60 seconds were made on a model-scale ship.
presents the conclusions. 3) Kinematic model: Perera et al. proposed to use the
Extended Kalman Filter (EKF) to estimate the translational
II. R ELATED WORK motion states and predict the trajectory of a vessel by means
of a curvilinear motion model (CMM) [11], [12]. States
Research on ship motion prediction generally revolves included in this model were heading angle, normal and
around a model that processes time series data, where each tangential accelerations, forward (surge) speed, and sideways
input channel contains data sampled at a fixed time interval. (sway) speed. By combining the EKF and the CMM the
Several metrics exist in which time series prediction models authors found that the estimated velocities and accelerations,
may be categorized. If the model uses existing explicit which were estimated based only on noisy position mea-
knowledge of how the vessel moves due to forces and/or surements, converged quickly (within 15 seconds) to small
velocities (dynamic/kinematic models), the term model-based variations around the true values. For the validity of the
predictors may be appropriate. If only sampled data is used to prediction they assumed constant accelerations, which is a
learn the behaviour of the vessel, the predictor is termed data- strong assumption given the nature of vessel motion. This was
based. We may also distinguish methods based on if they can acknowledged by the authors, deeming the approach valid
represent nonlinear behaviour. The subsections below outline only for short-term predictions.
the description and classification of existing methods for ship Perera later modified his approach to use a vector dot and
motion prediction. cross product algorithm for the prediction of vessel motion

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 3

Fig. 1. The Research Vessel Gunnerus of the Norwegian University of Science and Technology (NTNU) (bottom right vessel) approaching a dock in the
port of Aalesund, Norway.

[6]. Given the large inertia of a vessel, its trajectory creates this method. Ming et al. used SVM, together with several
a curve in the ocean plane, motivating the use of the CMM aiding methods, to predict the heave motion given waves
as vessel model in the EKF. Based on this property the impacting the vessel at four different directions [14].
algorithm calculates radii of the center of gravity and pivot The attributes of neural networks include the ability to
point of the vessel relative to a calculated center of planar adapt to input changes and to represent the nonlinear be-
motion for a vessel. The states and parameters related to the haviour of the input-output relation of physical systems.
vessel pivot point were estimated for a given time instance Employing a time-delay neural network with wavelet acti-
and used to predict the position and heading 30 seconds vation functions and using sensitivity analysis to determine
ahead of real time. The adaptability to varying conditions significant inputs, Zhang et al. performed prediction of the
is, according to the author, preserved by the use of the EKF heading of a vessel a few steps ahead [15]. They concluded
and predictions are valid under the assumption of constant that this type of prediction may be used for the benefit of
navigation conditions within a short future time interval. vessel control and safety.
Peng et al. applied data-based modelling to estimate the
unknown ship dynamics as well as to reconstruct the un-
B. Data-based motion prediction measured ship velocity. An Echo State Network [16] and a
To address the difficulty of obtaining a sufficiently de- fuzzy system [17] comprise the tools that was integrated into
tailed mathematical model, which transitions the relevant an observer and subsequently used in vessel maneuvering
states from one sampling time instance to the next, many control. The task of reconstructing the entire dynamic model
researchers have turned to data-based predictor models. The of the vessel was relaxed through the introduction of a
principal advantage of such methods is the ability to construct nominal mass matrix in [16]. Force produced by thrusters on
a model that relates a certain sampled input vector to a certain the vessel was assumed to be known and subsequently input
output state vector without knowledge of the parameters of to the data-based model to approximate the vessel dynamics.
the physical object. This output state vector is a set of vessel Zhang and Liu used a single layer feedforward network
states for which one wishes to determine numerical values (SLFN) to predict the heading angle of a vessel a few sample
ahead of real time. Support Vector Machines (SVM), neural intervals ahead [18]. This one-layer prediction network is
networks, and AR models are examples of such methods and common in the literature, although the choice of activation
the majority of data-based predictor models used for ship function, training method, number of hidden neurons, type
motion prediction are varieties of these general models. and number of input variables and the number of input lags
SVM features attributes such as strong generalization abil- vary greatly. Arriving at suitable values for these parameters
ity and global optimization [13]. Creating a model that is able is the key challenge to providing reliable predictions using
to generalize well to inputs, beyond those provided in the SLFNs. Often these parameters are derived using trial and er-
learning stage of the method, is one of the key advantages of ror, although online pruning methods for producing compact

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 4

networks exist [19]. For statically trained networks genetic outline how they are constructed and how they cooperate to
algorithms, grid search or random search algorithms may also predict the future ship motion.
be used to optimize architecture and hyperparameters.
Skulstad et al. applied a long-short term memory (LSTM) A. Hybrid predictor
network, a version of a recursive network, to maintain In order to utilize the two complementary predictors for
estimates of position and heading of a ship during loss creating a hybrid predictor, the vessel model predictor (see
of position reference signals from the Global Navigation Section III-B) will act as a foundation, predicting the com-
Satellite System [20]. They used a deep neural network plete position state due to the sampled data it receives. As the
similar to the one described in Section III-C. However, only relative water velocity is not measured onboard the vessel,
a one-step prediction was made and sensors not relying nor are the effects of the waves on the vessel motion directly
on external signals, such as the compass, wind sensor, and accounted for, a certain disagreement between the actual
sensors measuring operating conditions of the thrusters, were motion of the vessel and the position prediction output by the
still active. vessel model is expected. To compensate for the prediction
Maintaining accurate estimates of position and attitude errors made by the vessel model, the ML model is applied
during loss of GNSS signals is also of importance in the (see Section III-C). In this way, the hybrid prediction is the
automotive and aerospace domain. Examples of approaches sum of the prediction made by the vessel model and the ML
to mitigate such a sensor loss, through one-step predictions, model. Figure 2 shows how the two models are combined
are given in [21] and [22]. The former applies a KF in com- to create a predictor of the future ship position. The top
bination with an AR integrated moving average model and dashed box of Figure 2 shows the individual components of
a feedforward neural network to predict errors accumulating the vessel model. See Section III-B for a detailed description.
in the inertial navigation system while the latter makes use A fundamental difference between the vessel model and
of a radial basis function neural network for predicting the the ML model is the way they produce prediction output.
KF measurement update. While the ML model directly outputs predictions for the
The AR method makes use of history samples of the target entire prediction horizon (for future times th = [1 − 30]s)
state accompanied by predictor model parameters determined the vessel model requires iterations. Thus, during training of
by a least squares method [23]. It offers low computational the hybrid predictor, for each time instance in the input data
costs, but has drawbacks handling nonlinear, non-stationary the vessel model is iterated 30 times in order to produce
series [24]. Derivative methods to mitigate the effects of these targets for supervised training. This is illustrated by the
drawbacks exist, such as Nonlinear AR method and Time block named Actual ship position and the subtraction of
varying AR method. To predict the displacement of a landing the position predicted by the vessel model, η. During this
deck on a vessel, Yang improved upon the standard AR iterative process, external signals such as thruster RPM and
method by using Bayes Information Criterion to determine angle, wind speed and angle, and measured velocities are
the number of model coefficients, and a forgetting factor not updated as they are not known for future time instances.
to reduce the effect of older vessel states on the regression However, feedback loops are present inside the vessel model,
algorithm output [1]. causing dynamic behaviour within the prediction horizon in
Lately, studies on vessel traffic management have resulted terms of thruster forces and vessel velocities. Training the
in more emphasis on trajectory prediction in order to im- ML predictor involves using the position error targets and the
prove operational safety in congested waters [12] [25] [26]. associated input vector to get optimized hyperparameters that
However, these are longer-term predictions and fall outside reflect the dynamics of the error model (see Section III-C). As
the scope of this work. Similarly, prediction of a maximum there are no feedback loops between the targets and the input
envelope of roll, pitch, or heave motion, termed quiescent vector of the ML supervised learning approach, it is termed
period prediction (see [27] and [28]) is out of scope for the open-loop. Description of the variables included in the input
present study. vector may be found in Table III. The bottom dashed box
Prediction using time series methods similar to the ones of Figure 2 is repeated, applying identical hyperparameter
described above are also found in various other domains, such values, so as to create an ensemble of LSTM predictors (more
as: weather prediction [29], electrical load forecasting [30] on this in Section III-C1). To get hybrid position predictions
and automotive motion prediction [31]. during a docking operation, the sum of the vessel model
position prediction, η, and the LSTM model error prediction
is calculated.
III. P REDICTOR MODELLING
As the proposed method of this study is a combination of B. Vessel model
two predictor models originating from two separate fields; The vessel model uses established relations between ac-
model-based and data-based, the following sections will tuators, external environmental disturbances (wind) and the

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 5

Fig. 2. The prediction strategy showing the vessel model predictor (top dashed green box) and the ML model (bottom dashed green box).

motion of the hull through water to describe the forces acting speed and direction of the current is not measured.
on the hull through the maneuvering model of Fossen [32]. Therefore the velocity of the ship relative to the water,
The kinematic model is represented by ν r in (2), is not known. ν r is therefore
substituted by ν in (2).
η̇ = R(ψ)ν (1) • Simplification 2: Forces due to waves, given as τ wa
where η is the pose vector containing the positions and yaw in (2), are not accounted for. This is due to the lack of
angle. R(ψ) is the horizontal plane rotation matrix due to measurements of the wave state. Besides, ports provide
the yaw angle, ψ. ν is the velocity vector in surge, sway shelter from waves experienced in open ocean. There-
and yaw directions, respectively. Forces due to wind, waves, fore we do not include estimates of forces from waves
thrusters, hull friction, and inertia are given in (2) in the vessel model.

M RB ν̇ + C RB (ν)ν + M A ν̇ r + C A (ν r )ν r +
(2)
Dν r + D n (ν r )ν r = τ c + τ wi + τ wa A numerical model of the forces produced by the two main
azimuth thrusters was supplied by the thruster manufacturer.
where ν r =Tν − ν c is the relative ship velocity, and ν c = It is valid for all 4 quadrants of operation for the propeller
uc vc 0 is the current velocity. M RB is the rigid body (see Table I) and thus covers the key phases of the dock-
mass matrix, M A is the added mass matrix and C A and ing procedure of this study: the initial approach (transit),
C RB are matrices describing the Coriolis/centripetal forces. deceleration (windmilling) and low speed maneuvering. An
D and D n (ν r ) are linear and nonlinear damping matrices introduction to this type of propeller model is given in [33].
due to the hull moving through water. τ c , τ wi and τ wa are With regards to the force produced by the bow tunnel thruster,
forces on the ship due to thrusters, wind, and wave effects only nominal force is estimated through a thruster curve
respectively. At this point a few simplifications to the vessel provided by the thruster manufacturer.
model become relevant:
• Simplification 1: Forces due to current are not ac- To translate the propeller thrust into the three-dimensional
counted for in the vessel dynamic model (see (2)). The force, τ c , the azimuth angle and distance from the center of

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 6

TABLE I TABLE II
T HE 4 QUADRANTS OF PROPELLER OPERATION PARAMETERIZED BY P HYSICAL PARAMETERS OF THE VESSEL USED IN THE EXPERIMENT.
RPM AND INFLOW VELOCITY ( COURTESY OF [33]).
Parameter Description Value
Parameter 1st 2nd 3rd 4th m Mass of vessel 370 t
n ≥0 <0 <0 ≥0 DWT Deadweight 107 t
Va ≥0 ≥0 <0 <0 Lpp Length between perpendiculars 28.9 m
Bm Breadth middle (m) 9.6 m
dm Draught (m) 2.7 m
gravity of the vessel to each thruster is applied (see (3)).
 
0 c(αp ) c(αs )   designed to store data over an extended period of time,
T
1 s(αp ) s(αs )   tn 
 allowing it to capture the relatively slow changes observed in
τc =  × Tpa (3)
 lpx s(αp ) lsx s(αs )  data related to ship motion. Through the use of constant error
ltx Tsa
−lpy c(αp ) −lsy c(αs ) flow, embodied by the Constant Error Carousels (CECs) in
each LSTM block, and multiplicative gates that learn when
ltx , lpx and lsx are the distances along the longitudinal
to allow access to the CEC, events, or relations between
axis of the vessel from the vessel center of gravity to the
input- and output data, spaced by a significant time interval,
tunnel thruster, port main thruster and starboard main thruster
may be approximated [34]. In order to ensure satisfactory
respectively. αp is the azimuth angle of the port main thruster
performance of the LSTM in predicting future vessel states,
while αs is the azimuth angle of the starboard main thruster.
hyperparameters need to be set. This is done using the Matlab
The distance from the vessel center of gravity to each of
software, specifically the Bayesian optimization algorithm
the two main thrusters along the lateral axis of the vessel is
described in [35]. To limit the search space, and thus the
given by lpy and lsy . s(·) represents the sine function while
required computation time, three parameters were included
c(·) represents the cosine function. Forces produced by each
in the search:
thruster along the propeller axis are given by the variables
• Learning rate
Ttn , Tpa and Tsa for the tunnel thruster, port main thruster,
• Number of LSTM layers
and starboard main thruster, respectively. Only lateral force
• Number of blocks per layer
and torque about the up-down axis of the vessel is produced
by the bow tunnel thruster. 1) Ensembles: Due to randomness in the weight initializa-
Wind force is the only external disturbance in which we tion of the LSTM network, each instantiation of a network
use a deterministic model to estimate force. This is because with equal hyperparameters will output slightly different
the wind (velocity and direction) is the only one of the three predictions faced with the same input data. By averaging
environmental states measured. The three-dimensional force the output of several networks, using the same optimized
is given in (4). hyperparameters, the prediction error on previously unseen
  data can be reduced [36].
CX (γrw )AF w
1 2 
τ wi = ρa Vrw CY (γrw )ALw  (4) IV. E XPERIMENT
2
CN (γrw )ALw Loa
Table II shows the main physical dimensions of the RV
where ρa is the density of air, Vrw is the relative wind Gunnerus, a research vessel owned by the Norwegian Uni-
velocity, γrw is the relative wind angle, CX , CY and CN are versity of Science and Technology. In terms of propulsors,
wind coefficients specific for the hull/superstructure shape. two azimuth thrusters are mounted at the stern as well as
AF w and ALw are frontal and lateral projected areas and a bow tunnel thruster. The two azimuth thrusters are each
Loa is the overall length of the ship. driven by a 500 kW electric motor, while the electric motor
driving the bow thruster is rated at 200 kW. This yields a
C. Machine learning model cruising speed of about 10 knots.
Several choices exist when selecting a method for the ML
predictor. According to previous work in the domain of ship A. Data
motion prediction using ML (see Section II-B), SVMs, neural The experiment was conducted based on history data
networks (feedforward and recursive), and AR methods are acquired through log files created by a data acquisition
popular choices. We will apply an LSTM network, which has system onboard the RV Gunnerus. A one-year time period
shown outstanding performance in time-series modelling and was selected starting from August 2016 and ending in June
prediction. 2017. For all variables in the data set a sampling rate of 1
The sequential nature of time-series data related to motion Hz was observed.
of ships makes the LSTM a natural choice when searching In order to isolate successful dockings in the 2016-2017
for a representative model. This network type is specifically period, three sensor channels were used. Two Boolean signals

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 7

TABLE III
T HE VARIABLES USED IN THIS STUDY AS INPUT TO THE VESSEL MODEL
AND LSTM MODEL .

Range Range
Variable name Unit
(train) (test)
North m 1688/-2703 2267/-769
East m 1961/-486 1145/-1567
Heading angle deg 360/0 360/0
Surge speed knots 11.8/-1.82 11.98/-0.31
Sway speed knots 1.43/-1.43 1.05/-1.03
Heading rate deg/s 3.67/-3.43 2.84/-2.81
Roll deg 2.68/-3.9 2.99/-2.72
Pitch deg 0.53/-2.09 -0.02/-1.82
Heave m 0.14/-0.12 0.35/-0.41
Roll rate deg/s 2.22/-1.97 2.03/-2.17
Pitch rate deg/s 1.08/-1.28 0.89/-0.69
Heave rate m/s 0.17/-0.18 0.29/-0.25
Wind direction deg 360/0 360/0
Wind speed knots 19/0 15.6/0
Course deg 360/0 360/0
Total speed knots 11.8/0 12/0 Fig. 3. The various docking locations of the RV Gunnerus along the west
Port thruster RPM % 93.96/-67.64 99.19/-51.89 coast of Norway.
Port thruster angle deg 121.93/-90.33 156.04/-89.83
Starboard thruster RPM % 93.95/-67.58 100.08/-56.62
Starboard thruster angle deg 106.33/-117.33 90.33/-146.44 performed for convenience. The ECEF position recorded at
Tunnel thruster RPM % 102.1/-99.8 93/-61.2
the docking time instance was used as the origin for the
NED coordinates of each docking operation. Figure 4a and
4b shows the path taken by the vessel towards the docking
originating from the propulsion system, drive running (going location at coordinates (0,0) m. The former shows all paths
from true to false) and motor at zero speed (equals true), included as training instances for the ML algorithms, while
were applied in combination with a requirement of having the latter shows the test instances.
a total speed of less than 0.1 m/s. When the docking time
instances were successfully determined, 1000 samples prior B. Prediction for one docking approach
to these instances were extracted and made up the data set for
each docking operation. This interval may contain an initial As mentioned in the previous section, training data for the
period of automatic waypoint following control. However, ML predictor consisted of 68 individual docking operations.
the majority of the time is spent in the manual control mode, First running the vessel model predictor on each time instance
in which the ship operator guides the vessel to its docking (1000 instances per docking operation), predicting 30 seconds
location. Twenty-one sensor channels related to the motion ahead of real time, made it possible to generate an error signal
of the vessel were sampled (see the first column of Table III), by subtracting the vessel model position prediction from the
leading to a 1000x21 matrix of measurements per operation, actual position of the vessel. Thereby, the training targets,
spanning 15 locations along the west coast of Norway (see one vector with a 30-second prediction horizon per sampling
Figure 3). instance, for the supervised training of the LSTM networks
was created. Figure 5 shows the accuracy of the predictions
Table III gives all the input variables for the hybrid
in terms of average distance errors, calculated by (5), in the
predictor used in this study. Ranges are given as maximum
North-East plane for each docking operation in the training
and minimum values observed in the time series of each
data.
variable during 88 separate docking operations. Of these the  
first 68 were used for training and the last 20 were kept for M q
X
testing purposes. The unit deg is short for degrees. To get a ȳerr,i =  (Nij − N̂ij )2 + (Eij − Êij )2  /M (5)
clearer sense of the nature of each variable, and the extreme j=1

values observed in the training data compared to the testing ȳerr is the mean distance error between the predicted and the
data, the max/min values are given in columns 3 and 4 of true position of the ship in the prediction interval, M is the
Table III. number of samples per docking operation and i ∈ [1, 30] is
A further processing of the position of the ship was made the index of the prediction horizon, th . N and E represent
in order to generalize the position coordinates across docking the true north and east position, respectively, while N̂ and Ê
locations. A conversion from position given as latitude and are the predicted north and east positions.
longitude in the earth-centered, earth-fixed (ECEF) frame Given input data according to Section III-B at a certain
to the local north-east-down (NED) frame in meters was time instance, the vessel model predictor iteratively predicts

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 8

(a) Training

Fig. 5. The average distance between the true position and the position
estimated by the vessel model at 10, 20, and 30 seconds prediction horizons
on the training examples.

predictor has better position prediction accuracy. This is also

evident from the prediction error for the prediction horizon
end (distance between each green star and the adjacent red
star).
Both the vessel model predictions and the hybrid model
predictions diverge from the true position close to the origin.
This is attributed to the use of thrusters to temporarily push
the vessel against the dock while preparing the mooring
ropes. Figures 7 and 8 show that while the speed of the vessel
approaches zero at t ≈ 700s, indicating that the vessel has
docked, the thrusters are still producing thrust. In the same
(b) Testing time period the course angle of Figure 7 is invalid due to
Fig. 4. The North-East path taken by the RV Gunnerus for all docking zero speed. At t ≈ 630s the top plot of Figure 8 shows the
operations used in this study. port thruster being rotated. This is to push the vessel towards
the dock with the starboard side facing the dock.
The top plot of Figure 7 plots the course angle against
the position of the vessel 30 seconds ahead. In parallel, the the heading angle. For the final approach to the Trondheim
ML predictor offers predictions of the position prediction docking location, the course and heading angle deviate by
error made by the vessel model predictor. The sum of several degrees in the time period 300-500 s. This is due to
these two 30-element vectors constitute the end result of the the dock being located in the outlet of the Nidelva river, and
prediction approach: a set of coordinates given for future time the water flowing towards the sea induces force on the hull.
instances in the North-East plane. The effect on the vessel model prediction was a steady error
This is visualized in the plots of Figure 6, which shows of approximately 5 m in the North-East plane during this
the actual track of the docking approach and predictions time period.
made every 45 seconds. The red circle marks the start of
each prediction, while the green star marks the end for both
the hybrid predictor and the vessel model predictor. The red C. Average performance
star shows the true position at the prediction interval end. Figure 9 shows the performance of the hybrid predictor
If the red and green stars overlap, the position is predicted and the vessel model predictor. The number of samples
perfectly at 30 seconds ahead of real time. As the area for the averaging of the position prediction at future times
between the true track of the vessel (red solid line) and th = [1, 30]s includes the entire 1000 seconds prior to com-
the hybrid prediction vector (black dashed line) is smaller pleting the docking operation. Typically this involves a short
compared to the vessel model prediction vector, the hybrid initial period of transit speed, followed by deceleration to a

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 9

Fig. 7. The top plot shows the heading and course angle of the ship while
docking in Trondheim, while the bottom plot holds the ship speed.

Fig. 6. The prediction of ship position in the horizontal plane in the port
of Trondheim, Norway.

low-speed approach and finally gliding/decelerating towards

Fig. 8. The top plot shows the rotation angle of the main thrusters, while
close-to-zero speeds. This final period usually involves the the bottom plot shows the RPM percentage of all three thrusters.
application of the bow tunnel thruster (see the bottom plot
of Figure 8). Both the vessel model predictor and the LSTM
model predictor can introduce prediction error to the overall two of them are at a docking location not covered in the
hybrid predictor. However, the main sources of error come training data set. The third line is generated by predictions
from the vessel model and are due to the following reasons: carried out while docking in a port, which is covered only
• the vessel model predictor does not account for forces once in the training data set, and in an irregular fashion as
induced by current as they are not measured; well. It is irregular in the sense that the vessel did not follow
• change in the control commands input by the vessel the usual pattern of deceleration, but moved toward the dock
operator is unknown within each prediction interval and in lurches. The remaining black lines in Figure 9 depict the
therefore the initial value is applied. prediction errors incurred while docking at more frequently
The LSTM predictor assumes a portion of this error, which visited docking locations.
results in an improved overall position prediction, reducing Due to the nature of data-based models, where training
the average prediction error at th = 30s by almost half. data dictates the performance of the trained model, the more
While there are three docking operations that exhibit close repetitions of docking at a certain port will lead to the
to the same accuracy as the vessel model prediction average hybrid predictor providing better predictions at this location.
(three black lines close to the red dashed line of Figure 9), For the application described in this paper, if the trained

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 10

TABLE IV
T HE AVERAGE POSITION PREDICTION ERROR MADE BY THE VESSEL
MODEL PREDICTOR .

th [s] Training error [m] Testing error [m]

10 3.04 2.34
20 7.11 5.23
30 12.11 8.85

combined with a data-based predictor. The latter was im-

plemented using the LSTM neural network methodology.
This resulted in a hybrid predictor, where the data-based
LSTM corrected the predictions made by the vessel model.
A substantial increase in average accuracy was observed
throughout the prediction interval. At the maximum predic-
tion horizon of 30 seconds, the average distance error in the
position predictions was reduced by about 4 m, from 8.9 m
Fig. 9. The average position prediction error of the vessel model predictor (vessel model) to 4.7 m (hybrid model). Although the black-
by itself (dashed red line) and the hybrid predictor (dashed blue line) over box nature of the LSTM does not allow for direct insight
the 20 test sets and the prediction horizon (1-30 seconds). The solid black
lines represent the average position prediction error of the 20 individual into what causes the vessel model predictions to deviate,
docking operations included in the test set for the hybrid predictor. it compensates for the deviations, producing more accurate
predictions when both predictors are combined.
The current study applies prediction solely to provide
hybrid predictor was applied while docking the ship at a additional information for the ship operator while docking.
port for the first time, it would perform worse than if it Utilizing the proposed hybrid position prediction as input to
docked at its home port of Trondheim. However, by including an automatic motion controller could improve the efficiency
the vessel model predictor, which provides a deterministic and accuracy of autonomous docking operations. Providing a
evaluation of the position prediction, along with facilities in hybrid predictor that meets the two-sided goal of maintaining
the LSTM predictor to maintain its generalization abilities stability of the cascaded predictor-controller system, as well
(regularization, early stopping training and dropout layer), as to accurately predict the vessel dynamics, would be a key
an increase in position prediction accuracy is seen compared challenge. Along with the inclusion of wind predictions into
to the predictions made by the vessel model predictor alone. the hybrid predictor, this constitutes the direction of future
Downsides to this hybrid predictor include the requirement work.
of having both sufficient amount of data for training of the
data-based model as well as the parameters of the dynamic
ACKNOWLEDGEMENT
model described in Section III-B.
The second column of Table IV displays the averages of This work was supported in part by a grant from the
the values seen for th = [10, 20, 30]s in Figure 5. A lower av- Knowledge-Building Project for Industry “Digital Twins for
erage prediction error is observed for the testing data relative Vessel Life Cycle Service” (Project 280703) and in part by a
to the training data. This is due to the diversity of docking grant from the Research-Based Innovation “SFI Marine Op-
locations contained in each set. Out of 15 docking locations, eration in Virtual Environment” (Project 237929) in Norway.
four are represented in the test set, while 14 are represented The third author was partially funded by the Norwegian Re-
in the training set. Thus, as every docking location has its search Council (NTNU AMOS) at the Norwegian University
own set of challenges with respect to geographical layout and of Science and Technology (grant no. 223254).
environmental conditions, the ship operator needs to adapt
his or her docking strategy. This results in a larger spread R EFERENCES
in terms of thruster commands, which in turn affects the
[1] X. Yang, “Displacement motion prediction of a landing deck for
prediction accuracy of the vessel model. recovery operations of rotary UAVs,” Int. J. Control. Autom. Syst.,
vol. 11, no. 1, pp. 58–64, 2013.
[2] P. From, J. Gravdahl, and P. Abbeel, “On the influence of ship
V. C ONCLUSION motion prediction accuracy on motion planning and control of robotic
manipulators on seaborne platforms,” in Int. Conf. Robot. Autom., 2010,
Predicting the motion of a ship is complex. As a way to pp. 5281–5288.
[3] S. Küchler, T. Mahl, J. Neupert, K. Schneider, and O. Sawodny, “Active
reduce the uncertainty of the position predictor performance, control for an offshore crane using prediction of the vessels motion,”
predictions originating from the vessel dynamic model were IEEE/ASME Trans. Mechatronics, vol. 16, no. 2, pp. 297–309, 2011.

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].
 This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at http://dx.doi.org/10.1109/TIM.2020.3018568

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 11

[4] T. Praczyk, “Using evolutionary neural networks to predict spatial Motion Behaviors,” IEEE Trans. Intell. Transp. Syst., vol. 18, no. 11,
orientation of a ship,” Neurocomputing, vol. 166, pp. 229–243, 2015. pp. 3122–3134, 2017.
[5] Y. Shuai, G. Li, X. Cheng, R. Skulstad, J. Xu, H. Liu, and H. Zhang, [26] S. Gan, S. Liang, K. Li, J. Deng, and T. Cheng, “Long-Term Ship
“An efficient neural-network based approach to automatic ship dock- Speed Prediction for Intelligent Traffic Signaling,” IEEE Trans. Intell.
ing,” Ocean Eng., vol. 191, no. Available online, 2019. Transp. Syst., vol. 18, no. 1, pp. 82–91, 2017.
[6] L. P. Perera, “Navigation vector based ship maneuvering prediction,” [27] J. M. Giron-Sierra and S. Esteban, “The problem of quiescent period
Ocean Eng., vol. 138, pp. 151–160, 2017. prediction for ships: A review,” in IFAC Proc. Vol., vol. 43, no. 20.
[7] M. Triantafyllou, M. Bodson, and M. Athans, “Real time estimation IFAC, 2010, pp. 307–312.
of ship motions using Kalman filtering techniques,” IEEE J. Ocean. [28] J. G. Kusters, K. L. Cockrell, B. S. Connell, J. P. Rudzinsky, and
Eng., vol. OE-8, no. 1, pp. 9–20, 1983. V. J. Vinciullo, “FutureWavesTM : A real-time Ship Motion Forecasting
[8] S. Sutulo, L. Moreira, and C. Guedes Soares, “Mathematical models system employing advanced wave-sensing radar,” in Ocean. 2016
for ship path prediction in manoeuvring simulation systems,” Ocean MTS/IEEE Monterey, OCE 2016, 2016.
Eng., vol. 29, no. 1, pp. 1–19, 2001. [29] E. Soares, P. Costa, B. Costa, and D. Leite, “Ensemble of evolving data
clouds and fuzzy models for weather time series prediction,” Appl. Soft
[9] M. M. Sidar and B. F. Doolin, “On the Feasibility of Real-Time
Comput. J., vol. 64, pp. 445–453, 2018.
Prediction of Aircraft Carrier Motion at Sea,” IEEE Trans. Automat.
[30] N. Sapankevych and R. Sankar, “Time series prediction using support
Contr., vol. 28, no. 3, pp. 350–356, 1983.
vector machines: A survey,” IEEE Comput. Intell. Mag., vol. 4, no. 2,
[10] U. D. Nielsen, A. H. Brodtkorb, and J. J. Jensen, “Response predictions pp. 24–38, 2009.
using the observed autocorrelation function,” Mar. Struct., vol. 58, no. [31] S. Lefèvre, D. Vasquez, and C. Laugier, “A survey on motion pre-
September 2017, pp. 31–52, 2018. diction and risk assessment for intelligent vehicles,” ROBOMECH J.,
[11] L. P. Perera and C. Guedes Soares, “Ocean Vessel Trajectory Estima- vol. 1, no. 1, pp. 1–14, 2014.
tion and Prediction Based on Extended Kalman Filter,” in Adapt. 2010, [32] T. I. Fossen, Handbook of Marine Craft Hydrodynamics and Motion
Second Int. Conf. Adapt. Self-Adaptive Syst. Appl., 2010, pp. 14–20. Control. John Wiley and Sons Ltd., 2011.
[12] L. P. Perera, P. Oliveira, and C. Guedes Soares, “Maritime Traffic [33] O. N. Smogeli, “Control of Marine Propellers: From Normal to
Monitoring Based on Vessel Detection, Tracking, State Estimation, and Extreme Conditions,” Ph.D. dissertation, Norwegian University of
Trajectory Prediction,” IEEE Trans. Intell. Transp. Syst., vol. 13, no. 3, Science and Technology, 2006.
pp. 1188–1200, 2012. [34] J. Schmidhuber, “Deep Learning in neural networks: An overview,”
[13] Y. M. Yin, H. Y. Cui, M. Hong, and D. Y. Zhao, “Prediction of the Neural Networks, vol. 61, pp. 85–117, 2015.
vertical vibration of ship hull based on grey relational analysis and [35] J. Snoek, H. Larochelle, and R. Adams, “Practical Bayesian Optimiza-
SVM method,” J. Mar. Sci. Technol., vol. 20, no. 3, pp. 467–474, tion of Machine Learning Algorithms.” Adv. Neural Inf. Process. Syst.,
2014. pp. 2951–2959, 2012.
[14] M. W. Li, J. Geng, D. F. Han, and T. J. Zheng, “Ship motion prediction [36] S. G. Soares and R. Araújo, “An adaptive ensemble of on-line Extreme
using dynamic seasonal RvSVR with phase space reconstruction and Learning Machines with variable forgetting factor for dynamic system
the chaos adaptive efficient FOA,” Neurocomputing, vol. 174, pp. 661– prediction,” Neurocomputing, vol. 171, pp. 693–707, 2016.
680, 2016.
[15] W. Zhang and Z. Liu, “Real-time ship motion prediction based on time
delay wavelet neural network,” J. Appl. Math., vol. 2014, 2014.
[16] Z. Peng, J. Wang, and D. Wang, “Distributed Containment Maneu-
vering of Multiple Marine Vessels via Neurodynamics-Based Output
Feedback,” IEEE Trans. Ind. Electron., vol. 64, no. 5, pp. 3831–3839,
2017.
[17] ——, “Distributed Maneuvering of Autonomous Surface Vehicles
Based on Neurodynamic Optimization and Fuzzy Approximation,”
IEEE Trans. Control Syst. Technol., vol. 26, no. 3, pp. 1083–1090,
2018.
[18] J. Yin, W. Zhang, T. Li, and J. Hu, “Modified minimal resource
allocating network for ship motion predictive control,” in Proc. 2010
Int. Conf. Intell. Control Inf. Process. ICICIP 2010, no. PART 1, 2010,
pp. 231–235.
[19] J. C. Yin, Z. J. Zou, and F. Xu, “On-line prediction of ship roll motion
during maneuvering using sequential learning RBF neuralnetworks,”
Ocean Eng., vol. 61, pp. 139–147, 2013.
[20] R. Skulstad, G. Li, T. I. Fossen, B. Vik, and H. Zhang, “Dead Reck-
oning of Dynamically Positioned Ships: Using an Efficient Recurrent
Neural Network,” IEEE Robot. Autom. Mag., vol. 26, no. 3, pp. 39–51,
2019.
[21] Q. Xu, X. Li, and C.-y. Chan, “Enhancing Localization Accuracy
of MEMS-INS/GPS/In-Vehicle Sensors Integration During GPS Out-
ages,” IEEE Trans. Instrum. Meas., vol. 67, no. 8, pp. 1–13, 2018.
[22] L. Chen and J. Fang, “A hybrid prediction method for bridging GPS
outages in high-precision POS application,” IEEE Trans. Instrum.
Meas., vol. 63, no. 6, pp. 1656–1665, 2014.
[23] J. Ma, T. Li, and G. Li, “Comparison of Representative Method for
Time Series Prediction,” in 2006 Int. Conf. Mechatronics Autom., 2006,
pp. 2448–2453.
[24] W.-y. Duan, L.-m. Huang, Y. Han, and R. Wang, “IRF - AR Model
for Short-Term Prediction of Ship Motion,” in Proc. Twenty-fifth Int.
Ocean Polar Eng. Conf., 2015, pp. 59–66.
[25] Z. Xiao, L. Ponnambalam, X. Fu, and W. Zhang, “Maritime Traffic
Probabilistic Forecasting Based on Vessels’ Waterway Patterns and

Copyright (c) 2020 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].