A Simulation and Visualization Platform for Sailing Robots with

Application to Marine Area Scanning

2023 IEEE International Conference on Robotics and Biomimetics (ROBIO) | 979-8-3503-2570-6/23/$31.00 ©2023 IEEE | DOI: 10.1109/ROBIO58561.2023.10355044

Shuyi Lin1 , Na Zhang1 , Qinbo Sun1,2 , Weimin Qi1,2 and Huihuan Qian1,2,†

Abstract— This paper aims to develop a simulation and

visualization platform for sailing robots that enables real-time
simulation, based on Unreal Engine 5 (UE5), Matlab, and
MySQL database. This platform provides researchers who are
unfamiliar with UE5 with a convenient 3D sailing robot testing
platform by creating a bridge for data communication between
UE5 and other testing platforms using the MySQL database.
During the simulation process, the sailboat dynamic model
and control strategies are implemented in other platforms
like Matlab, Python, or C++, which has better ability in
programming and mathematic calculations. UE5 has a strong
physics engine for water environment simulation. A 3D sea
environment for sailboats is constructed using UE5. Another
platform like Matlab computes sailboat state based on the Fig. 1: Overview of the visualization and simulation plat-
sailboat model and control strategy, then transfer data to UE5 form. Other platforms on the left side refer to testing plat-
through MySQL database. The simulation in UE5 provides forms like Matlab, Python, or C++. We implement sailboat
a total of valuable 6 Dof data in total including sailboat 3 physics models and control strategies in this platform. On the
dimensional position and rotation data. With real-time sim-
ulation and visualization, this platform can be utilized for right side is the Unreal Engine 5 platform. A sea environment
testing autonomous sailboat navigation or control methods and for sailboats is implemented in UE5. The MySQL database
assisting remote sea experiments. is utilized for real-time data communication.

I. I NTRODUCTION
which makes the simulation platform crucial to autonomous
Simulation and visualization platform is important for
vehicle studies.
researchers in autonomous robots research areas. A robotic
simulation platform for autonomous vehicles has the physical Recent years, with the development of 3D gaming engine,
engine to simulate the required physical environment and more researchers are trying to construct simulation platform
the robot’s motion state in the environment using certain for USV, UAV, and UGV based on game engine, like
physics models. It requires accurate physical models and Unity, and Unreal Engine. For instance, AirSim is a high-
real-time rendering effects to simulate accurate robot motion fidelity simulator for autonomous arial vehicles constructed
and display the robot state intuitively. Researchers focus by Microsoft based on Unreal Engine [2]. This simulator
on optimizing the navigation and control methods of an supports both visual and physical simulation. Researchers in
autonomous vehicle, conduct experiments frequently for UAV areas have constructed simulation environment [3], [4],
testing, and use the experiment data to do reinforcement visualization environment, or Virtual Reality platform [5] for
learning to study better control methods. During the research doing virtual experiment and deep learning [5]. This proves
process, there is a strong need for a mature simulation that a mature simulation platform can boost the research
and visualization platform which helps with verifying the development in autonomous vehicles.
Guidance, Navigation, and Control (GNC) methods, and Specifically, in USV research areas, mature simulation
remote testing in real experiments. For theoretical studies environment is important but rare. Autonomous surface ve-
in GNC strategies of autonomous vehicles, researchers are hicles simulator requires high-fidelity visual and physical
building a simulation platform to help with the study. Re- simulation of the both water environment and wind en-
searchers carry out virtual testing on the platform to verify vironment. Water environment including waves, buoyancy
the control strategy and algorithm. Virtual testing increases feature, water current, and thruster under water [6]. Wind
testing efficiency and reduces the potential damage to the environment includes wind currents and thruster above water
testing vehicles compared with physical experiments [1], [6]. Game engine like Unreal Engine has strong function
of physical engine simulation and realistic rendering effect,
*This paper is supported by Project 2022A1515240063 from Guangdong which makes it a good choice to construct simulator for
Basic and Applied Basic Research Foundation, China. autonomous surface vehicles. Researchers have constructed
1 The Chinese University of Hong Kong, Shenzhen, Guangdong, China.
2 Shenzhen Institute of Artificial Intelligent and Robotics for Society, The
visualization environment and simulation environment for
Chinese University of Hong Kong, Shenzhen, Guangdong, China. USV. Some constructed a simple visualization and simulation
†Corresponding author is Huihuan Qian, [email protected] environment for their own experiment based on OpenGL

Authorized licensed use limited to: Kingston Univ Library. Downloaded on May 24,2024 at 20:48:46 UTC from IEEE Xplore. Restrictions apply.
[7] and Matlab [8], or Gazebo, Unity [9], and Unreal
Engine. These simulation and visualization platform can be
utilized for virtual experiment for control methods [6], and
reinforcement learning to test GNC strategy [5], [9].
In this paper, we presented a simulation and visualization
platform for sailing robots that enables real-time data com-
munication based on Unreal Engine, Matlab, and MySQL
database. This platform can be utilized as a simulation
platform to do virtual testing and help with reinforcement
learning algorithm, and a real-time visualization platform for
remote testing in real sea experiment. We also package the
platform into executable file that can run on other computers,
which enables researchers who are unfamiliar with UE5
development to utilize the platform easily. Fig. 3: The water environment and sailboat model in Unreal
The rest of the paper is structured as follows: Section 2 Engine 5.
introduces the architecture of our simulation platform. Sec-
tion 3 elaborates on the experiments. Section 4 discusses and
analyzes the experiment data. Finally, section 5 concludes the we can generate different sea environment for experiments.
overall work, discussing the contribution and future work. Specific variable values are chosen after multiple testing to
have both and stable. The details of environment construction
II. A RCHITECTURE in UE5 is further explained in [11].
This real-time simulation and visualization platform con-
B. Data Communication through MySQL Database
tains 3 parts. There is a Matlab end, an UE5 end, and a
MySQL database used as an intermediate way to communi- This section explains how data is communicated between
cate data in real-time. UE5 and the Matlab through MySQL database in the plat-
form.
In MySQL local database, a table called “state changing”
is created to temporarily store the data transferred from Mat-
lab to UE5 platform. Table “state changing” has 4 columns
and 1 row of data, containing the id and 3 DoF motion state
of a sailboat at current moment. The 3 DoF data are the
x and y position of the sailboat in Cartesian coordinates,
and the heading angle of the sailboat. Another table called
“data analysis” is created in database to store the simulation
Fig. 2: Data communication between platforms data exported from UE5 platform. When sailboat simulation
runs on UE5 platform, the 6 DoF data, including sailboat 3
dimensional position and rotation data, will be updated to
A. Environment Construction in UE5 “data analysis” table in database at a frequency of 100 Hz.
The platform constructed in UE5 contains 2 parts. The This table can be utilized for data analysis. UE5 connects to
testing vehicle, which is a sailboat 3D model constructed MySQL database to fetch and transfer data through a plugin
based on a real autonomous sailboat OceanVoy460, and the called MySQLIntegration. This plugin has multiple functions
water environment. that support UE5 platform to form connection with MySQL
We construct the 3D sailboat model based on the Ocean- database, fetch data or update data. As shown in Fig. 4, UE5
Voy460, which is a physical vessel that measures 4.6 m in utilizes the plugin to work in 3 steps, forming connection,
length, 2.2 m in width, and 7.3 m in height with sail and keel. fetching data, and updating sailboat state.
This autonomous sailboat has been constructed and used In Step 1, UE5 establishes connection to the local
for testing sailboat model and control strategy in real sea database using the “Create New Connection” function. This
experiment. After construction in SolidWorks and Blender is achieved by providing server information, database name,
platform, this 3D sailboat model is imported in the UE5 user ID, and passwords.
water environment as a testing vehicle. This 3D model has The second step is to fetch sailboat state data from local
physical properties including mass, gravity, buoyancy, linear database.
damping, and angular damping. After fetching data, Step 3 is to set sailboat state in
Water environment is constructed in UE5 using the Water real-time according to the fetched data. Function “Select
plugin. It generates physical waves based on Gerstner wave data from Query” in blueprint of step 2 triggered the event
model [10]. This model has a few variables, including “Event On Query Select Status Changed”. The selecting
number of waves, wave length, wave height, amplitude, wave result represents the current position and heading angle of
direction, and wave steepness. By changing the variables the sailboat, which is then fetched and set to be the new

Authorized licensed use limited to: Kingston Univ Library. Downloaded on May 24,2024 at 20:48:46 UTC from IEEE Xplore. Restrictions apply.
 UE Plugin:
MySQLIntegration
Unreal Engine Platform MySQL Database

UE connect to connects
1. UE 2. UE fetches
databaseto database data from database

3. UE updates
data in database

Fig. 4: Blueprint. UE5 connects to MySQL database through a UE5 plugin. This figure shows how UE5 utilize functions
in this plugin in blueprint logic to connect to database, fetches data, and updates data.

position and heading angle of the sailboat. Blueprint of step DoF data are transferred to UE5 platform through MySQL
3 presents the whole blueprint logic of how UE5 receive data database. Matlab also does data analysis. UE5 platform
transferred from Matlab through MySQL database. exports sailboat 6 DoF data during sailboat simulation into
For the other testing platform researchers utilize to do MySQL table, After simulation, Matlab uses the exported
simulation or testing control strategy, they have convenient data to do further data analysis.
ways to connect to MySQL database and fetch or update
III. M ARINE AREA SCANNING
data. For instance, Matlab supports connection to MySQL
database through Matlab program statements. Programming We implement sailboat physics model, and appropriate
languages like python or C++ also supports convenient control strategy in the simulation platform and do experiment
connection to MySQL database though built-in libraries. on utilizing autonomous sailboat to scan specific ocean area.

C. Utilizing Matlab to Implement Sailboat Model and Con- A. Sailboat Dynamic Model
trol Strategy The dynamic model and control strategy utilized in the
The other part of this simulation platform is a testing platform are based on the modeling and control strategy of
platform that researchers often choose in experiments for autonomous sailboat proposed in [12] and [13]. This model
simulation, testing deep learning algorithm, or data analyze. includes simple but essential variables for investigating sail-
These platform can be Matlab or programming language boat control strategy. The model is shown in the formula
like python or C++. Researchers are usually more familiar below.

with these platform, where they implement sailboat dynamic 
 ẋ = v cos(θ) + p1 atw cos (ψtw )
model and control strategy to do virtual testing. Comparing  = v sin(θ) + p1 atw sin (ψtw )
ẏ



to UE5 platform, they are more functional and flexible in θ̇ = w (1)
implementing physics model and math computations. gs sin(δs )−gr p11 sin(δr )−p2 v 2
v̇ =


 p9
By connecting this platform and UE5 together, researchers

ẇ = gs (p6 −p7 cos(δs ))−g r p8 cos(δr )−p3 wv


can utilize the simulation environment in UE5 platform to p10

help with their testing conveniently. In the sailboat dynamics model represented by (1), we
In our experiment, we utilize Matlab to implement sail- calculate the sailboat’s next motion state based on its current
boat model and control strategy. Matlab platform computes state and the environmental data. The sailboat state variables
sailboat next state data including x and y position in Carte- include its Cartesian coordinates (x and y), sailing speed (v),
sian Coordinate, and heading angle. After computing, the 3 direction (θ), angular velocity (w), angle of the main sail (δs ),

Authorized licensed use limited to: Kingston Univ Library. Downloaded on May 24,2024 at 20:48:46 UTC from IEEE Xplore. Restrictions apply.
and the angle of the rudder (δr ), force generated by main sail Therefore, we summarize 4 different cases when the heading
(gs ) and rudder (gr ). Environmental variables consist of the angle needs to be adjusted and implement the control strategy
true wind speed (atw ) and direction (ψtw ) relative to the in Matlab, presented in Fig. 5. Algorithm 1 shows how to
earth, and apparent wind speed (aaw ) and direction (ψaw ) reset the target heading angle θ̄ when encountering different
relative to the sailboat. Variables p1 to p11 in equations are cases and adjust the rudder and mainsail angle to reach the
coefficients. target heading angle.
By the first 4 lines in (1), we may compute the next x and
y position, heading angle, and sailboat speed. x0 , y0 , and w0 Algorithm 1 Function
represent current x and y position and heading angle, then if y <= 0 and sin(θ) < 0 then
we may compute the new state data x1 , y1 and w1 by (2), θ̄ = π/2 − sign(dir) ∗ err;
For instance, when dt = 0.1, we compute the next sailboat else if y > 100 and sin(θ) > 0 then
state 0.1s later. θ̄ = −π/2 + sign(dir) ∗ err;

 x1 = x0 + ẋ · δt end if

 y = y + ẏ · δt
1 0 if x > 100 then
(2) dir = −1; err = 0.05;
θ = θ 0 + θ̇ · δt
 1


v1 = v0 + v̇ · δt else if x < 0 then
dir = 1; err = 0.1;
The environmental variables, the wind speed and direction end if
can be set in the platform to create a specific wind environ- if cos(θ − θ̄) > 0 then
ment. In this experiment, we set the wind speed to be 3 m/s, δr = δrmax ∗ sin(θ − θ̄);
and wind angle to be 0.1 + π. else
δr = δrmax ∗ sign(sin(θ − θ̄));
end if
δs = abs(π/3 ∗ (cos(ψtw − θ) + 1)/2);

C. Matlab connects to MySQL database

With sailboat model and control strategy implemented,
Matlab can successfully compute sailboat next state data
and transfer the data to UE5 to drive the simulation through
connecting to Matlab.
Matlab compute the next sailboat state in frequency of
10 Hz and update the data in the database. Meanwhile, an
actor called MySQL component in Unreal Engine platform
is fetching the data and including X, Y, and Heading (the x
and y position of sailboat in Cartesian coordinates and the
Fig. 5: We conclude 4 different cases when heading angle θ heading angle of the sailboat) from database in frequency of
needs to be adjusted to a new target angle, represented as θ̄ 100 Hz. After fetching data every 0.01 s, sailboat position
in Algorithm 1. Encountering different cases, we reset the θ̄ and heading angle will be set according to the new state
into different value. If the sailboat is exceeding line y = 100 data. By assuring that the frequency of fetching data from
or y = 0 like in case a) and b), then we set variable dir to database in UE5 end is no slower than the updating of data
opposite value. in the Matlab end, we investigate a real-time data transfer
from MatLab to UE5.
IV. E XPERIMENT
B. Control Strategy for Marine Area Scanning and Matlab
In the experiment, we test a control strategy for ocean area
Scenario Implementation
scanning in this visualization and simulation platform.
The control strategy is designed specifically for marine
area scanning. In this experiment, we utilize the sailboat to A. Verifying Control Strategy for Ocean Area Scanning
scan a specific area of ocean, a square area with 100 m Fig. 6 presents data from one simulation result. Fig. 6(a)
side length. The goal is to control the sailboat to sail in shows the computed simulation route in Matlab. The red
“S” shape in a square area where x ∈ [0, 100], and y ∈ arrow shows the wind direction at every moment, and wind
[0, 100]. To scan and monitor the square area, sailboat starts speed is set to 3m/s. The blue arrow shows the sailboat
sailing at position (0,0), with heading angle θ = 0, pointing heading direction at every moment. The simulation initially
toward the positive x-axis. Sailboat remains its direction until starts at (0,0) position, heading toward positive x axis.
it exceeds the square boundary, which means that the heading Sailboat keeps its heading direction until it reaches line
angle θ remains the same until x ∈ / [0, 100] or y ∈/ [0, 100]. y = 100, y = 0, x = 100, or x = 0. From the figure we

Authorized licensed use limited to: Kingston Univ Library. Downloaded on May 24,2024 at 20:48:46 UTC from IEEE Xplore. Restrictions apply.
 y = 100, the target heading angle changes to π2 , heading to
the southeast. In this case, the sailboat is sailing against the
wind, and sailing speed increases.
B. Experiment 2 and 3: Change Sailboat Sailing Path Using
Control Strategy
We conduct two more experiments, in which we change
the value of variable err in control strategy. This variable
controls the target sailing angle which correspondingly ad-
justs sailing path. In the second experiment, we double the
(a) Path in Matlab (b) Sailboat path in UE5 value of variable err. In the third experiment, we triple the
value of variable err. The experiment result is presented in
Fig. 7, and Fig. 8.
Experiment1 : dir = 1, err = 0.1; dir = −1, err = 0.05;
Experiment2 : dir = 1, err = 0.2; dir = −1, err = 0.1;
Experiment3 : dir = 1, err = 0.3; dir = −1, err = 0.15;
In Algorithm 1, we observe that as the error variable err
(c) Heading angle computed in Matlab (d) Speed computed in Matlab
decreases, the target sailing angle approaches π2 , leading to
an increase in sailboat sailing time within the same area.
Fig. 6: This figure includes the Matlab computed path, In our experiment, the sailboat starts sailing from position
heading angle change and speed change in simulation as well (0,0) heading northwest. With different err value, sailing
as the simulation path in UE5. path varies.
In experiment 1, sailboat finish scanning the whole 100
m × 100 m target square area at timestamp 10000, which
is 1000 s. Sailboat average sailing speed is 1.32 m/s. In
intuitively see how sailboat keeps its heading direction when experiment 2, sailboat finish scanning target area at times-
sailboat position is in the 100 × 100 target scanning square tamp 5800, which is 580 s. In experiment 3, sailboat finish
area, and how sailboat tries to adjust its heading angle to turn scanning target area at timestamp 3500, which is 350 s. By
around when they reach the scanning area boundary. When setting different target heading angle, sailboat is controlled
the sailboat exceeds boundary x = 100, variable dir will be to sail in different path and finish scanning a whole area
set to -1 so that next target heading angle will be heading with different sailing time. This can be utilized in real
back and helping the sailboat goes back to the target scanning circumstances to set different scanning path according to
area. Matlab scanning path figure shown in Fig. 6(a) verify needs.
the correctness of control strategy for scanning specified
square marine area. Fig. 6(b) shows the simulation result in V. CONCLUSIONS
UE5. We see that this platform successfully connect Matlab In this paper, we present a visualization and simulation
and UE5 platform together to present a control strategy platform for autonomous sailboat base on UE5, Matlab, and
computed by Matlab on UE5 3D water environment. MySQL database. We construct water environment in UE5,
Fig. 6(c) and Fig. 6(d) present the changes in sailboat implement sailboat model and control strategy in Matlab.
heading angle and sailboat speed during the simulation. By connecting these two platform with MySQL database,
The x-axis of these two graphs is the timestamp. The time UE5 and Matlab can communicate data and drive sailboat
duration between two timestamps is 0.1 s. In this experiment, simulation successfully.
the sailboat finishes scanning the whole target area when the We test a marine area scanning control strategy using
timestamp is 10000, equal to 1000 s. These simulation data this platform. The experiment shows that this platform can
are computed in Matlab. From the graph of heading angle, we successfully simulate the sailboat sailing process. The sail-
can see that the sailboat has the same heading direction and boat sailing path computed in Matlab is the same with the
the same speed when the sailboat is sailing straight within sailboat simulation path in UE5. The control strategy can
the target square scanning area. Apparent wind velocity is successfully navigate the sailboat to sail in the target area.
different for the sailboat when it is heading toward different The sailboat’s scanning route can be adjusted by modifying
directions. For when the sailboat is heading in the northeast parameter values in the control strategy.
direction, the true wind direction is perpendicular to the This visualization and simulation platform can be utilized
target heading angle. From Fig. 6(a), we see that the control to facilitate researchers with conducting virtual testing on
strategy successfully keeps the sailing angle to the target different control strategy. The platform can work well with
heading angle. In this case, the speed of apparent wind is other platform through MySQL database since most testing
zero for the sailboat. After the sailboat first exceeds the line platform like Matlab, or programming language like python

Authorized licensed use limited to: Kingston Univ Library. Downloaded on May 24,2024 at 20:48:46 UTC from IEEE Xplore. Restrictions apply.
 (a) Path in Matlab (b) Sailboat path in UE5 (c) Sailboat path in UE5 (d) Headings of sailboat

Fig. 7: Simulation path in Matlab and UE5. err = 0.2.

(a) Path in Matlab (b) Sailboat path in UE5 (c) Speed of sailboat (d) Headings of sailboat

Fig. 8: Simulation path in Matlab and UE5. err = 0.3.

and c++ supports convenient connection to MySQL database. [4] C. Ma, Y. Zhou, and Z. Li, “A new simulation environment based on
What’s more, this platform can be utilized by researchers airsim, ros, and px4 for quadcopter aircrafts,” in 2020 6th International
Conference on Control, Automation and Robotics (ICCAR). IEEE,
who are unfamiliar with Unreal Engine. Using the packaging 2020, pp. 486–490.
method in UE5, this platform can be packaged as executable [5] S. Wang, J. Chen, Z. Zhang, G. Wang, Y. Tan, and Y. Zheng,
file. “Construction of a virtual reality platform for uav deep learning,” in
2017 Chinese Automation Congress (CAC). IEEE, 2017, pp. 3912–
Comparing to physical sea experiment in real world, this 3916.
simulation and visualization platform currently still has its [6] M. Paravisi, D. H. Santos, V. Jorge, G. Heck, L. M. Gonçalves,
and A. Amory, “Unmanned surface vehicle simulator with realistic
limitation in replicating complex and various water and wind environmental disturbances,” Sensors, vol. 19, no. 5, p. 1068, 2019.
environment. The platform’s consistent conditions make it [7] S. Schmitt, F. Le Bars, L. Jaulin, and T. Latzel, “Obstacle avoidance for
ideal for testing basic sailboat models and control strategies. an autonomous marine robot—a vector field approach,” Quantitative
Monitoring of the Underwater Environment: Results of the Interna-
Looking ahead, we aim to enhance and expand our platform tional Marine Science and Technology Event MOQESM´ 14 in Brest,
in several ways. We will focus on a more in-depth analysis France, pp. 119–131, 2016.
of exported 6 DoF data, particularly concerning roll and [8] W. Qi, Q. Sun, Y. Cao, and H. Qian, “Stable station keeping of au-
tonomous sailing robots via the switched systems approach for ocean
pitch, to validate sailing stability. Furthermore, we plan to observation,” in 2023 IEEE International Conference on Robotics and
incorporate various sailboat models and control strategies Automation (ICRA). IEEE, 2023, pp. 11 404–11 410.
into this visualization and simulation platform. Additionally, [9] R. P. Méxas, F. R. Leta, and E. W. G. Clua, “Comparison of rein-
forcement and imitation learning algorithms in autonomous sailboat
our future endeavors involve the utilization of Artificial digital twins,” IEEE Latin America Transactions, vol. 20, no. 9, pp.
Intelligence tools to further investigate sailboat models and 2153–2161, 2022.
control strategies based on the platform. [10] D. Henry, “On three-dimensional gerstner-like equatorial water
waves,” Philosophical Transactions of the Royal Society A: Mathe-
matical, Physical and Engineering Sciences, vol. 376, no. 2111, p.
R EFERENCES 20170088, 2018.
[11] N. Zhang, S. Lin, Q. Sun, W. Qi, and H. Qian, “A data-driven
visualization platform for sailing robots,” in 2023 IEEE International
[1] M. S. P. De Melo, J. G. da Silva Neto, P. J. L. Da Silva, J. M.
Conference on Real-time Computing and Robotics (RCAR), 2023, pp.
X. N. Teixeira, and V. Teichrieb, “Analysis and comparison of robotics
146–151.
3d simulators,” in 2019 21st Symposium on Virtual and Augmented
[12] J. Melin, K. Dahl, and M. Waller, “Modeling and control for an
Reality (SVR). IEEE, 2019, pp. 242–251.
autonomous sailboat: a case study,” in World Robotic Sailing champi-
[2] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity onship and International Robotic Sailing Conference. Springer, 2015,
visual and physical simulation for autonomous vehicles,” in Field pp. 137–149.
and Service Robotics: Results of the 11th International Conference. [13] L. Jaulin and F. Le Bars, “Sailboat as a windmill,” in Robotic sailing
Springer, 2018, pp. 621–635. 2013: proceedings of the 6th international robotic sailing conference.
[3] S. Nedevschi et al., “Employing simulators for collision-free au- Springer, 2013, pp. 81–92.
tonomous uav navigation,” in 2022 IEEE 18th International Confer-
ence on Intelligent Computer Communication and Processing (ICCP).
IEEE, 2022, pp. 313–318.

Authorized licensed use limited to: Kingston Univ Library. Downloaded on May 24,2024 at 20:48:46 UTC from IEEE Xplore. Restrictions apply.