sensors

Article
Improved Performance for PMSM Sensorless Control Based on
Robust-Type Controller, ESO-Type Observer, Multiple Neural
Networks, and RL-TD3 Agent †
Marcel Nicola 1,2, * , Claudiu-Ionel Nicola 1,2, * , Cosmin Ionete 2 , Dorin S, endrescu 2 and Monica Roman 2

1 Research and Development Department, National Institute for Research, Development and Testing in
Electrical Engineering—ICMET Craiova, 200746 Craiova, Romania
2 Department of Automatic Control and Electronics, University of Craiova, 200585 Craiova, Romania;
[email protected] (M.N.); [email protected] (C.-I.N.); [email protected] (C.I.);
[email protected] (D.S, .); [email protected] (M.R.)
* Correspondence: [email protected] (M.N.); [email protected] (C.-I.N.)
† This paper is an extension version of the conference paper: Nicola, M.; Nicola, C.-I.; Ionete, C.; S, endrescu, D.;
Roman, M. Improved Performance for PMSM Control Based on Robust Controller and Reinforcement
Learning. In Proceedings of the 2022 26th International Conference on System Theory, Control and
Computing (ICSTCC), Sinaia, Romania, 19–21 October 2022.

Abstract: This paper summarizes a robust controller based on the fact that, in the operation of a
permanent magnet synchronous motor (PMSM), a number of disturbance factors naturally occur,
among which both changes in internal parameters (e.g., stator resistance Rs and combined inertia
of rotor and load J) and changes in load torque TL can be mentioned. In this way, the performance
of the control system can be maintained over a relatively wide range of variation in the types of
parameters mentioned above. It also presents the synthesis of robust control, the implementation in
MATLAB/Simulink, and an improved version using a reinforcement learning twin-delayed deep
deterministic policy gradient (RL-TD3) agent, working in tandem with the robust controller to achieve
superior performance of the PMSM sensored control system. The comparison of the proposed control
Citation: Nicola, M.; Nicola, C.-I.;
Ionete, C.; S, endrescu, D.; Roman, M.
systems, in the case of sensored control versus the classical field oriented control (FOC) structure,
Improved Performance for PMSM based on classical PI-type controllers, is made both in terms of the usual response time and error
Sensorless Control Based on speed ripple, but also in terms of the fractal dimension (DF) of the rotor speed signal, by verifying
Robust-Type Controller, ESO-Type the hypothesis that the use of a more efficient control system results in a higher DF of the controlled
Observer, Multiple Neural Networks, variable. Starting from a basic structure of an ESO-type observer which, by its structure, allows the
and RL-TD3 Agent. Sensors 2023, 23, estimation of both the PMSM rotor speed and a term incorporating the disturbances on the system
5799. https://doi.org/10.3390/ (from which, in this case, an estimate of the PMSM load torque can be extracted), four variants of
s23135799
observers are proposed, obtained by combining the use of a multiple neural network (NN) load
Academic Editors: Marian Barbu torque observer and an RL-TD3 agent. The numerical simulations performed in MATLAB/Simulink
and Răzvan S, olea validate the superior performance obtained by using properly trained RL-TD3 agents, both in the
case of sensored and sensorless control.
Received: 5 May 2023
Revised: 14 June 2023
Keywords: PMSM; robust control; extended state observer; neural networks; reinforcement learning
Accepted: 19 June 2023
Published: 21 June 2023

1. Introduction
Copyright: © 2023 by the authors. In today’s world, the control of actuation systems based on servomotors has become
Licensee MDPI, Basel, Switzerland. a priority, due to the fact that these systems can be found in a wide range of products
This article is an open access article
that contribute both to improving the efficiency of certain activities and to improving
distributed under the terms and
ergonomics [1,2]. It is worth remembering that PMSM is incorporated into servomotors
conditions of the Creative Commons
used in robotics, computer peripherals, the aerospace industry, electric drives, etc. [3,4].
Attribution (CC BY) license (https://
Thus, PMSM control takes on new dimensions, in terms of the complex concerns of re-
creativecommons.org/licenses/by/
searchers to improve the performance of control systems. It is worth remembering that the
4.0/).

Sensors 2023, 23, 5799. https://doi.org/10.3390/s23135799 https://www.mdpi.com/journal/sensors

Sensors 2023, 23, 5799 2 of 29

FOC control strategy is essential for PMSM control, and ensures superior control perfor-
mance by organizing the control system into two cascaded levels, an inner level of current
control and an outer level of PMSM rotor control [5,6]. Typically, the controllers used within
the FOC control strategy structure are PI controllers [7,8]. The relatively good performance
of these types of controllers is well-known, but when significant parametric variations occur,
the performance of these types of control systems, based on PI-type controllers, decreases
significantly. A number of PMSM control systems have been developed and adapted,
including adaptive [9,10], optimal [11,12] and predictive [13,14], but also fuzzy [15,16] and
neuro-fuzzy control systems have been invented with good results [17,18]. A backstepping
controller for IM is developed in [19].
The sensorless control is also used in modern PMSM control systems to increase the
reliability and operational safety of the PMSM, by using PMSM rotor speed observers and
eliminating the speed sensor. Common observers include the Luenberger observer [20], the
sliding mode observer (SMO) [21], the model reference adaptive system (MRAS) [22] for
the deterministic case, and the Kalman filter [23] for the stochastic case.
Since there are significant variations in both the parameters describing the operation
of the PMSM and the load torque during normal operation of the PMSM, robust control
systems [24–28] may be an ideal choice of control system for the PMSM.
This paper is a continuation of the paper presented in [28] and focuses on extending
the performance of the sensored control system using a robust controller type, and also
in the case of the sensorless control system using an ESO-type observer [29]. This type of
observer is designed by exploiting the form of the differential equations describing the
PMSM. Thus, it is possible to obtain both PMSM rotor speed estimation and load torque
estimation, since the ESO-type observer incorporates the disturbances acting on the system
(e.g., load torque) in an additional state variable. This additional state variable is also
estimated by the ESO-type observer. In terms of methods to improve the PMSM rotor speed
estimation using the ESO-type observer structure, it is proposed to use this observer in
combination with a multiple NN [30,31] load torque observer and an RL-TD3 agent [32,33]
to obtain a better performance on rotor speed estimation compared to the measured rotor
speed provided by a speed sensor.
The paper presents the main control structures and observers of the PMSM, starting
with the differential equation description, the robust controller synthesis, and the ESO-type
observer synthesis that performs the PMSM rotor speed estimation. Improved variants
of using the ESO-type observer in tandem with the multiple NN load torque observer
and RL-TD3 agent are also presented. In addition, the control structures, the reward and
the training stage for the multiple NN load torque observer and RL-TD3 agent for speed
estimation are presented.
The main contributions of this article are:
• The synthesis of a robust PMSM controller using the d-q frame mathematical model
for the sensored case of the PMSM control structure;
• The robust controller synthesis using MATLAB and integration with PMSM sensored
control system;
• The improvement in the control performance of the PMSM sensored control system, by
combining the robust controller and an RL-TD3 agent that provides additional correc-
tion signals to adjust the ud and uq commands generated by the main robust controller;
• The comparative presentation of the performance of the proposed controllers used in
the PMSM control system structure, in terms of response time, speed ripple error and
DF of the rotor speed signal;
• The ESO-type observer synthesis for the sensorless case of the PMSM control system
for rotor speed estimation;
• The improvement in the performance of the basic ESO observer structure for PMSM
rotor speed estimation, by providing load torque estimates using two-layer feed-
forward multiple NN networks implemented with the Neural Net Fitting MATLAB
application [31];
Sensors 2023, 23, 5799 3 of 29

• The improvement in the performance of the ESO basic observer structure for PMSM
rotor speed estimation, by implementing an RL-TD3 agent [33] in MATLAB/Simulink
that provides correction signals to estimate the PMSM rotor speed value as close as
possible to the sensored rotor speed provided by a PMSM speed sensor;
• Comparison of the performance of the four observer variants for estimating the pro-
posed PMSM rotor speed, in terms of response time and speed ripple error.
The rest of the article is organized as follows: Section 2 presents the proposed sensored
and sensorless control structure of PMSM. The robust control of the PMSM is presented in
Section 3, while the MATLAB/Simulink implementation of the PMSM sensored control
system together with the improvement in the control performance by using an RL-TD3
agent is presented in Section 4. The PMSM sensorless control system using improved
ESO-type observer variants is presented in Section 5, and numerical simulations of the
operation of the PMSM sensorless control system are presented in Section 6. Section 7
presents some conclusions and suggestions for future approaches.

2. Sensored and Sensorless PMSM Control

Traditionally, PMSM control is achieved using the FOC strategy, in which a PMSM
rotor speed sensor is integrated and the name of the control structure is sensored control.
Figure 1 shows the schematic diagram of the FOC-type strategy for the PMSM control
system, where the controllers used in the current and voltage loops are PI-type.
Lq

did


 dt = − RLds id + Ld n p ωiq + 1
Ld ud
diq Ld
= − RLqs iq − 1
 λ0
Lq n p ωid − Lq n p ω + Lq uq

dt (1)
dω 3 np 1 B





 dt = λ0 i q + L d − L q i d i q −
2 J J TL − J ω
dθe

= npω

dt

In Relation (1), the notations are the usual ones in the d-q rotating reference frame [1,2,28].
Therefore, it can be denoted these notations as follows: the stator voltages, ud and uq , in
the d-q frame; the stator currents, id and iq , in the d-q frame; the stator inductances, Ld and
Lq , in the d-q frame; and the stator resistances, Rd and Rq , in the d-q frame. Other common
parameters of the PMSM include: the flux linkage is denoted by λ0 , the number of pole
pairs is denoted by np , the rotor moment inertia combined with load moment inertia is
denoted by J, the load torque is denoted by TL , the viscous friction coefficient is denoted by
B, and the PMSM rotor speed is noted with ω.
In short, the description of the nonlinear operating equations of the PMSM is given by
Relation (1).
In the usual case, where Ld = Lq is Rd = Rq = Rs , the system shown in (1) can be
rewritten in the following form [1,2,28]:
.   Rs
 
− i + n i 1
  
id L q d p ω q
L q
0   0
.   − RLqs iq − n p ω id − λL0q n p ω   u
+  0 L1q  d +  0 
 
 iq  =  (2)
 
.
  u q TL
ω 3 n p λ0
i − ω B 0 0 − J
2 J q J

In the following sections we propose both an improved control structure, in the case of
sensored control, and the use of ESO-type observer combinations, in the case of a structure
called sensorless control. Thus, Figure 2 shows the schematic diagram of the proposed
PMSM sensorless control system based on a robust controller and RL-TD3 agent algorithm,
using an improved ESO-type observer combined with a multiple NN load torque estimation
and an RL-TD3 agent for speed estimation.
In terms of improving control structures, a robust controller combined with an RL-
TD3 agent is presented, which provides superior performance compared to classical
PI controllers.
  R  1 
  s id  n p iq  L 0 
 
id   L q
  q   0 
    Rs 0   1  d 
u
 
iq     L iq  n p id  L n p     0 L  u    0 
Sensors 2023, 23, 5799 4 of 29
Sensors 2023, 23, x FOR PEER REVIEW 4 of(2)
31

   q q  q   L T
q
    
 3 n p 0 B   0 0   J
i  
Four combinations  of observers
2 RJ
q are
J proposed for PMSM speed estimation. The first
   1 which can estimate not only the
type of observer is based
 
on s
the i  n 
ESO-type i observer,
 L 0 
 L
d p q
 
PMSM rotor speedid   ω, but also the
q load torque T  
L , by adding
q
anothervariable
0  containing
   and  Rby  1  A 
u dsecond
iq  n p id  0 n p     0
disturbance elements s a special structure of the observer. proposed variant
    over
iqtrained       0 this
u provides  (2)
is a multiple NN, Lq different load
Lq torque ranges, L q  q   TL  value to an
which
  which,
 in turn, will have a better  PMSM rotor   
ESO-type observer
 3 n p 0 B   0 0  speed estimation
J 
relative
i  
to the measured PMSM rotor speedq (sensored control structure case). The following two
 2 J presented J above,   
variants are derived from those with the addition of an RL-TD3 agent
that provides superior estimates of rotor PMSM.

Figure 1. Schematic diagram of the FOC-type strategy for the PMSM control system based on
PI-type controllers.

In the following sections we propose both an improved control structure, in the case
of sensored control, and the use of ESO-type observer combinations, in the case of a
structure called sensorless control. Thus, Figure 2 shows the schematic diagram of the
proposed PMSM sensorless control system based on a robust controller and RL-TD3
Figurealgorithm,
agent 1. Schematic diagram
using of the FOC-type
anofimproved ESO-typestrategy for thecombined
PMSM control system based on
Figure 1. Schematic diagram the FOC-type strategyobserver
for the PMSM controlwith a multiple
system based onNN
PI-type
PI-type controllers.
load torque estimation and an RL-TD3 agent for speed estimation.
controllers.
In the following sections we propose both an improved control structure, in the case
of sensored control, and the use of ESO-type observer combinations, in the case of a
structure called sensorless control. Thus, Figure 2 shows the schematic diagram of the
proposed PMSM sensorless control system based on a robust controller and RL-TD3
agent algorithm, using an improved ESO-type observer combined with a multiple NN
load torque estimation and an RL-TD3 agent for speed estimation.

Figure 2. Schematic diagram of the proposed PMSM sensorless control system based on a robust
Figure 2. Schematic diagram of the proposed PMSM sensorless control system based on a robust
controller and an RL-TD3 agent algorithm, using an improved ESO-type observer combined with a
controller and an RL-TD3 agent algorithm, using an improved ESO-type observer combined with a
multiple NN load torque estimation and an RL-TD3 agent for speed estimation.
multiple NN load torque estimation and an RL-TD3 agent for speed estimation.

3. Robust Control of PMSM

Consider the operating point (x*, u*) obtained from the linearization of the nonlinear
PMSM
Figuremodel in Section
2. Schematic 2. Consider
diagram the operating
of the proposed point (x*,
PMSM sensorless u*) obtained
control by linearizing
system based on a robust the
controllerPMSM
nonlinear and an RL-TD3
model in agent algorithm,
Section using
2. The an improved
explanation of ESO-type observer
the operating combined
point with
is given bya the
multiple NN
following load torque
Relation estimation and an RL-TD3 agent for speed estimation.
(3) [28].
h iT h iT
x ∗ = id∗ iq∗ ω∗ ; u∗ = u∗d u∗q | TL (3)
Sensors 2023, 23, 5799 5 of 29

Around the operating point (x*, u*), the nonlinear equation of the PMSM takes the
following form:
f ( x ∗ , u∗ ) = 0 (4)
The following linearized system can be related to the nonlinear Equation (4):
.
x (t) = Ax (t) + Bw(t) + Du(t)
(5)
y(t) = Cx (t)

where the matrices are defined by Relation (6).

− RLqs ω∗ iq∗
  1 
Lq 0 0    
− n p λ0  1 0 0 0 0 0
− RLqs
 ∗  1
A= L q ; B =  0 0 ; C = ;D= (6)
 
ω Lq 0 0 1 0 0 0
0 3 n p λ0
− BJ 0 0 − 1J
2 J

Relation (7) shows the reference values selected for the operating point:

id∗
   
0
 iq∗  =  1  (7)
ω∗ 1200

For robust control of the PMSM, Figure 3 schematically shows the control structure
   
i i
of the PMSM, showing the reference r = dre f , the measured vector y = d , and the
ωre f ω
load torque TL , which acts as a disturbance variable on the PMSM control system. After
the synthesis of the robust control K, based on the synthesis of robust controllers, when the
Sensors 2023, 23, x FOR PEER REVIEW   are described in the d-q rotating reference frame, the values of the control
PMSM equations 6 of 31
ud
vector u = are obtained.
uq

Figure3.3.Schematic
Figure Schematicforfor
thethe PMSM
PMSM robust
robust control.
control.

Following
Following thethe steps
steps in synthesis
in the the synthesis of a robust
of a robust control algorithm
control algorithm [23],
[23], Figure Figure 4
4 shows
shows
the the generalized
generalized plant. Inplant. In addition
addition to inputs/outputs
to inputs/outputs and references,
and references, it contains
it contains the K the
K controller
controller structure,
structure, the the nominal
nominal plant
plant P, the
P, the uncertainties
uncertainties block
block ∆ and
∆ and the input
the input and and
output uncertainties w and z.
output uncertainties w and z.
The relation between these elements can be written as follows:

w = ∆·z (8)
   
ud r
u= =K (9)
uq y
where ∆ represents a diagonal matrix defining the uncertainties and u is the controller
feedback, i.e., the command given by the robust controller K.

Figure 4. Schematic for the generalized plant.

 Figure 3. Schematic for the PMSM robust control.

Following the steps in the synthesis of a robust control algorithm [23], Figure 4
shows the generalized plant. In addition to inputs/outputs and references, it contains the
Sensors 2023, 23, 5799 K controller structure, the nominal plant P, the uncertainties block ∆ and the input and6 of 29
output uncertainties w and z.

Sensors 2023, 23, x FOR PEER REVIEW 7 of 31

Typically, the sensitivity function is denoted by S, the complementary sensitivity

function is denoted by T, and the additive robustness matrix is denoted by R. Their ex-
pressions are given in (12)–(14) respectively.
1
S (12)
Figure4.4. Schematic
Figure Schematic for
for the
thegeneralized
generalizedplant.
plant. 1  Gnom K
The global
The relationtransfer
betweenmatrix
these elements
from r andcanTbeto wri
KG y−renisas follows:
given by Relation (10):
T w  L  z (13)
1  GnomK  r  (8)
y − r = Fu ( Fl ( P(s), K (s)), ∆) (10)
u d  K  r  TL
u R    K   (9)
(14)
where: the generalized plant is denoted by  u qP,
1the
GnomKy  controller is denoted by K, and the
robust
whereThe
∆ represents
upper/lower main a diagonal
fractional linear
objectives matrixcontrol
defining
transform
of robust arethe
is denoted uncertainties
tracking and u is
by Fu/lperformance
[24,28]. andthedisturbance
controller
feedback, i.e.,over
According
a enuation the command
to the entire given
[23], the by therange
generalized
frequency robustofcontroller
plant P verifies K. following
the
the controlled process.relation:
These objectives
can The global transfer
be summarized matrix in
as shown from r and (15).
Relation
 TL toThe
y−rweighting
is given
 bymatrices
Relationin(10):
Relation (15) are
z
usually chosen as follows: W1S for a low-pass-type wfilter, W2U for a high-pass-type filter,
Fuy−Fl r
y high-frequency
r P (=s ), (rs ),   r 
and the matrix W3T provides aPKenuation for the complementary sensi- (10) (11)
 TL  TL 
  
 r 
tivity function T [24,28].
where: the generalized plant is denoted yby P, the robust u controller is denoted by K, and
W1 S S
the upper/lower fractional linear transform is denoted by Fu/l [24,28].
z  PW
Tr plant 2U KS the system.
1following
Figure 5 shows the block diagram of the extended It also shows the weighting
(15)
According to [23], the generalized verifies relation:
function Wi (with i = 1, 2, 3) and the linearized nominal transfer function of the PMSM
W3T Tw 
given in (6) and denoted by Gnom .  z 
 y  r  
   P r  (11)
 r  TL 
   
 y  u 
Figure 5 shows the block diagram of the extended system. It also shows the
weighting function Wi (with i = 1, 2, 3) and the linearized nominal transfer function of the
PMSM given in (6) and denoted by Gnom.

Figure 5. Schematic of the extended system.

Figure 5. Schematic of the extended system.
4. MATLAB/Simulink Implementation
Typically, the sensitivity and Numerical
function is denoted Simulations
by S, the for PMSM
complementary sensitivity func-
Sensored Control System Using a Robust Controller Combined with RL-TD3 Agent
tion is denoted by T, and the additive robustness matrix is denoted by R. Their expressions
Based
are given on what respectively.
in (12)–(14) was described in the previous section on the synthesis of a robust
controller for the sensored control of PMSM, this section presents the MATLAB/Simulink
implementation of such a controller, while the 1second part of this section presents a way
S= (12)
to improve the performance of the robust controller
1 + Gnom Kusing an RL-TD3 agent.

4.1. Robust Controller Synthesis Based on MATLAB/Simulink

KG
T= (13)
The nominal parameters of the PMSM 1+ used inKthe following numerical simulations
Gnom
are shown in Table 1. Figure 6 shows the diagram of the MATLAB/Simulink model block
for the proposed PMSM sensored control system using a robust controller.
Sensors 2023, 23, 5799 7 of 29

K
R= (14)
1 + Gnom K
The main objectives of robust control are tracking performance and disturbance at-
tenuation over the entire frequency range of the controlled process. These objectives can
be summarized as shown in Relation (15). The weighting matrices in Relation (15) are
usually chosen as follows: W1S for a low-pass-type filter, W2U for a high-pass-type filter,
and the matrix W3T provides high-frequency attenuation for the complementary sensitivity
function T [24,28].
W1S S
k Tr→z k∞ = W2U KS < 1 (15)
W3T T

4. MATLAB/Simulink Implementation and Numerical Simulations for PMSM

Sensored Control System Using a Robust Controller Combined with RL-TD3 Agent
Based on what was described in the previous section on the synthesis of a robust
controller for the sensored control of PMSM, this section presents the MATLAB/Simulink
implementation of such a controller, while the second part of this section presents a way to
improve the performance of the robust controller using an RL-TD3 agent.
Sensors 2023, 23, x FOR PEER REVIEW 8 of 31
4.1. Robust Controller Synthesis Based on MATLAB/Simulink
The nominal parameters of the PMSM used in the following numerical simulations
are shown
Table in Table
1. PMSM nominal 1. Figure 6 shows
parameters usedthe diagram simulation.
in numerical of the MATLAB/Simulink model block
for the proposed PMSM sensored control system using a robust controller.
Parameter Value Unit
Stator resistance—Rs
Table 1. PMSM nominal parameters used in numerical simulation. 2.875 Ω
Inductances on d-q rotating reference Parameter frame—L d and Lq 0.0085 Value H Unit
Combined inertia
Stator resistance—Rs
of rotor and load—J 0.008 2.875
kg·m 2
Ω
Combined
Inductances viscous friction
on d-q rotating of rotor
reference andd load—B
frame—L and Lq 0.005 0.0085 N·m·s/rad
H
Flux induced
Combined byof the
inertia rotorpermanent
and load—J magnets of the rotor in 0.008 kg·m2
Combined viscous friction of rotor and load—B 0.175 0.005 Wb
N ·m·s/rad
the
Fluxstator phases—λ
induced 0
by the permanent magnets of the rotor in the stator phases—λ0 0.175 Wb
PMSM
PMSM PolePolepairs
pairs number—n
number—n P P 4 4 - -

Figure 6.
Figure 6. Diagram
Diagram of
of MATLAB/Simulink
MATLAB/Simulink model
model block
block for
for the
the proposed
proposed PMSM
PMSM sensored
sensored control
control
system using a robust controller.
system using a robust controller.

The implementation
The implementation of of the
the robust
robust controller
controller follows
follows the
the steps
steps described
described inin Section
Section 3,
3,
as well as a set of instructions specific to robust synthesis, used in the MATLAB Robust
as well as a set of instructions specific to robust synthesis, used in the MATLAB Robust
Control toolbox
Control toolboxenvironment
environment[25].
[25].Therefore,
Therefore,to to define
define a variation
a variation in some
in some parameters,
parameters, e.g.,
e.g., load torque T L, around nominal values, the command ureal()
load torque TL , around nominal values, the command ureal() is used. Using the is used. Using the
musyn()
musyn() command, the numerical values for the implementation of the robust controller
K(s) are obtained as shown in Relation (16).
  2279 40.38 154.7 62260 1965  62180 1960 75.58  4.052
  362.8 1126  4392 1969 62160 1967  0.00062 303.2  52.76

 
Sensors 2023, 23, 5799 8 of 29

command, the numerical values for the implementation of the robust controller K(s) are
obtained as shown in Relation (16).
 
−2279 40.38 154.7 62260 −1965 −62180 1960 75.58 −4.052

 −362.8 −1126 −4392 1969 62160 −1967 −0.00062 303.2 −52.76 


 1037 131.3 −648.5 0.00006 −0.01642 −0.00006 0.01639 −1037 638.2 

" #  −5.336 −0.0000002 −0.02882 −0.713 −0.000004 0.2849 0.000004 −2.664 0.02882 
AK BK
 
K (s) (s) =  −94.65 −0.000007 −0.7645 −0.000004 −0.7119 0.000004 0.2838 94.65 −7.236 (s) (16)
 
CK DK
5.362 0 0.02779 0 0 −1 0 2.638 −02779
 
 
93.5 0 0.8716 0 0 0 −1 −93.5 7.128
 
 
 
 −15.88 −0.5068 1.272 529.2 −16.7 528.6 16.66 0 0 
−0.5068 −6.669 −37.08 16.74 528.4 −16.72 −527 0 0

Furthermore, the controller K synthesized in Relation (16) and partitioned as shown in

Relation (17) is also implemented in Simulink, as shown in Figure 6 [26,28].
 
K11 (s) K12 (s)
K (s) = (17)
K21 (s) K22 (s)

where each component of the controller K(s) can be written as a transfer function: num(s)/den(s):

N11 (s) N (s) N (s) N22 (s)

K11 (s) = , K12 (s) = 12 , K21 (s) = 21 , K22 (s) = (18)
D (s) D (s) D (s) D (s)

The numerical values of the numerators and denominators of the components of the
robust controller K(s), namely, N11 (s), N12 (s), N21 (s), N22 (s), and D(s) are as follows [26,28]:

N11 (s) = 2 · 10−5 · s6 + 2 · 10−5 · s5 + 4 · 10−4 · s4 + 0.0312 · s3 + 0.0532 · s2 + 0.0201 · s + 0.0122 (19)

N12 (s) = 8 · 10−5 · s6 + 7 · 10−5 · s5 + 5 · 10−5 · s4 + 0.0422 · s3 + 0.0976 · s2 + 0.0143 · s + 0.0211 (20)

N21 (s) = 5 · 10−5 · s6 + 6 · 10−5 · s5 + 7 · 10−5 · s4 + 0.0235 · s3 + 0.0232 · s2 + 0.0089 · s + 0.0678 (21)

N22 (s) = 2 · 10−4 · s6 + 3 · 10−4 · s5 + 0.005 · s4 + 0.351 · s3 + 0.6232 · s2 + 0.3015 · s + 0.0242 (22)

D (s) = 3 · 10−4 · s7 + 2 · 10−4 · s6 + 9 · 10−4 · s5 + 0.0196 · s4 + 1.4102 · s3 + 2.5062 · s2 + 1.2128 · s + 0.0973 (23)
The following are the general rules for the selection of the weighting matrices in
Relation (15) that characterize the robust control [26,28]:
 0.0036·s+2.1522 
s 0
W1S (s) = 0.0036·s+2.1522 (24)
0 s
 
0.01 0
W1U (s) = (25)
0 0.01
" #
0.0033·s+1
0.0053·s+0.75 0
W1T (s) = 0.0033·s+1 (26)
0 0.0053·s+0.75

Bode plots can be drawn to highlight the robustness properties of the synthesized
controller. Thus, Figure 7a shows Bode plots for the transfer function ω ref to z1 , while
Figure 7b shows Bode plots for the transfer function TL to z1 . From the analysis of these
Bode plots, it can be seen that the robust control system of the PMSM ensures the stability
of the closed-loop system, even in the case of parametric variations.
Sensors 2023, 23, 5799 9 of 29

Sensors 2023, 23, x FOR PEER REVIEW 10 of 31

(b)

Figure
Figure 7. Graphical
7. Graphical representationfor
representation forBode
Bode plots:
plots: (a)
(a) the
the transfer
transferfunction
functionfrom
fromωrefωto z1; and (b)
ref to z1 ; and (b) the
the transfer function from TL to z1.
transfer function from TL to z1 .
Figure 8 shows the evolution of the parameters of interest of the PMSM sensored
Figure 8 shows the evolution of the parameters of interest of the PMSM sensored
control system using the classical FOC control structure with PI-type controllers, namely:
control system using the classical FOC control structure with PI-type controllers, namely:
rotor speed—ω; electromagnetic torque—Te; load torque—TL; stator currents—ia, ib, and
rotor speed—ω;
ic; and electromagnetic
d-q rotating reference frametorque—T e ; load
id and iq. The torque—T
numerical L ; stator
simulations arecurrents—i a , ib , and
performed by
ic ; and d-q rotating
applying a sequencereference frame to
of step signals id the iq . The
andPMSM numerical
rotor simulations
speed reference are performed
as follows: ωref =
by [1000, 1250,a1500,
applying 900] rpm,
sequence together
of step with to
signals a load
the torque
PMSMTLrotor with aspeed
value reference
of de 0.5 Nm. It
as follows:
ω refcan be seen
= [1000, that 1500,
1250, the response
900] rpm,time of the control
together with asystem is 29 ms,
load torque TL and
withthere is no
a value ofover-
de 0.5 Nm.
shoot.
It can be seen that the response time of the control system is 29 ms, and there is no overshoot.

Figure 8. Time evolution parameters of the PMSM sensored control system using a PI controller for
Figure 8. Time evolution parameters of the PMSM sensored control system using a PI controller for
rotor speed reference ωref = [1000, 1250, 1500, 900] rpm and load torque TL = 0.5 Nm.
rotor speed reference ω ref = [1000, 1250, 1500, 900] rpm and load torque TL = 0.5 Nm.
Figure 9 shows the numerical simulations of the PMSM sensored control system,
when both the classical PI-type controller (FOC-type strategy) and the robust-type con-
troller are used. Thus, numerical simulations are performed after applying a sequence of
step signals to the PMSM rotor speed reference as follows: ωref = [1000, 1200, 1400, 1000]
rpm, together with a load torque TL with a value of 0.5 Nm. For the PMSM control, be er
Sensors 2023, 23, 5799 10 of 29

Figure 9 shows the numerical simulations of the PMSM sensored control system, when
both the classical PI-type controller (FOC-type strategy) and the robust-type controller are
used. Thus, numerical simulations are performed after applying a sequence of step signals
to the PMSM rotor speed reference as follows: ω ref = [1000, 1200, 1400, 1000] rpm, together
Sensors 2023, 23, x FOR PEER REVIEW
with a load torque TL with a value of 0.5 Nm. For the PMSM control, better 11 of 31
behavior of the
robust controller is observed compared to the PI-based controller.
Sensors 2023, 23, x FOR PEER REVIEW 11 of 31

Figure
Figure 9. Comparison
9. Comparison of of
thethetime
timeevolution
evolution of
ofthe
therotor speed
rotor for for
speed the PMSM sensored
the PMSM controlcontrol
sensored sys- system
tem using the PI controller and the robust controller, for a rotor speed reference ωref = [1000, 1200,
using1400,
the PI controller
1000]
Figure
and
rpm and load
9. Comparison
the
torque
of the
robust controller,
timeTLevolution
for a rotor speed reference
= 0.5 Nm.of the rotor speed for the PMSM sensoredω = [1000, 1200,
refcontrol sys- 1400,
1000] tem
rpmusing
andthe PI torque
load controllerTLand theNm.
= 0.5 robust controller, for a rotor speed reference ωref = [1000, 1200,
1400,However, if there
1000] rpm and load is a 50%
torque TLvariation
= 0.5 Nm. in the load torque TL and a 50% variation in the
combined
However, rotor and load
if there is ainertia
50% Jvariation
parameter,in Figure 10 shows
the load a comparison
torque TL and aof50% the per-
variation in
However,
formance
the combined theifrobust-type
ofrotor there
andisloada 50% variation
controller
inertia in the
Jand the PI-type
load torque
parameter, TL and
controller.
Figure a can
50%be
10 Itshows variation
aseen in the
that
comparisonthe of the
combined
robust rotor and
controller has load
the inertia
same J parameter,
performance as Figurethe
when 10 parameters
shows a comparison
have nominalof the per-
values
performance of the robust-type controller and the PI-type controller. It can be seen that the
formance
(see Figureof9),
thewhile
robust-type controller provides
the PI controller and the PI-type controller.
a response It canthan
time more be seen
50%that the
higher
robust controller
robust controllerhashasthe
the same
same performance
performance as as
whenwhen the parameters
the parameters have nominal
have nominal values values
than when the PMSM parameters have nominal values.
(see Figure 9), while
(see Figure 9), whilethethePI PI
controller
controllerprovides
provides a a response timemore
response time more than
than 50%50% higher than
higher
whenthan
thewhen
PMSM the parameters
PMSM parameters havehave nominal
nominal values.
values.

Figure 10. Comparison of the time evolution of the rotor speed for the PMSM sensored control
system using the PI controller and the robust controller, for a rotor speed reference ωref = [1000,
1300,
Figure1700,
10. 900] rpm andof
Comparison load
thetorque TL = 1 Nm.of the rotor speed for the PMSM sensored control
time evolution
Figure 10. Comparison of the time evolution of the rotor speed for the PMSM sensored control system
system using the PI controller and the robust controller, for a rotor speed reference ωref = [1000,
using1300, 1700,
the PI 900] rpm and
controller andload
the torque
robustTcontroller,
L = 1 Nm. for a rotor speed reference ω ref = [1000, 1300, 1700,
900] rpm and load torque TL = 1 Nm.
 Sensors 2023, 23, x FOR PEER REVIEW 12 of 31

Sensors 2023, 23, 5799 11 of 29

4.2. Improvement in the Robust Control of PMSM Sensored Control System Using RL-TD3
Agent
4.2. Improvement in the Robust Control of PMSM Sensored Control System Using RL-TD3 Agent
In order
In order toto improve
improve the the performance
performance of of the
the robust
robust controller
controller according
accordingto to[28],
[28],an
an
RL-TD3 agent is used which, after the training phase, will provide correction signalsover
RL-TD3 agent is used which, after the training phase, will provide correction signals over
the uud and
the and uuqq control
control signals
signals of
of the
the robust
robustcontroller,
controller, so
so that
thatthe
thecontrol
controlsystem
systemconsisting
consisting
d
of the robust controller plus the RL-TD3 agent will provide superior performancefor
of the robust controller plus the RL-TD3 agent will provide superior performance forthe
the
PMSM control. The proposed block diagram of the implementation in Simulink ofthe
PMSM control. The proposed block diagram of the implementation in Simulink of the
robust controller
robust controller in in tandem
tandem with
with the
the RL-TD3
RL-TD3 agent
agent for
for the
thesensored
sensoredcontrol
controlof
ofaaPMSM
PMSMisis
shown in
shown in Figure
Figure 11. 11.

Figure 11.
Figure 11.Block
Blockdiagram
diagramofofthe
the MATLAB/Simulink
MATLAB/Simulink implementation
implementation of of
thethe proposed
proposed PMSM
PMSM sen-
sensored
sored control system using a robust controller plus RL-TD3 agent.
control system using a robust controller plus RL-TD3 agent.

Using the
Using theelements
elements andand structure
structure fromfrom the MATLAB
the MATLAB Reinforcement
Reinforcement LearningLearning
toolbox,
toolbox,
Figure 12 Figure 12 shows
shows the RL-TD3the RL-TD3
agent agent implementation
implementation subsystem.
subsystem. Note that theNote that
inputs to the
this
inputs to this subsystem, viz.: PMSM rotor speed ω, iq current, iqerror current
subsystem, viz.: PMSM rotor speed ω, iq current, iqerror current error and speed error ω error , error and
speed
are error ωerrorfor
observations , are observations
this for this while
type of algorithm, type ofthealgorithm,
outputs are while the outputs
the correction are the
signals for
correction
the ud and signals for the
uq control ud and
signals and uqare
control signals actions.
considered and are considered
The generalized actions. The gener-
reward plays
alized
the rolereward playscriterion
of a global the role for
of aoptimizing
global criterion for optimizing
the performance the control
of the performance of the
system. The
control system.
expression of theThe expression
reward is givenof by
thethe
reward is given
following by the[32,33]:
relation following relation [32,33]:

 
 2
25iqerror  5  
 j 22
2 error  0.1 ut j1 
2
!
R RLTD3 
R RL−TD3 = − 5iqerror + 5ωerror + 0.1∑
(27)
 j
j u t −1  (27)
j
Note that in Relation (27), the term ut 1 includes the actions generated in the pre-
j
vious step.that
Note Thein training
Relation performance
(27), the term for
ut−200 epochsthe
1 includes of actions
the RL-TD3 agentinisthe
generated shown in
previous
Figure 13.
step. The training performance for 200 epochs of the RL-TD3 agent is shown in Figure 13.
The following is a series of numerical simulations performed in MATLAB/Simulink
for a PMSM with the nominal values given in Table 1 on the performance of the robust
control system, but also with the enhancements of the RL-TD3 agent, where the variables
of interest are those presented in Section 4.1. Therefore, Figure 14 shows the performance
of the PMSM sensored robust control system for rotor speed reference ω ref = [1000, 1250,
1500, 900] rpm and load torque TL = 0.5 Nm, while Figure 15 shows the performance of the
PMSM sensored robust control system plus RL-TD3 agent.
Figure 16 shows the results of the performance of the robust control system plus
RL-TD3 agent for PMSM control, for a 100% increase in the nominal value of the load
torque and a 50% increase in the combined rotor and load inertia J parameter. It can be
Sensors 2023, 23, 5799 12 of 29

Sensors 2023, 23, x FOR PEER REVIEW 13 of 31

seen that the control system maintains its performance, but with a corresponding increase
Sensors 2023, 23, x FOR PEER REVIEW
in currents. 13 of 31

Figure
Figure 12.12.
Figure
12. MATLAB/Simulink
MATLAB/Simulinksubsystem
MATLAB/Simulink subsystem
subsystemimplementation
implementation
implementationof the
of the RL-TD3
RL-TD3
of the agent
agent
RL-TD3 forimprovement
for
agent improvementin in in
for improvement
robust controller
robust u
controller d and uq command.
robust controller uduandd anduuqcommand.
q command.

Sensors 2023, 23, x FOR PEER REVIEW 14 of 31

Figure
Figure 13.13. RL-TD3
RL-TD3
RL-TD3 agentperformance
agent
agent performanceevolution
performance evolution for
evolutionfor udduand
u
for and uuqq command
command improvement.
improvement.
uq command
Figure 13. d and improvement.

The following is a series of numerical simulations performed in MATLAB/Simulink

The following is a series of numerical simulations performed in MATLAB/Simulink
for a PMSM with the nominal values given in Table 1 on the performance of the robust
for a PMSM with the nominal values given in Table 1 on the performance of the robust
control system, but also with the enhancements of the RL-TD3 agent, where the variables
control system, but also with the enhancements of the RL-TD3 agent, where the variables
of interest are those presented in Section 4.1. Therefore, Figure 14 shows the performance
of of
interest are those presented in Section 4.1. Therefore, Figure 14 shows the performance
the PMSM sensored robust control system for rotor speed reference ωref = [1000, 1250,
of 1500,
the PMSM sensored
900] rpm and loadrobust control
torque system
TL = 0.5 Nm, for rotor
while speed
Figure reference
15 shows the ω ref = [1000, 1250,
performance of
1500, 900] rpm and load torque T L = 0.5 Nm, while Figure 15 shows the performance of
the PMSM sensored robust control system plus RL-TD3 agent.
the PMSM sensored
Figure 16 shows robust
the control
results ofsystem plus RL-TD3
the performance ofagent.
the robust control system plus
RL-TD3 agent for PMSM control, for a 100% increase of
Figure 16 shows the results of the performance in the
the robust
nominalcontrol
value of system plus
the load
RL-TD3 agent for PMSM control, for a 100% increase in the nominal value
torque and a 50% increase in the combined rotor and load inertia J parameter. It can be of the load
torque
seen and
that athe
50% increase
control in the
system combined
maintains its rotor and loadbut
performance, inertia
with Japarameter.
corresponding It can
in-be
seen that the control
crease in currents. system maintains its performance, but with a corresponding in-
crease in currents.

Figure 14. Time evolution parameters of the PMSM sensored control system using robust control-
Figure 14. Time evolution parameters of the PMSM sensored control system using robust controller,
ler, for a rotor speed reference ωref = [1000, 1250, 1500, 900] rpm and load torque TL = 0.5 Nm.
for a rotor speed reference ω ref = [1000, 1250, 1500, 900] rpm and load torque TL = 0.5 Nm.
Sensors 2023, 23, 5799 13 of 29
Figure 14. Time evolution parameters of the PMSM sensored control system using robust control-
ler, for a rotor speed reference ωref = [1000, 1250, 1500, 900] rpm and load torque TL = 0.5 Nm.

Figure 15. Time evolution parameters of the PMSM control system based on robust controller with
Figure 15. Time evolution parameters of the PMSM control system based on robust controller
RL-TD3 agent, for rotor speed reference ωref = [1000, 1250, 1500, 900] rpm and load torque 15
Sensors 2023, 23, x FOR PEER REVIEW TL of
= 0.5
31
with RL-TD3 agent, for rotor speed reference ω ref = [1000, 1250, 1500, 900] rpm and load torque
Nm.
TL = 0.5 Nm.

Figure 16. Time evolution parameters of the PMSM sensored control system using robust controller
Figure 16. Time evolution parameters of the PMSM sensored control system using robust controller
plus RL-TD3 agent, for a rotor speed reference ωref = [1000, 1250, 1500, 900] rpm, load torque TL = 1
plus
Nm, RL-TD3
and 50% agent, for
increase in aJ parameter.
rotor speed reference ω ref = [1000, 1250, 1500, 900] rpm, load torque
TL = 1 Nm, and 50% increase in J parameter.
Figure 17 shows a comparison of the time evolution of the rotor speed for the PMSM
Figure 17 shows a comparison of the time evolution of the rotor speed for the PMSM
sensored control system using the PI controller, the robust controller, and the robust
sensored control system using the PI controller, the robust controller, and the robust
controller plus RL-TD3 agent. It can be seen that, compared to the robust controller, the
controller plus RL-TD3 agent. It can be seen that, compared to the robust controller, the
response time of the robust controller used in tandem with RL-TD3 agent decreases from
response time of the robust controller used in tandem with RL-TD3 agent decreases from
22.9 ms to 18.2 ms, while the PI controller provides a response time of 30.1 ms.
22.9 ms to 18.2 ms, while the PI controller provides a response time of 30.1 ms.
 plus RL-TD3 agent, for a rotor speed reference ωref = [1000, 1250, 1500, 900] rpm, load torque TL = 1
Nm, and 50% increase in J parameter.

Figure 17 shows a comparison of the time evolution of the rotor speed for the PMSM
sensored control system using the PI controller, the robust controller, and the robust
Sensors 2023, 23, 5799 controller plus RL-TD3 agent. It can be seen that, compared to the robust controller, 14
theof 29
response time of the robust controller used in tandem with RL-TD3 agent decreases from
22.9 ms to 18.2 ms, while the PI controller provides a response time of 30.1 ms.

Figure 17. Comparison of the time evolution of rotor speed for the PMSM sensored control system
Figure 17. Comparison of the time evolution of rotor speed for the PMSM sensored control system
using PI controller, robust controller, and robust controller plus RL-TD3 agent.
using PI controller, robust controller, and robust controller plus RL-TD3 agent.
In addition to the response time and ripple of the PMSM rotor speed signal, the
In addition to the response time and ripple of the PMSM rotor speed signal, the
calculation of the DF of the PMSM rotor speed is also proposed as an element to compare
calculation of the DF of the PMSM rotor speed is also proposed as an element to compare
the performance of PMSM control systems for sensored control structures.
the performance of PMSM
The box-counting control
method systems
is used for sensored
to calculate control structures.
DF, according to [34]. In principle, it
The box-counting method is used to calculate DF, according
starts from a square that can contain the signal, and the length of one to [34].
side In principle,
of this square is it
starts from a square that can contain the signal, and the length of one side of this square
is used as the unit of measurement. The side of the square is chosen so that it contains
the signal, and the size of the side of the square is chosen as a power of 2 to speed up the
calculations. By dividing the unit of measurement by 2, the algorithm is repeated until the
value is below a predefined threshold. We can write that the form of each division has the
expression 1/2k, where k is the current step. The division on the second axis is of the form
nk /22k where the total number of occupied domains on the scale given by the initial signal
is denoted by nk . In the current step, the values obtained for each of the two axes using
the algorithm described above are retained, namely the coordinates of a point Mk (x,y),
represented in the Cartesian system by logarithmic coordinates. The sequence of points
M1 , M2 , . . . , Mn is calculated sequentially at each step. The stop condition is when the set
threshold value has been exceeded. The slope of a line closest to points M1 , M2 , . . . , Mn is
calculated by least squares, and the value obtained represents the DF of the initial signal.
In the MATLAB environment, using the command “[n,r] = boxcount(signal,‘slope’)”, the
vectors of the two dimensions are obtained according to the algorithm described above.
Moreover, by using the commands “df = −diff(log(n))./diff(log(r))” and “[‘DF =
’num2str(mean(df(1:length(df)))) ‘+/− ’num2str(std(df(1:length(df))))])” the coordinates
of the Mk points in logarithmic coordinates are obtained, as well as the slope of the line
closest to the points provided by the algorithm representing DF.
Figures 18–20 show the DF of the rotor speed signal when using PI controller, robust
controller, and robust controller plus RL-TD3 agent for PMSM sensored control system.
Table 2 shows the comparative performance of the proposed controllers for the PMSM
sensored control system. The elements that are compared are response time, rotor speed
ripple and DF of the rotor speed signal. The performance improvement is clearly seen when
using a robust controller compared to the case of using the classical PI controllers in the
sensored control structure of the PMSM. Moreover, the improvement in the performance of
the PMSM control system can be further enhanced by using the robust controller in tandem
with an RL-TD3 agent. Note that by analyzing the obtained values of the DF of the rotor
speed signal for the three cases of controllers presented, the hypothesis presented in [34] is
verified, i.e., when using a controller with superior performance, the DF of the controlled
signal is high.
 Moreover,
algorithm by using
described above.the commands “df = −diff(log(n))./diff(log(r))” and “[‘DF =
’num2str(mean(df(1:length(df))))
Moreover, by using the commands ‘+/− ’num2str(std(df(1:length(df))))])”
“df = −diff(log(n))./diff(log(r))”theand
coordinates
“[‘DF =
of the Mk points in logarithmic coordinates
’num2str(mean(df(1:length(df)))) are obtained, as well as the slope
‘+/− ’num2str(std(df(1:length(df))))])” of the line
the coordinates
closest
of the Mtok the points
points provided bycoordinates
in logarithmic the algorithmare representing
obtained, as DF.
well as the slope of the line
Figures 18–20 show the DF of the rotor speed signal when
closest to the points provided by the algorithm representing DF.using PI controller, robust
Sensors 2023, 23, 5799 15 of 29
controller,
Figures and robust
18–20 showcontroller
the DF ofplustheRL-TD3 agent
rotor speed for PMSM
signal sensored
when using control system.
PI controller, robust
controller, and robust controller plus RL-TD3 agent for PMSM sensored control system.

Figure 18. DF of the rotor speed signal in the case of using the PI controller for the PMSM control
system.
Figure18.
Figure 18. DF
DFof
ofthe
therotor
rotorspeed
speedsignal
signalinin the
the case
case of of using
using thethe PI controller
PI controller for for
the the
PMSMPMSM control
control system.
system.

Sensors 2023, 23, x FOR PEER REVIEW 17 of 31

Figure 19.
Figure 19. DF
DFofofthe rotor
the speed
rotor signal
speed in the
signal casecase
in the of using the robust
of using controller
the robust for thefor
controller PMSM
the PMSM
Figure 19.
control DF of the rotor speed signal in the case of using the robust controller for the PMSM
system.
control system.
control system.

Figure20.
Figure 20.DF
DFofofthe
therotor
rotor speed
speed signal
signal in
in the
the case
caseof
ofusing
usingthe
therobust
robustcontroller
controllercombined with
combined thethe
with
RL-TD3 agent for the PMSM control system.
RL-TD3 agent for the PMSM control system.

Table
Table 2. 2 shows ofthe
Performances comparative
proposed performance
controllers for PMSM of the proposed
sensored controllers for the
control system.
PMSM sensored control system. The elements that are compared are response time, rotor
speed ripple and DF of Response
the rotorTime
speed signal. The
Rotor performance
Speed Ripple improvement is clearly
Controller Used in PMSM Sensored Control System DF of Rotor Speed Signal
seen when using a robust controller
[ms] compared to[rpm]
the case of using the classical PI con-
PI controller trollers in the sensored control structure of the PMSM.
30.1 108.1Moreover, the0.95548
improvement in the
+/− 0.118370
Robust controller performance of the PMSM control22.9 system can be further
87.3 enhanced 0.97560
by using+/−the robust
0.053594
Robust controller combined with RL-TD3 agent 18.2 52.4 0.99204 +/− 0.023511
controller in tandem with an RL-TD3 agent. Note that by analyzing the obtained values
of the DF of the rotor speed signal for the three cases of controllers presented, the hy-
pothesis presented in [34] is verified, i.e., when using a controller with superior perfor-
mance, the DF of the controlled signal is high.

Table 2. Performances of proposed controllers for PMSM sensored control system.

Sensors 2023, 23, 5799 16 of 29

5. PMSM Sensorless Control System Using Improved ESO-Type Observer Variants

This section presents the sensorless control of the PMSM, where the controller used
is the one described in Section 4.2, i.e., a robust controller working in tandem with an
RL-TD3 agent. Once the controller is fixed, there is the problem of estimating the speed
of the PMSM rotor by means of a speed observer. Due to the structure of the nonlinear
equations of the PMSM, an ESO-type observer is chosen according to [29]. This section also
proposes improvements to PMSM rotor speed estimation, where the element of comparison
is the rotor speed provided by the speed sensor in the PMSM sensored control structure.
Therefore, first of all, the ESO-type observer, as its name implies, extends the estimated
states by an additional state, and from the description of the ESO-type observer equations,
it is also possible to estimate the load torque TL . In order to achieve an improvement in
the estimated speed of the PMSM rotor, it is proposed to use a multiple NN, which uses
multiple NNs trained over portions of the load torque variation, and selected accordingly
by a stateflow to provide an estimate of the load torque. A second way to improve PMSM
rotor speed estimation is to use an RL-TD3 agent that, after training, provides signals that
overlap the ESO-type observer output to provide an estimate of the PMSM rotor closer to
the value provided for the PMSM sensored control structure.

5.1. ESO-Type Observer Description

To estimate the PMSM rotor speed using the ESO-type observer, we start from the
form of the PMSM nonlinear description equations. These equations were described in
Section 2, but can be written in the general form given by Equation (28).
.
 
y(n) (t) = f y(t), y(t), . . . , y(n−1) (t), d(t), t + bu(t) (28)

Equation (28) describes a monovariable nonlinear system in which perturbations occur,

where y(l) denotes the derivative l of the output of the system y, u denotes the input to the
system y, and d denotes the perturbation acting on the system y.
.
By denoting x1 = y, x2 = y, . . . , xn = y(n−1) , the following form is obtained:
 .
xi = xi+1 , i = 1, . . . , n − 1
. (29)
x n = f ( x1 , x2 , . . . , xn , d, t) + bu

An additional state is also selected in the following form:


xn+1 = f ( x1 , x2 , . . . , xn , d, t)
. (30)
x n +1 = h ( t )

which gives the form of the function h(t):

.
h(t) = f ( x1 , x2 , . . . , xn , d, t) (31)

Note that in the descriptions given by the previous equations, the variable f includes
the concentrated disturbances acting on the system.
Starting from the equations of the PMSM nonlinear system in Section 2, and adding
the previous transformations, the equations of an ESO-type observer can be synthesized in
the following form [29]:
(.
x̂i = x̂i+1 + β i (y − x̂1 ), i = 1, . . . , n
. (32)
x̂ n+1 = β n+1 (y − x̂1 )

Thus, Relation (32) describes the component equations of the system describing the
ESO-type observer which, based on a minimum of information, will provide the estimate
of the states of the system together with the extended state including the f term.
Sensors 2023, 23, 5799 17 of 29

By adjusting the above relation to estimate the PMSM rotor speed, the following
relation can be written:  
dω 1 3
= n p λ0 iq − TL − Bω (33)
dt J 2
where the term f, which includes the generalized disturbances acting on the system of
equations describing the operation of the PMSM, is expressed as follows:

B T
f =− ω− L (34)
J J

Substituting Relation (34) into Relation (33) gives the following equation:

dω
= f + biq (35)
dt
.
If in the previous relations x1 = ω, x2 = f and f = h, the matrix expression of the
system containing the PMSM rotor speed is obtained as follows:
 .
x = Ax + Bu + Eh
(36)
y = Cx

where the system matrices (36) are given by:

     
0 1 b   0
A= ; B= ; C= 1 0 ; E= (37)
0 0 0 1

Based on this, the ESO-type observer structure can be written as follows:

 .
x = Ax + Bu + L( x1 − z1 )
(38)
y = Cz

The bandwidth of the ESO-type observer system is denoted by ω 0 . The poles of

this system are assigned the value −ω 0 , and to determine the gain of the observer L, the
identification is derived from the equality of polynomials as follows [29]:

λ ( s ) = s 2 + L 1 s + L 2 = ( s + ω0 ) 2 (39)


L1 = 2ω0
(40)
L2 = ω02
 
2ω0
L= (41)
ω02
For fast convergence, the chosen ω 0 is equal to 900, and in the implementation in
Figure 21, it can be seen that the PMSM rotor speed is estimated by the state variable x1 of
the observer, and the term f is estimated by the state variable x2 of the observer.
Furthermore, considering the estimated PMSM rotor speed, the estimated PMSM rotor
angle can also be obtained as:

Zt
θ̂ (t) = θ (t0 ) + ω̂ (t)dt (42)
t0

where: θ (t0 ) is the initial value of the rotor position relative to a fixed reference point.
  2 0 
L 2  (41)
 0 
For fast convergence, the chosen ω0 is equal to 900, and in the implementation in
Figure 21, it can be seen that the PMSM rotor speed is estimated by the state variable x1 of
Sensors 2023, 23, 5799 18 of 29
the observer, and the term f is estimated by the state variable x2 of the observer.
Furthermore, considering the estimated PMSM rotor speed, the estimated PMSM
rotor angle can also be obtained as:
The estimate of the load torque can also be t obtained in the form of the following
equation, where x1 and x2 represent ˆ (the  ( t 0 ) of the
t )  states ˆ ( t )ESO-type
dt observer: (42)
0 t

where:  (t0 ) is the initial value of the rotor position relative to a fixed reference point.
 
B
T̂L = J − x1 − x2 (43)
The estimate of the load torque can alsoJ be obtained in the form of the following
equation, where x1 and x2 represent the states of the ESO-type observer:
In the following section, a method is proposed to improve the estimation of the PMSM
rotor speed, by replacing EquationTˆ(43),  B was used 
which
L  J x1  x 2  (43) TL
to estimate the load torque
with a multiple NN trained on the specific portions
J of the load torque variation.

Figure 21. ESO-type observer block diagram. The red do ed lines highlight the output quantities of
Figure 21. ESO-type observer block diagram. The red dotted lines highlight the output quantities of
the observer.
the observer.

5.2. Load Torque Estimation Using Multiple NN

TL load torque estimation is performed using multiple NNs, each of which is a two-
layer feed-forward network [30,31]. The Neural Net Fitting application from MATLAB is
used in the implementation and proper training of each NN over 5 load torque ranges TL ,
i.e., between 0 Nm and 1 Nm. After the numerical simulation on a predetermined interval
in the load torque range specified above, the input data of each NN, namely the PMSM
rotor speed ω, the derivative of the PMSM rotor speed dω/dt, id and iq currents from d-q
frame, but also the output of each NN given by the load torque required to generate and
train it, were recorded. Therefore, in the Neural Net Fitting application, the input data,
represented by sets of 500,000 values for each variable, is divided into 70% data needed for
training, 15% data needed for validation, and 15% data needed for testing.
The algorithm chosen for training, validation, and testing is Bayesian regulariza-
tion. This type of algorithm minimizes a linear combination of squared errors and
weights. It also modifies the linear combination so that, at the end of training, the re-
sulting network has good generalization qualities. After generating the required 5 NNs
in the selected load torque range, each NN is selected using a stateflow implemented in
MATLAB [30,31]. Figure 22 shows the MATLAB/Simulink subsystem implementation of
multiple NN for load torque estimation and selection. Implementation of a NN for load
torque estimation in the Neural Net Fitting app from MATLAB is shown in Figure 23.
Details of this implementation, in terms of the block diagram of a NN for load torque esti-
mation, represented by two-layer feed-forward network with sigmoid hidden neurons (150)
and linear output neurons for regression tasks, is shown in Figure 23a, while Figure 23b
shows the Simulink NN exported model for load torque estimation, and this stage is per-
formed using the Neural Net Fitting application after the process of training, validation
and testing of each NN.
 estimation in the Neural Net Fi ing app from MATLAB is shown in Figure 23. Details of
this implementation, in terms of the block diagram of a NN for load torque estimation,
represented by two-layer feed-forward network with sigmoid hidden neurons (150) and
linear output neurons for regression tasks, is shown in Figure 23a, while Figure 23b
shows the Simulink NN exported model for load torque estimation, and this stage is
Sensors 2023, 23, 5799 19 of 29
performed using the Neural Net Fi ing application after the process of training, valida-
tion and testing of each NN.

Figure 22. MATLAB/Simulink subsystem implementation of multiple NN for load torque estima-
Figure
Sensors 2023, 23, x FOR PEER 22. MATLAB/Simulink subsystem implementation of multiple NN for load torque estimation
REVIEW 21 of 31
tion and selection
and selection using using stateflow.
stateflow. “*” is specific
“*” is specific to the
to the multi multi
switch switch
block block inand
in Simulink Simulink andthe
represents repre-
sents the default
default selection. selection.

(a) (b)

Figure 23. Figure 23. Implementation

Implementation of an NNoffor
anload
NN for loadestimation
torque torque estimation in Neural
in Neural NetworkNetwork
FittingFitoolbox
ing toolbox
from MATLAB: (a) block diagram of a NN for load torque estimation; (b) Simulink NN exported
from MATLAB: (a) block diagram of a NN for load torque estimation; (b) Simulink NN exported
model for load torque estimation.
model for load torque estimation.
The number of epochs used in the training phase is 1000, and the learning rate is 0.1.
The number of epochs used in the training phase is 1000, and the learning rate is
Figure 24 shows the best validation performance, which represents the mean square error
0.1. Figure 24 shows the best validation performance, which represents the mean square
(MSE) values obtained on the entire data set selected for training, validation and testing
during the iterative calculation process. Therefore, it can be seen that, at epoch 518, a MSE
value of 0.00031339 is obtained from the total computation period of the Bayesian regu-
larization algorithm. Figure 25 shows the evolution of the gradient in the iterative calcu-
lation algorithm, and it can be seen that, when applying the NN algorithm that provides
 Figure 23. Implementation of an NN for load torque estimation in Neural Network Fi ing toolbox
from MATLAB: (a) block diagram of a NN for load torque estimation; (b) Simulink NN exported
model for load torque estimation.

Sensors 2023, 23, 5799

The number of epochs used in the training phase is 1000, and the learning20rate is 0.1.
of 29

Figure 24 shows the best validation performance, which represents the mean square error
(MSE) values obtained on the entire data set selected for training, validation and testing
during
error the iterative
(MSE) calculation
values obtained process.
on the entireTherefore, it can be
data set selected forseen that, at
training, epoch 518,
validation anda MSE
testing during the iterative calculation process. Therefore, it can be seen that,
value of 0.00031339 is obtained from the total computation period of the Bayesian at epoch 518,regu-
a MSE value of 0.00031339 is obtained from the total computation period of the Bayesian
larization algorithm. Figure 25 shows the evolution of the gradient in the iterative calcu-
regularization algorithm. Figure 25 shows the evolution of the gradient in the iterative
lation algorithm, and it can be seen that, when applying the NN algorithm that provides
calculation algorithm, and it can be seen that, when applying the NN algorithm that
the load torque estimation at epoch 524, the gradient value is 0.00001489.
provides the load torque estimation at epoch 524, the gradient value is 0.00001489.

Sensors 2023, 23, x FOR PEER REVIEW 22 of

Figure24.
Figure 24.Best
Bestperformance
performance validation
validation of aofneural
a neural fi ing
fitting network
network for load
for load torque
torque estimation.
estimation.

Figure 25. TrainingFigure

stage 25.
of aTraining stage ofnetwork
neural fitting a neuralfor
fi ing network
load torque for load torque estimation.
estimation.

Figure 26 showsFigure 26 shows

the sample thedistribution
error sample errorindistribution
the iterativein calculation
the iterative process.
calculation
It process.
can be seen that the sample error corresponding to an NN for
can be seen that the sample error corresponding to an NN for load torque estimation is load torque estimation
distributed
distributed between valuesbetween
of ±0.01. values
The of ±0.01. Theanalysis
regression regression analysisofdiagrams
diagrams of the trainin
the training
sample data, test sample data, and the entire data set during the
sample data, test sample data, and the entire data set during the application of the Bayesian application of th
Bayesian regularization type algorithm, are shown in Figure 27. It can be seen that th
regularization type algorithm, are shown in Figure 27. It can be seen that the correlation
correlation factor R value corresponding to a NN for load torque estimation is around th
factor R value corresponding to a NN for load torque estimation is around the value of 0.94.
value of 0.94.
Figure 28 shows the load torque and estimated load torque using multiple NN es-
timation time evolution after numerical simulation of the PMSM control system for the
following load torque sequence [0.1, 0.7, 0.9, 0.5, 0.3] Nm. It can be seen that the multiple
NN estimation is very good, which creates the conditions for obtaining superior PMSM
 Figure 25. Training stage of a neural fi ing network for load torque estimation.

Figure 26 shows the sample error distribution in the iterative calculation process. It
can be seen that the sample error corresponding to an NN for load torque estimation is
Sensors 2023, 23, 5799 distributed between values of ±0.01. The regression analysis diagrams of the training21 of 29
sample data, test sample data, and the entire data set during the application of the
Bayesian regularization type algorithm, are shown in Figure 27. It can be seen that the
correlation factor R value corresponding to a NN for load torque estimation is around the
rotor
valuespeed
of 0.94.estimation results when the load torque is provided by multiple NNs in the
ESO observer structure.

Sensors 2023, 23, x FOR PEER REVIEW 23 of 31

Figure 26.
Figure 26. Error
Error histogram
histogramafter
aftertraining
trainingstage
stageofofa aneural
neuralfitting
fi ing network
network forfor load
load torque
torque estima-
estimation.
tion.

Figure27.27.
Figure Regression
Regression factor
factor after
after training,
training, validation,
validation, and testing
and testing stagesstages of a neural
of a neural fitting fi ing network
network
forload
for loadtorque
torque estimation.
estimation.

Figure 28 shows the load torque and estimated load torque using multiple NN es-
timation time evolution after numerical simulation of the PMSM control system for the
following load torque sequence [0.1, 0.7, 0.9, 0.5, 0.3] Nm. It can be seen that the multiple
NN estimation is very good, which creates the conditions for obtaining superior PMSM
 Sensors 2023, 23, 5799 22 of 29

5.3. Improved Estimation of PMSM Rotor Speed Using ESO-Type Observer and RL-TD3 Agent
Figure 27. Regression factor after training, validation, and testing stages of a neural fi ing network
Sensors 2023, 23, x FOR PEER REVIEW Similar
for load toestimation.
torque using an RL-TD3 agent to improve the performance of the PMSM control
system, an agent whose implementation is shown in Figure 29 is selected and its reward is
definedFigure 28following
by the shows therelation
load torque
[32,33]:and estimated load torque using multiple NN es-
timation time evolution after numerical simulation of the PMSM ! control system for the
5.3. Improved
following Estimation
load torque sequence of PMSM
[0.1, 0.7, Rotor
20.9, 0.5, Speed
0.3] Nm. Using
It can ESO-Type
be Observer
seen that the (44) and R
multiple
 2
r ESO = − Qωerror + R∑ ut−1
j
NN estimation is very good, which creates the conditions for obtaining superior PMSM
Similar to using an RL-TD3 agent jto improve the performance of the P
rotor speed estimation results when the load torque is provided by multiple NNs in the
system,
ESO
where anstructure.
observer
Q = 0.5, Ragent whose
= 0.1, and
j implementation
ut−1 is the history of previous is shown
weightedinactions.
Figure 29 is selected an
is defined by the following relation [32,33]:

rESO   Q  error
2
  
 R  utj1
2

 j 
j
where Q = 0.5, R = 0.1, and ut 1 is the history of previous weighted actions.
Figure 30 shows the evolution of the performance of the RL-TD3 age
during the training phase over a period of 200 episodes. After the RL-TD3 a
stage, it will cooperate with the ESO-type observer in the sense that the R
will provide correction signals that overlap the value of the rotor speed est
vided by the ESO-type observer itself, thus achieving superior performance
rotor speed estimation. Thus, by implementing this observer variant, an imp
rotor speed estimation is achieved for the sensorless variant of the PMSM
ture.28.Load
Figure28.
Figure Loadtorque
torqueand
andestimated
estimatedload
loadtorque
torqueusing
usingaamultiple
multipleNN
NNestimation
estimationtime
timeevolution.
evolution.

Figure
Figure 29. 29. MATLAB/Simulink
MATLAB/Simulink subsystem
subsystem implementation
implementation of the
of the RL-TD3 agent RL-TD3 agent
for improvement in for im
PMSM
PMSM rotorrotor
speedspeed estimation.
estimation.

Figure 30 shows the evolution of the performance of the RL-TD3 agent algorithm
during the training phase over a period of 200 episodes. After the RL-TD3 agent training
stage, it will cooperate with the ESO-type observer in the sense that the RL-TD3 agent will
provide correction signals that overlap the value of the rotor speed estimation provided
by the ESO-type observer itself, thus achieving superior performance to the PMSM rotor
speed estimation. Thus, by implementing this observer variant, an improvement in rotor
speed estimation is achieved for the sensorless variant of the PMSM control structure.
Sensors 2023, 23, 5799 23 of 29
Figure 29. MATLAB/Simulink subsystem implementation of the RL-TD3 agent for improvement in
PMSM rotor speed estimation.

Figure
Figure 30.
30. Performance
Performanceevolution
evolutionfor
forthe
theRL-TD3
RL-TD3agent
agentforfor
improvement
improvementinin
PMSM rotor
PMSM speed
rotor es-
speed
timation.
estimation.

6.
6. Numerical
Numerical Simulation
Simulation of of the
the PMSM
PMSM Sensorless
SensorlessControl
ControlSystem
System
MATLAB/Simulink implementation ofofthe
MATLAB/Simulink implementation the proposed
proposed PMSMPMSM sensorlesss
sensorlesss controlcontrol
sys-
system
tem basedbased on robust
on robust controller
controller combined
combined withwith RL-TD3
RL-TD3 agentagent for improvement
for improvement in ud
in ud and
and uq command
uq command usingusing improved
improved ESO-types
ESO-types versionversion observers
observers forestimation
for speed speed estimation
is shownis
Sensors 2023, 23, x FOR PEER REVIEW 25 of 31 for
shown in Figure 31. Figure 32 shows the MATLAB/Simulink subsystem implementation
in Figure 31. Figure 32 shows the MATLAB/Simulink subsystem implementation
for ESO-type
ESO-type basic
basic version
version observer
observer for PMSM
for PMSM rotorrotor speed
speed estimation.
estimation.

Figure 31. MATLAB/Simulink implementation of the proposed PMSM sensorlesss control system
Figure 31. MATLAB/Simulink implementation of the proposed PMSM sensorlesss control system
based on a robust controller combined with an RL-TD3 agent for improvement in ud and uq com-
based on using
mand, a robust controller
improved combinedversion
ESO-types with anobservers
RL-TD3 agent for improvement
for speed ud specific
in is
estimation. “*” and uq command,
to the
multi switch block in Simulink and represents the default selection.
using improved ESO-types version observers for speed estimation. “*” is specific to the multi switch
block in Simulink and represents the default selection.
 Figure 31. MATLAB/Simulink implementation of the proposed PMSM sensorlesss control system
based on a robust controller combined with an RL-TD3 agent for improvement in ud and uq com-
Sensors 2023, 23, 5799 24 of 29
mand, using improved ESO-types version observers for speed estimation. “*” is specific to the
multi switch block in Simulink and represents the default selection.

Sensors 2023, 23, x FOR PEER REVIEW 26 of 31

Figure32.
Figure 32.MATLAB/Simulink
MATLAB/Simulink subsystem
subsystem implementation
implementation for an ESO-type
for an ESO-type basicobserver
basic version versionforobserver
for PMSM
PMSM rotorrotor
speedspeed estimation.
estimation. The redThe redlines
dotted do highlight
ed lines the
highlight the outputofquantities
output quantities of the ob-
the observer.
For an evolution of the PMSM rotor speed, reference ωref = [500, 800, 1100, 700] rpm
server.
and load Fortorque TL = [0.1,
an evolution 0.7,PMSM
of the 0.9, 0.5, 0.3]speed,
rotor Nm over the time
reference ω ref interval 0–1s,
= [500, 800, sequenced
1100, 700] rpm as [0,
0.25,
and0.5, 0.75,
load 1] s,TFigures
torque 33–36
L = [0.1, 0.7, 0.9,show theNm
0.5, 0.3] time
overevolution parameters
the time interval 0–1s, of PMSM sensorless
sequenced as [0,
control system based on a robust controller with an RL-TD3 agent for four versions of the
0.25, 0.5, 0.75, 1] s, Figures 33–36 show the time evolution parameters of PMSM sensorless
ESO-type observer
control system foron
based rotor speed
a robust estimation
controller withpresented
an RL-TD3in Section
agent 5. versions of the
for four
ESO-type observer for rotor speed estimation presented in Section 5.
The comparison between the time evolution of the PMSM rotor speed estimates
The comparison between the time evolution of the PMSM rotor speed estimates using
using the four observer variants and the time evolution of the measured PMSM rotor
the four observer variants and the time evolution of the measured PMSM rotor speed is
speed
shownis shown
in Figurein37.
Figure 37.

Figure 33.33.Time
Figure Timeevolution
evolution parameters
parameters of ofthe
thePMSM
PMSM sensorless
sensorless control
control system
system based based on a robust
on a robust
controller with an RL-TD3 agent, using the ESO-type basic version observer for rotor
controller with an RL-TD3 agent, using the ESO-type basic version observer for rotor speed estimation; speed esti-
mation; rotor reference
rotor speed speed reference ωref800,
ω ref = [500, = [500,
1100,800,
700]1100, 700]load
rpm and rpm and Tload
torque torque TL = [0.1, 0.7, 0.9, 0.5,
L = [0.1, 0.7, 0.9, 0.5, 0.3] Nm.
0.3] Nm.
 Figure 33. Time evolution parameters of the PMSM sensorless control system based on a robust
Sensors 2023, 23, 5799 controller with an RL-TD3 agent, using the ESO-type basic version observer for rotor 25 ofspeed
29 esti-
mation; rotor speed reference ωref = [500, 800, 1100, 700] rpm and load torque TL = [0.1, 0.7, 0.9, 0.5,
0.3] Nm.

Sensors 2023, 23, x FOR PEER REVIEW 27 of 31

Figure
Figure34.34.Time
Timeevolution parametersofofthethe
evolution parameters PMSM
PMSM sensorless
sensorless control
control systemsystem
based onbased on a robust
a robust
controller
controllerwith
withan
anRL-TD3 agent,using
RL-TD3 agent, usinganan ESO-type
ESO-type versionversion observer
observer with awith a multiple
multiple NN observer
NN observer
forfor
rotor speed
rotor speed estimation;
estimation;rotor
rotor speed referenceωω
speed reference refref==[500,
[500,800,
800,1100,
1100, 700]
700] rpm
rpm andand load
load torque TL =
torque
[0.1,
TL0.7, 0.9,0.7,
= [0.1, 0.5, 0.3]
0.9, 0.5,Nm.
0.3] Nm.

Figure 35.35.Time
Figure Timeevolution parametersofof
evolution parameters thethe PMSM
PMSM sensorless
sensorless control
control systemsystem based
based on on a robust
a robust
controller with
controller withananRL-TD3 agent,using
RL-TD3 agent, usinganan ESO-type
ESO-type version
version observer
observer combined
combined with anwith an RL-TD3
RL-TD3
agent observer
agent observerforforrotor speedestimation;
rotor speed estimation; rotor
rotor speed
speed reference
reference ω ref =ωref = 800,
[500, [500,1100,
800,700]
1100,
rpm700]
andrpm and
load torque
load torqueTTL ==
L [0.1,
[0.1,0.7,
0.7, 0.9, 0.5,0.3]
0.9, 0.5, 0.3]Nm.
Nm.
 Figure 35. Time evolution parameters of the PMSM sensorless control system based on a robust
Sensors 2023, 23, 5799 controller with an RL-TD3 agent, using an ESO-type version observer combined with an26RL-TD3 of 29
agent observer for rotor speed estimation; rotor speed reference ωref = [500, 800, 1100, 700] rpm and
load torque TL = [0.1, 0.7, 0.9, 0.5, 0.3] Nm.

Figure 36. Time

Figure 36. Time evolution
evolution parameters
parametersofofthe thePMSM
PMSMsensorless
sensorlesscontrol
control system
system based
based onon a robust
a robust
controller with an RL-TD3 agent, using an ESO-type version with a multiple NN observer com-
controller with an RL-TD3 agent, using an ESO-type version with a multiple NN observer combined
bined with an RL-TD3 agent for rotor speed estimation; rotor speed reference ωref = [500, 800,
Sensors 2023, 23, x FOR PEER REVIEW 28 1100,
of 31
with an RL-TD3 agent for rotor speed estimation; rotor speed reference ω ref = [500, 800, 1100, 700]
700] rpm and load torque TL = [0.1, 0.7, 0.9, 0.5, 0.3] Nm.
rpm and load torque TL = [0.1, 0.7, 0.9, 0.5, 0.3] Nm.

Figure
Figure 37. Comparison of
37. Comparison of the
thetime
timeevolution
evolutionfor
forthe
therotor
rotorspeed
speedinin a sensored
a sensored case,
case, and
and estimated
estimated
rotor speeds in sensorless cases (improved ESO-types version observers) for the PMSM control
rotor speeds in sensorless cases (improved ESO-types version observers) for the PMSM control system
system based on a robust controller with an RL-TD3 agent.
based on a robust controller with an RL-TD3 agent.

Table
Table 33 summarizes
summarizesthethecomparative
comparative performance
performance of four
of the the four observer
observer variants
variants of the of
the PMSM
PMSM rotorrotor speed
speed and measured
and measured PMSM PMSM
rotor rotor
speedspeed as a function
as a function of the response
of the response time
time and estimated speed ripple. Note that the error ripple for an analyzed variable, rel-
ative to the reference variable, can be associated with the root mean square (RMS) value
of the error between the two signals. While, in Table 2, the error ripple was calculated for
the error signal given by the reference rotor speed and the measured rotor speed, in Table
3 the error ripple is calculated for the error signal given by the measured rotor speed and
Sensors 2023, 23, 5799 27 of 29

and estimated speed ripple. Note that the error ripple for an analyzed variable, relative
to the reference variable, can be associated with the root mean square (RMS) value of the
error between the two signals. While, in Table 2, the error ripple was calculated for the
error signal given by the reference rotor speed and the measured rotor speed, in Table 3
the error ripple is calculated for the error signal given by the measured rotor speed and
the estimated rotor speed. It can be noted that the best performance in PMSM rotor speed
estimation, in terms of the lowest estimated response time and speed ripple, is obtained
when using the ESO-type observer with the load torque estimated using the multiple NN
observer combined with the RL-TD3 agent.

Table 3. Performances of proposed speed observers for PMSM sensorless control system.

Improved of Response Improved of Speed

Time in Comparison Ripple in Comparison
Response Time Estimated Speed Ripple
Observer with ESO-Type Basic with ESO-Type Basic
[ms] [rpm]
Version Observer Version Observer
[%] [%]
ESO-type basic version observer 30.7 – 3.43 –
ESO-type version observer with load torque estimated using
27.4 11 2.48 27
multiple NN observer
ESO-type version observer combined RL-TD3 agent 23.5 24 1.85 46
ESO-type version observer with load torque estimated using
21.1 32 1.39 59
multiple NN observer combined with RL-TD3 agent

7. Conclusions
Naturally, PMSM operation is subject to a number of perturbations, including both
changes in internal parameters (e.g., stator resistance Rs and combined inertia of rotor and
load J) and changes in load torque TL . In order to maintain the performance of the control
system over a relatively wide range of variation of the above types of parameters, a PMSM
control system based on a robust controller is used. The article also presents the synthesis of
the robust controller, its implementation in MATLAB/Simulink, and an improved version
using an RL-TD3 agent that works in tandem with the robust controller to achieve superior
performance of the PMSM sensored control system.
Regarding the sensorless control of the PMSM, starting from a basic structure of an
ESO-type observer, four variants of observers are presented, obtained by combining the
use of a multiple NN and an RL-TD3 agent.
Future work proposes the implementation in real time and the optimization of the
performance, both in terms of control and in terms of PMSM rotor speed estimation, by
modifying the robust control weights accordingly, and implementing multiple NN and
reward of the RL-TD3 agent.

Author Contributions: Conceptualization, M.N. and C.-I.N.; Data curation, M.N., C.-I.N., C.I., D.S, .
and M.R.; Formal analysis, M.N., C.-I.N., C.I., D.S, . and M.R.; Funding acquisition, M.N.; Investigation,
M.N. and C.-I.N.; Methodology, M.N., C.-I.N., C.I., D.S, . and M.R.; Project administration, M.N.;
Resources, M.N. and C.-I.N.; Software, M.N. and C.-I.N.; Supervision, M.N. and C.I.; Validation,
M.N., C.-I.N., C.I. and D.S, .; Visualization, M.N., C.-I.N., D.S, . and M.R.; Writing—original draft, M.N.,
C.-I.N., C.I., D.S, . and M.R.; Writing—review & editing, M.N. and C.-I.N. All authors have read and
agreed to the published version of the manuscript.
Funding: This paper was elaborated as part of the “NUCLEU Program within the framework of the
National Research, Development and Innovation Plan for 2022–2027, developed with the support of
Ministry of Research, Innovation and Digitization, Project No PN 23 33 02 04”.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: Not applicable.
Conflicts of Interest: The authors declare no conflict of interest.
Sensors 2023, 23, 5799 28 of 29

References
1. Krishnan, R. Permanent Magnet Synchronous and Brushless DC Motor Drives; CRC Press/Taylor & Francis: Boca Raton, FL, USA, 2017.
2. Bose, B.K. Modern Power Electronics and AC Drives; Prentice Hall PTR: Hoboken, NJ, USA, 2002.
3. Zhang, X.; Wang, C.; Wang, T.; Wang, G.; Song, Z.; Huang, J. Research on the control strategy of five-phase fault-tolerant servo
system for aerospace. In Proceedings of the CSAA/IET International Conference on Aircraft Utility Systems (AUS 2020), Online,
18–21 September 2020; pp. 1090–1094.
4. Lekshmi, S.; Lal Priya, P.S. Range Extension of Electric Vehicles with Independently Driven Front and Rear PMSM Drives by
Optimal Driving and Braking Torque Distribution. In Proceedings of the IEEE International Conference on Power Electronics,
Smart Grid and Renewable Energy (PESGRE2020), Cochin, India, 2–4 January 2020; pp. 1–6.
5. Golesorkhie, F.; Yang, F.; Vlacic, L.; Tansley, G. Field Oriented Control-Based Reduction of the Vibration and Power Consumption
of a Blood Pump. Energies 2020, 13, 3907. [CrossRef]
6. Tang, X.; Zhang, Z.; Liu, X.; Liu, C.; Jiang, M.; Song, Y. A Novel Field-Oriented Control Algorithm for Permanent Magnet
Synchronous Motors in 60◦ Coordinate Systems. Actuators 2023, 12, 92. [CrossRef]
7. Guo, J.; Fan, T.; Li, Q.; Wen, X. An Angle-Compensating, Complex-Coefficient PI Controller Used for Decoupling Control of a
Permanent-Magnet Synchronous Motor. Symmetry 2022, 14, 101. [CrossRef]
8. Wei, Z.; Zhao, M.; Liu, X.; Lu, M. A Novel Variable-Proportion Desaturation PI Control for Speed Regulation in Sensorless PMSM
Drive System. Appl. Sci. 2022, 12, 9234. [CrossRef]
9. Huang, Y.; Zhang, J.; Chen, D.; Qi, J. Model Reference Adaptive Control of Marine Permanent Magnet Propulsion Motor Based
on Parameter Identification. Electronics 2022, 11, 1012. [CrossRef]
10. Vujji, A.; Dahiya, R. Speed Estimator for Direct Torque and Flux Control of PMSM Drive using MRAC based on Rotor flux. In
Proceedings of the IEEE 9th Power India International Conference (PIICON), Sonepat, India, 28 February–1 March 2020; pp. 1–6.
11. Park, J.-H.; Lim, H.-S.; Lee, G.-H.; Lee, H.-H. A Study on the Optimal Control of Voltage Utilization for Improving the Efficiency
of PMSM. Electronics 2022, 11, 2095. [CrossRef]
12. Nicola, M.; Nicola, C.-I. Improvement of Linear and Nonlinear Control for PMSM Using Computational Intelligence and
Reinforcement Learning. Mathematics 2022, 10, 4667. [CrossRef]
13. Wang, Y.; Li, H.; Yang, L.; Wang, X. Modulated Model-Free Predictive Control with Minimum Switching Losses for PMSM Drive
System. IEEE Access 2020, 8, 20942–20953. [CrossRef]
14. Zhou, C.; Yu, F.; Zhu, C.; Mao, J. Sensorless Predictive Current Control of a Permanent Magnet Synchronous Motor Powered by a
Three-Level Inverter. Appl. Sci. 2021, 11, 10840. [CrossRef]
15. Kakouche, K.; Rekioua, T.; Mezani, S.; Oubelaid, A.; Rekioua, D.; Blazek, V.; Prokop, L.; Misak, S.; Bajaj, M.; Ghoneim, S.S.M.
Model Predictive Direct Torque Control and Fuzzy Logic Energy Management for Multi Power Source Electric Vehicles. Sensors
2022, 22, 5669. [CrossRef]
16. Masoud, U.M.M.; Tiwari, P.; Gupta, N. Designing of an Enhanced Fuzzy Logic Controller of an Interior Permanent Magnet
Synchronous Generator under Variable Wind Speed. Sensors 2023, 23, 3628. [CrossRef]
17. Regaya, C.B.; Farhani, F.; Zaafouri, A.; Chaari, A. Adaptive Proportional-Integral Fuzzy Logic Controller of Electric Motor Drive.
Eng. Rev. 2021, 42, 26–40. [CrossRef]
18. Hoai, H.-K.; Chen, S.-C.; Chang, C.-F. Realization of the Neural Fuzzy Controller for the Sensorless PMSM Drive Control System.
Electronics 2020, 9, 1371. [CrossRef]
19. Farhani, F.; Regaya, C.B.; Zaafouri, A.; Chaari, A. Real Time PI-Backstepping Induction Machine Drive with Efficiency optimiza-
tion. ISA Trans. 2017, 70, 348–356. [CrossRef] [PubMed]
20. Zhu, Y.; Tao, B.; Xiao, M.; Yang, G.; Zhang, X.; Lu, K. Luenberger Position Observer Based on Deadbeat-Current Predictive
Control for Sensorless PMSM. Electronics 2020, 9, 1325. [CrossRef]
21. Gao, W.; Zhang, G.; Hang, M.; Cheng, S.; Li, P. Sensorless Control Strategy of a Permanent Magnet Synchronous Motor Based on
an Improved Sliding Mode Observer. World Electr. Veh. J. 2021, 12, 74. [CrossRef]
22. Comanescu, M.; Xu, L. Sliding-mode MRAS speed estimators for sensorless vector control of induction Machine. IEEE Trans. Ind.
Electron. 2006, 53, 146–153. [CrossRef]
23. Urbanski, K.; Janiszewski, D. Sensorless Control of the Permanent Magnet Synchronous Motor. Sensors 2019, 19, 3546. [CrossRef]
24. Gu, D.-W.; Petkov, P.H.; Konstantinov, M.M. Robust Control Design with MATLAB (Advanced Textbooks in Control and Signal
Processing), 2nd ed.; Springer: Berlin/Heidelberg, Germany, 2013.
25. Cai, R.; Zheng, R.; Liu, M.; Li, M. Robust Control of PMSM Using Geometric Model Reduction and µ-Synthesis. IEEE Trans. Ind.
Electron. 2018, 65, 498–509. [CrossRef]
26. MathWorks—Robust Control Toolbox. Available online: https://www.mathworks.com/products/robust.html (accessed on
23 November 2021).
27. Mendoza-Mondragón, F.; Hernández-Guzmán, V.M.; Rodríguez-Reséndiz, J. Robust Speed Control of Permanent Magnet
Synchronous Motors Using Two-Degrees-of-Freedom Control. IEEE Trans. Ind. Electron. 2018, 65, 6099–6108. [CrossRef]
28. Nicola, M.; Nicola, C.-I.; Ionete, C.; S, endrescu, D.; Roman. Improved Performance for PMSM Control Based on Robust Controller
and Reinforcement Learning. In Proceedings of the 26th International Conference on System Theory, Control and Computing
(ICSTCC), Sinaia, Romania, 19–21 October 2022; pp. 207–212.
Sensors 2023, 23, 5799 29 of 29

29. Nicola, C.-I.; Nicola, M.; Selis, teanu, D. Sensorless Control of PMSM Based on Backstepping-PSO-Type Controller and ESO-Type
Observer Using Real-Time Hardware. Electronics 2021, 10, 2080. [CrossRef]
30. Nicola, M.; Nicola, C.-I.; Duţă, M. Sensorless Control of PMSM using FOC Strategy Based on Multiple ANN and Load Torque
Observer. In Proceedings of the International Conference on Development and Application Systems (DAS), Suceava, Romania,
21–23 May 2020; pp. 32–37.
31. MathWorks—Deep Learning Toolbox—Neural Net Fitting. Available online: https://www.mathworks.com/help/deeplearning/
neuralnetfitting-app.html (accessed on 20 April 2020).
32. Nicola, M.; Nicola, C.-I. Improvement of PMSM Control Using Reinforcement Learning Deep Deterministic Policy Gradient
Agent. In Proceedings of the 21st International Symposium on Power Electronics (Ee), Novi Sad, Serbia, 27–30 October 2021;
pp. 1–6.
33. MathWorks—Twin-Delayed Deep Deterministic Policy Gradient Reinforcement Learning Agent. Available online: https://www.
mathworks.com/help/reinforcement-learning/ug/td3-agents.html (accessed on 12 September 2021).
34. Nicola, M.; Nicola, C.-I.; Sacerdot, ianu, D.; Vintilă, A. Comparative Performance of UPQC Control System Based on PI-GWO,
Fractional Order Controllers, and Reinforcement Learning Agent. Electronics 2023, 12, 494. [CrossRef]

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.