This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 1

Reinforcement Learning-based Load Forecasting of

Electric Vehicle Charging Station Using Q-Learning
Technique

Abstract—The electric vehicles (EVs) rapid growth can po- flow technique and a regional charging time-shift technique
tentially lead power grids to face new challenges due to load were employed for forecasting the energy shift at the day-
profile changes. To this end, a new method is presented to time and midnight, respectively. In [15], a simulation based
forecast the EV charging station loads with machine learning
techniques. The plug-in hybrid EVs (PHEVs) charging can be on Monte Carlo was used to predict the EVs uncontrolled
cetergorized into three main techniques (smart, uncoordinated, energy consumption. The authors also took into account EVs
and coordinated). To have a good prediction of the future PHEV uncertainty parameters, for example the start time of charging,
loads in this paper, the Q-learning technique, which is a kind of the duration of charging, the EVs hourly number for battery
the reinforcement learning (RL), is used for different charging swapping, and the travel distance. While simulations with
scenarios. The proposed Q-learning technique improves the fore-
casting of the conventional artificial intelligence techniques such Monte Carlo technique are very precise, the behavior of EV
as the recurrent neural network (RNN) and the artificial neural owners has shown to greatly affect the flexibility and accuracy
network (ANN). Results prove that PHEV loads can accurately of the model. Moreover, learning-based techniques have been
be forecasted by using the Q-learning technique under three investigated to address the critical challenges caused by the
different scenarios (smart, uncoordinated, and coordinated). The behavior of EV owners. For instance, the authors in [16]
simulations of three different scenarios are obtained in the
KERAS open source software to validate the effectiveness and and [17] employed a charging station along with a battery
advantages of the proposed Q-learning technique. storage system in order to inject energy/power back into the
network or defer their demand. The smart PHEVs load demand
Index Terms—Electric vehicle charging stations, machine
learning, Q-learning, ensemble forecasting, load forecasting. forecasting in [18] was explored, where the main objective is
the load profile of daily grid smoothly, while ensuring that
the load demand of each PHEV is satisfied. To predict the
I. I NTRODUCTION PHEVs load demand, a model was only employed to predict

E LECTRIC vehicles (EVs) have largely been penetrated to

power systems because of their great benefits than fossil
fuel vehicles, e.g., higher energy security and lower emissions
the coordinated PHEVs load demand, an ensemble learning
technique was explored in [19]. A novel technique based on
data mining has been investigated in [20] to predict the PHEVs
of greenhouse gases [1]. Many works related to the electric load demand.
vehicle charging stations were done in the literature [2], [3]. To reduce a system computation burden and to have the
By increasing the EVs penetration, the energy management of complicated controllers real-time implementation, the deep
power grid would become more complicating and challenging learning techniques are used more and more to improve the
due to effects of EV on market prices and electricity demands. highly nonlinear systems prediction performance [21]. In the
As a result, accurate EV charging load demand forecasting is literature, to reach the above goals [22], [23], [21], [24],
one of the key concerns from the power grid points of view [4], different machine learning techniques are employed. Artificial
[5], [6], [7]. neural network (ANN) was mainly employed in [25] for
Recent advances in the plug-in hybrid EV (PHEV) are the dataset that does not have a time dependency among
employing the reinforcement learning (RL) for the PHEVs the available datasets. Moreover, the recurrent neural net-
energy management [8]. Moreover, an integration of the PHEV works (RNN) are mostly employed for the dataset that is
to the microgrids by incorporating the forecasting techniques time-dependent. Some examples of RNN techniques are the
was explored in [9]. In addition, implementing the blockchain gated recurrent units (GRU) and the long short-term mem-
technique for the electricity trading was proposed in [10]. ory (LSTM). Additionally, the convolutional neural network
Furthermore, the PHEV energy management was proposed (CNN) is used for the media dataset (two dimensional data).
in [11] with the data-driven techniques for the discharge depth Also, the reinforcement learning (RL) techniques such as Q-
of battery estimations. learning or SARSA are mainly employed to enhance the real-
In the literature, the EV charging stations load forecasting time learning of complicated problems. These techniques are
has widely been investigated. For example, an online ride- based on the action-reward mechanism. Thus, each action that
hailing trip data was used in [12] to forecast the EVs load moves an agent from one state to another can impose a unique
demand. EVs stochastic load demand was studied in [13], reward for the RL technique. In this paper, the Q-learning
where authors employed a queuing theory to model plug-in technique, which is one of the RL techniques, is used for
hybrid electric vehicles (PHEVs) and EVs load demands. To forecasting the PHEV loads. The Q-learning technique can
forecast the PHEVs load demand, a mathematical consumption optimally search between all available actions and states. Thus,
modeling was investigated in [14], where an inverse load the solutions of other machine learning techniques such as

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 2

TABLE I
ANN, RNN, LSTM, GRU, and CNN can be used as initial PHEV T YPES OF D IFFERENT C HARGERS
inputs to the Q-learning technique to improve its performance.
Although the load demand of EVs has been predicted with Charge type Input voltage Maximum power (kW)
different techniques, a general technique has not yet been level 1 120 VAC 1.44
investigated that models all scenarios of load demand of level 2 208-240 VAC 11.5
level 3 208-240 VAC 96
PHEVs, e.g., smart, coordinated, and uncoordinated. There- level 3 (DC) 208-600 VDC 240
fore, this paper is the first attempt to investigate a reinforce-
ment learning technique for predicting the load demand of
smart, coordinated, and uncoordinated PHEVs to the best of C. Smart PHEV Charging
the authors’ knowledge. Thus, three well-known AI techniques
(i.e., artificial neural network (ANN), recurrent neural network In this scenario, PHEVs start charging when the battery
(RNN), and Q-learning) are compared for the PHEV charging SOC is not full and the electricity price is low. Thus, the
load forecasting. The proposed Q-learning technique will use vehicles are started to charge based on mutual interest of both
the initial forecasting of the other two techniques ((ANN) and utilities and owners. The PDF of this scenario can be modeled
(RNN)). as:
The main contribution in this paper is to use a new rein- 1 −1 ts −µ 2

forcement learning technique to find an optimal framework to f (ts ) = √ e( 2 ( ψ ) ) , µ = 1, ψ = 3 (3)

ψ 2π
predict the smart, coordinated, and uncoordinated PHEVs load
demands. This paper is organized as mentioned below: µ is the mean of values. The PHEV battery is started to charge,
Section II discusses a plug-in hybrid electric vehicle charg- when it is connected to the charger. The PHEV daily driven
ing station formulations. Section III describes the machine miles PDF can be modeled as [28]:
learning techniques that are used in this paper. Section IV −(ln(m)−µ)2
1
presents simulation and forecasting results. Finally, Section V f (ts ) = √ e 2ψ 2 , m>0 (4)
concludes the obtained results. mψ 2π
m is the PHEV daily driven distance (mile). The SOC of
II. C HARGE M ODELING OF P LUG - IN H YBRID E LECTRIC PHEV relates to the miles that vehicle is daily driven. More-
V EHICLES over, it is related to its all-electric ranges (AER) [26].
Uncertain parameters such as charging strategy, charging (
0 m > AER
duration, charged PHEVs number, start time, battery capacity, SOC = (5)
AER−m
and the state of charge (SOC) of battery which have great AER × 100% m ≤ AER
impacts on the PHEVs behavior. This paper studies PHEVs
The PHEVs charging duration (tD ) can be formulated as:
three different charging scenarios as: smart, coordinated, and
uncoordinated charging. CBT × (1 − SOC) × DOD
tD = (6)
η×J
A. Uncoordinated PHEV Charging CBT is PHEV battery capacity (kWh). DOD is PHEV bat-
In this scenario, PHEV users leave their home in the tery depth of discharge. η is the charger efficiency. J is PHEV
morning and return in the evening. Moreover, after returning charger rate (kW). Table I summarizes the types of charging,
back home, users start to plug in and charge PHEVs. This time which depend on the charger charging levels [26]. As shown,
could be around 6:00 PM. Thus, the probability distribution the first and second Levels of charging are assigned to the
function (PDF), when the covariance (ψ) is selected as 6.00 PHEVs charging at home. Thus, because of the applications
PM for this scenario, is modeled as [26], [27]: in public transportation, PHEV chargers with the third level
of charging are not discussed in the current paper. Moreover,
1 PHEVs are classified into 4 groups with different shares and
f (ts ) = , b = 19, a = 18, a ≤ ts ≤ b (1)
b−a market characteristics based on Table II.
ts is the charging start time of PHEV. a and b are constant In this paper, the market share of PHEV can be modeled
values. as a discrete distribution. Based on the shares of market, each
PHEV group is selected randomly via a normal distribution.
B. Coordinated PHEV Charging M in CBT + M ax CBT
µCBT = (7)
In this scenario, during off-peak hours PHEVs are plugged 2
in by vehicle owners. More importantly, charging during M ax CBT − M in CBT
evening peak hours are avoided by PHEV users. Therefore, ψCBT = (8)
4
PHEV users postpone the charging time to 9.00 PM, where
M in and M ax are the minimum and maximum values.
the electricity price is low. Consequently, this scenario PDF is
To generate the data of training for forecasting the PHEV
defined as:
charging demand with the proposed reinforcement learning
1 technique, three charging technique simulated data (smart,
f (ts ) = , a ≤ ts ≤ b, a = 21, b = 24 (2) uncoordinated, and coordinated) are obtained by using (1)-(8).
b−a

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 3

TABLE II
D IFFERENT CLASSES OF PHEV

Class Market share Min-Max CBT (kWh)

Micro car 0.2 8-12
Mid-size car 0.3 14-18
Economy car 0.3 10-14
Light truck/SUV 0.3 19-23

Fig. 2. The RNN block diagram.

be used for different applications such as forecasting the

trend of the stock markets, forecasting the renewable energies,
Fig. 1. The ANN block diagram. forecasting the weather conditions, etc. The main difference
between the ANN and the RNN techniques is the line between
the nodes of each hidden layer that is used in the RNN
III. M ACHINE L EARNING T ECHNIQUES FOR PHEV technique. The output of the RNN unit can be found by using
C HARGING L OAD F ORECASTING the following equation,
Principles of two machine learning approaches, i.e., ANN 
and RNN are primarily introduced for predicting the PHEV a = b + U ht−1 + W xt
 t


charging loads in three different scenarios (i.e., smart, unco-
ht = g(at ) (10)
ordinated, and coordinated). Moreover, the main principles of 

the Q-learning technique will be introduced in this section. 
ot = f (c + V ht )
The proposed Q-learning technique uses the optimal solutions
of ANN and RNN as initial inputs. In (10), ht is the memory of the tth RNN unit. b, U , and
W are the bias, the weight of ht−1 , and the weight of xt ,
A. Artificial Neural Network respectively. Moreover, ot is the output of each RNN cell. c
and V are the bias and weight of the output of RNN cell. g
In Fig. 2, a block diagram of the ANN [29] is shown.
and f are the activation functions (softmax, tanh, etc) of the
As shown in this figure, the ANN technique uses three main
memory and the output of the RNN cell, respectively.
layers: one input layer (X), two hidden layers (H), and one
output layer (Y ). As shown in Fig. 2, several nodes are used in
each input, output, and hidden layers of the ANN technique. A C. Q-learning technique
weight (ωij ) is used in each line between all consecutive layers One of the model-free reinforcement learning techniques is
(i (previous layer) and j (current layer)) of the ANN technique Q-learning technique that employs Markov decision process
as shown in Fig. 2. Each output node can be obtained using (MDP) to find the best policy among all available actions [?].
the following equation: Q-learning can be categorized as one of the off-policy tech-
nl−1 niques. Q-learning technique is very similar to the SARSA
X
Ojl = l
σj ( (Oil−1 ωij ) + blj ) (9) for using the reward, the previous state, the current state, and
i=0 available actions for selecting the best future action. The main
where Ojl is the each node output (jth node and the lth layer), equation, which is used for the Q-learning technique to select
Ojl−1 is the output of the (l − 1)th layer and the jth node, the future action is written below,
blj is the jth node bias in the ANN lth layer, σjl is the jth (
node activation function in the lth layer, and nl−1 is the nodes Qnew (st , at ) ← Q(st , at ) + α(rt
number in the (l − 1)th layer. (11)
+λ maxa Q(st+1 , at ) − Q(st , at )).
B. Recurrent Neural Network where s(t) is the current state and a(t) is the current action
Recurrent neural networks (RNN) are used for the time- of the Q-learning technique. Q(st , at ) is the Q-value of action
series dataset [30] to predict the future of a dataset. It can a(t) and state s(t). Qnew (st , at ) is the updated Q-value of

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 4

the state s(t) and the action a(t). Moreover, α is the Q-

learning method learning rate (0 ≤ α ≤ 1). Moreover, λ
is the Q-learning technique discount factor (0 ≤ λ ≤ 1).
In addition, maxa (Q(st+1 , at )) is the optimal future value
estimation. rt is the proposed Q-learning reward for the future
action and the future state. The Q-value initial conditions of
the proposed technique is zero. Algorithm I presents the Q-
learning technique pseudo-code which is employed to find the
optimal solution of the PHEV loads. In each iteration of the
proposed Q-learning technique, the Q-value based on the st
and at is updated.

Algorithm 1 The Proposed Q-Learning Algorithm Pseu-

docode.
Select arbitrarily Q(st , at ) (a)
while iter ≤ the iteration max number do
Select st
while step ≤ the step max number do
1. Select at based on the pseudocode and the Q-values
in (11)
2. Observe st+1 and rt based on at
3. Calculate Qnew (st , at ) using equation (11)
4. st = st+1

end
end

In step 3 of algorithm 1, the best action of the next step is

obtained by using the following equation,

at = arg max(Qt (a)). (12)

a
(b)
Fig. 3. Implementation of the ANN and RNN techniques for the PHEV load
IV. I MPLEMENTATIONS OF THE T HREE M ACHINE forecasting, (a) ANN technique, (b) RNN technique.
L EARNING T ECHNIQUES TO F ORECAST THE PHEV LOADS
A. Implementation of the ANN and RNN Techniques
the next 24-hour time horizon. The reward function which is
To implement the ANN technique for the PHEV load
used in the proposed Q-learning technique is written below,
forecasting, the input and the output data of the ANN units
should appropriately be selected. Because the PHEV load 1
rt (λi ) = (13)
forecasting is a kind of time-series data, the input and output exp (POptimalt − λi (PAN N/RN Nt ))
of the ANN unit should use the previous PHEV load data.
To improve the accuracy of the PHEV load forecasting, the where λi is the random number of ith action. For each time
previous 24-hour PHEV loads are used to predict an one-hour horizon, the proposed technique selects between two available
ahead. Fig. 3 shows the implementations of the ANN and RNN actions (i = 0 and i = 1 mean ANN and RNN techniques, re-
for forecasting the future PHEV load based on the last 24-hour spectively). PAN N/RN Nt is the forecast of power of the ANN
PHEV loads. In Fig. 3, Iki is the kth input PHEV load of ith or RNN technique. Also, POptimalt is the data of the PHEV
day. Also, Iki+1 is the kth prediction of PHEV load for i + 1th for the tth time horizon. In the proposed technique, the pre-
day. dicted results of the ANN and RNN techniques are used and
compared with the real data to forecast the future load of
the PHEV with the RL technique. Thus, in the proposed RL
B. Q-learning technique technique, for the next 24 hour time horizon, 24 optimal ac-
To implement the Q-learning technique for forecasting tions are selected. Thus, t can be a number between 1 and
of the PHEV load, the predictions of the ANN and RNN 24 for the 24-hour time horizon. In algorithm 2, POptimalt ,
techniques for previous days should be used. To predict the PAN Nt , and PRN Nt are the optimal PHEV load of the tth
PHEV load for the next 24-hour time horizon, the Q-learning day, the prediction of the ANN technique for the PHEV load
technique selects the best action based on the results of ANN for the tth day, and the prediction of the RNN technique for
and RNN to find optimal predictions of the PHEV loads for the PHEV load of the tth day, respectively. In algorithm 2,

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 5

TABLE III
p is a random number between 0 and 1. This random num- N ODES NUMBER IN HIDDEN LAYERS OF THE ANN AND RNN
ber improves the ability of the Q-learning technique to search TECHNIQUES FOR THE SENSITIVITY ANALYSIS .
around the optimal points. From algorithm 2, when action is
equal to 1 the reward function is selected based on the RNN Technique 1st H. 2nd H. 3rd H.
technique. Otherwise, the reward function is selected based on ANN-1 200 200 -
RNN-1 300 300 -
the result of the ANN technique. In algorithm 2, rt−AN N/RN N ANN-2 500 500 -
is the reward function based on (13) for the ANN or RNN RNN-2 500 500 -
techniques. ANN-3 300 300 300
RNN-3 300 300 300

Algorithm 2 Pseudocode of the selection of the reward func-

tion in the proposed technique. 80

λ0 = random(0, 1), λ1 = random(0, 1) 60

Real Data
ANN
if (at = 0 & rt (λ0 ) ≤ rt−AN N & rt (λ0 ) ≤ rt−RN N ) then

PHEV load (kW)

RNN
rt = rt (λ0 ) 40 Q-learning

end
if (at = 0 & rt (λ0 ) ≤ rt−AN N & rt (λ0 ) > rt−RN N ) then 20
rt = 0
0
end
if (at = 1 & rt (λ1 ) ≤ rt−RN N & rt (λ1 ) ≤ rt−AN N ) then -20
rt = rt (λ1 ) 2 4 6 8 10 12 14 16 18 20 22 24
end Time(Hour)
if (at = 1 & rt (λ1 ) ≤ rt−RN N & rt (λ1 ) > rt−AN N ) then
Fig. 4. Forecasting of the uncoordinated PHEV charging load (kW) with a
rt = 0 30% PHEV load penetration.
end
Table IV provides critical information such as the mean
square errors (MSE) and the epoch numbers that are used
for different machine learning techniques that are used in this
paper during the training. The MSEs of ANN and Q-learning
techniques are lower (the accuracy is higher) than the RNN
V. C ASE S TUDIES technique. The main reason why the ANN technique has a
higher accuracy than the RNN technique is due to the higher
epoch number that is used during the training of the ANN
To demonstrate the merits and effectiveness of the proposed technique. A 50% PHEV penetration in Fig. 5 illustrates the
machine learning techniques, in this section, three different uncoordinated PHEV hourly predicted loads. As shown in
techniques (ANN, RNN, and Q-learning) are compared under both Fig. 4 and Fig. 5, the charging of the PHEV load starts
different scenarios. The hidden layer number that is used for from 6PM and ends at 6AM. Also, the artificial intelligence
ANN and RNN machine learning techniques is presented in technique has a small MSE (high accuracy) with the 50%
Table III. The input number of the machine learning techniques PHEV load penetration similar to Fig. 4. Contrarily, the RNN
is 24 (i.e., to forecast the charging load of the next hour, the technique, which uses a lower epoch number, has a higher
last 24-hour data is employed). To evaluate the proposed load MSE (lower efficiency) as shown in Table IV. Moreover, the
forecasting model, the open-source KERAS software [31] is proposed Q-learning technique in Algorithm 1, which uses the
used in this paper. Moreover, as discussed in Section II, the results of the ANN and RNN techniques to predict the PHEV
MATLAB is employed to generate the initial data of PHEVs loads, has a lower MSE (a higher accuracy) than the other
for different charging techniques. two techniques. This proves the advantages of the Q-learning
technique in comparison to the ANN and RNN for predicting
the PHEV loads.

B. Load Forecasting of Coordinated PHEVs Charging

A. Load Forecasting of Uncoordinated PHEVs Charging The coordinated PHEVs load demand is predicted in this
subsection. Fig. 6 illustrates the coordinated PHEVs load de-
The uncoordinated PHEV load demands are forecasted by mand, using the ANN, RNN, and Q-learning techniques, when
employing the machine learning techniques such as ANN, the PHEVs penetration is 30%. The three machine learning
RNN, and the proposed Q-learning techniques in this paper. techniques can precisely track the PHEVs real data when
Fig. 4 presents a 30% PHEV penetration forecasted load. 1,000, 3000, and 10000 training iterations are applied to the
Fig. 4 presents the three machine learning techniques to ANN, RNN, and the Q-learning techniques. Both Figs. 6 and
accurately predict the load of PHEVs. 7 illustrate that in the coordinated PHEV charging technique,

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 6

TABLE IV
C OMPARISON OF THE PHEV CHARGING LOAD FORECASTING WITH THE 100
ANN, RNN, AND Q- LEARNING TECHNIQUES (MSE AND E POCH ).
80 Real Data
ANN

PHEV load (kW)

Machine Learning Charging Penetration MSE(kW) Epoch 60 RNN
Q-learning
ANN-1 Uncoordinated 30% 4.30 3000
RNN-1 Uncoordinated 30% 10.65 1000 40
Q-learning-1 Uncoordinated 30% 0.89 10000
ANN-1 Coordinated 30% 8.05 3000 20
RNN-1 Coordinated 30% 10.00 500
0
Q-learning-1 Coordinated 30% 5.83 10000
ANN-1 Smart 30% 7.22 3000
-20
RNN-1 Smart 30% 7.34 500 2 4 6 8 10 12 14 16 18 20 22 24
Q-learning-1 Smart 30% 5.19 10000 Time(Hour)
ANN-2 Uncoordinated 30% 1.66 3000
RNN-2 Uncoordinated 30% 24.88 500 Fig. 6. Coordinated PHEV charging forecasting result (kW), with a PHEV
Q-learning-2 Uncoordinated 30% 1.50 10000 penetration of 30%.
ANN-2 Coordinated 30% 9.42 3000
RNN-2 Coordinated 30% 9.80 500
Q-learning-2 Coordinated 30% 7.20 10000 200
ANN-2 Smart 30% 7.22 3000
RNN-2 Smart 30% 7.34 500 Real data
Q-learning-2 Smart 30% 5.89 10000 150 ANN

PHEV load (kW)

RNN
ANN-3 Uncoordinated 30% 0.0023 1000 Q-learning
RNN-3 Uncoordinated 30% 0.0020 1000
Q-learning-3 Uncoordinated 30% 0.0020 1000 100
ANN-2 Coordinated 30% 1.32 3000
RNN-2 Coordinated 30% 10.86 500
Q-learning-3 Coordinated 30% 0.90 10000 50
ANN-3 Smart 30% 7.11 3000
RNN-3 Smart 30% 7.23 500
0
Q-learning-3 Smart 30% 4.63 10000 2 4 6 8 10 12 14 16 18 20 22 24
Time(Hour)

300 Fig. 7. Coordinated PHEV charging forecasting result (kW), with a PHEV
250
Real Data penetration of 50%.
ANN
RNN
PHEV load (kW)

200 Q-learning

150 the other two techniques. The proposed Q-learning technique

100 has a lower MSE (a higher accuracy) than the ANN technique
50
due to selecting the best policy among the available actions for
each state (each hour). Using a higher data range for training
0
the Q-learning technique can further improve the accuracy of
-50 the proposed technique.
2 4 6 8 10 12 14 16 18 20 22 24
Time(Hour)
D. Sensitivity Analysis of Three Charging Techniques
Fig. 5. Uncoordinated PHEV charging load forecasting (kW), with a PHEV
penetration of 50%. In Figs. 10, 11, and 12, a sensitivity analysis of three
PHEV charging techniques is presented for different node
numbers in hidden layers and the number of hidden layers.
the time of charging process starts at 8PM and continues until Table. III shows the number of hidden layers and nodes that
10AM. As can be seen in Table IV, the Q-learning technique
has a smaller error (a lower MSE) than the ANN and RNN
techniques for forecasting the charging load of the coordinated 50
PHEV. Furthermore, the Q-learning technique result shows
Real Data
less error from the real data in Fig. 6 and 7. 40
ANN
PHEV load (kW)

RNN
30 Q-learning
C. Load Forecasting of Smart PHEVs Charging
20
The smart charging load demand is predicted for both
50% and 30% penetration levels in this section. Fig. 8 and 10
Fig. 9 illustrate that all techniques can accurately predict the
PHEV loads, when 1,000, 3,000, and 10000 training iterations 0
2 4 6 8 10 12 14 16 18 20 22 24
are used by the ANN, RNN, and Q-learning techniques,
Time(Hour)
respectively. However, the error of the RNN technique for
predicting the PHEV load is higher than the ANN and Q- Fig. 8. Smart PHEV charging forecasting result (kW), with a PHEV
learning technique due to using less number of iterations than penetration of 30%.

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 7

200 80
Real data
ANN Real data
150 RNN 60 Q-learning-1
PHEV load (kW)

PHEV load (kW)

Q-learning Q-learning-2
Q-learning-3

100 40

50 20

0 0
2 4 6 8 10 12 14 16 18 20 22 24 2 4 6 8 10 12 14 16 18 20 22 24
Time(Hour) Time(Hour)

Fig. 9. The smart PHEV charging forecasting result (kW), with a PHEV Fig. 11. Sensitivity analysis of Q-learning technique for the uncoordinated
penetration of 50%. PHEV charging (kW), with a PHEV penetration of 30%.

100
learning techniques with a higher number of nodes and hidden
80 Real data layers.
Q-learning-1
PHEV load (kW)

Q-learning-2
60 Q-learning-3
50

40
40

PHEV load (kW)

Real data
20 Q-learning-1
30 Q-learning-2
Q-learning-3
0
2 4 6 8 10 12 14 16 18 20 22 24 20
Time(Hour)
10
Fig. 10. Sensitivity analysis of Q-learning for the coordinated PHEV charging
(kW), with a PHEV penetration of 30%. 0
2 4 6 8 10 12 14 16 18 20 22 24
Time(Hour)

are used for forecasting the PHEV loads by using the ANN Fig. 12. Sensitivity analysis of Q-learning technique for the smart PHEV
and RNN techniques. Table IV illustrates the accuracy of the charging (kW), with a PHEV penetration of 30%.
three different machine learning techniques for three different
charging scenarios. Fig. 10 illustrates the forecasting accuracy
of the Q-learning technique when the node numbers in hidden E. Discussion
layers change from 200 to 500 and the number of hidden layers The flexibility, speed, and accuracy are discussed for three
is changed from 2 to 3 for both ANN and RNN techniques deep learning techniques in order to forecast the PHEV
as shown in Table III. As proven in Fig. 10, the errors of charging loads.
the Q-learning technique are lower for the coordinated PHEV 1) Higher speed: Utilizing two machine learning tech-
charging technique, when a higher number of nodes and niques such as ANN and RNN can enhance the Q-learning
hidden layers are used to predict the PHEV loads. The MSEs technique ability to achieve a real-time implementation of
of ANN, RNN, and Q-learning techniques are summarized in the PHEV load forecasting. The main concern of the three
Table IV. As proven, the forecasting accuracy improves with machine learning techniques in this paper is the training speed
the increasing number of nodes and hidden layers. of the machine learning techniques. If the imported data
Fig. 11 and Table IV illustrate that the sensitivity analysis of the PHEVs for ANN and RNN are adequately trained,
of the uncoordinated PHEV charging with ANN, RNN, and then the trained ANN and RNN networks can be employed
Q-learning techniques when a 30% of penetration is applied. as the input data for a real-time implementation of the Q-
Fig. 11 shows the uncoordinated PHEV load charging of learning technique to forecast. As a result, using the long term
the Q-learning technique. As shown, the accuracy of the Q- information of the ANN and RNN techniques can significantly
learning technique enhances when more number of nodes improve the ability of the Q-learning technique to reach a real-
and hidden layers are used for training the ANN and RNN time implementation.
techniques. Also, Fig. 12 illustrates the Q-learning technique 2) Higher accuracy: The three machine learning techniques
sensitivity analysis of the smart PHEV load forecasting when in this paper selects the best technique for predicting the
different nodes and hidden layers are used for the 30% of charging of the PHEV loads, it is vital to explore all available
penetration. Thus, a higher number of nodes and hidden layers deep learning techniques. Furthermore, it can conclude from
of ANN and RNN techniques can improve the ability of simulation results that large deviations from the initial charg-
the proposed Q-learning technique to predict the PHEV load. ing points can cause large error of predictions of the machine
Finally, Table IV has a better accuracy for ANN, RNN, and Q- learning techniques. This can cause error on the prediction of

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 8

the Q-learning technique which uses the imported information [3] A. Moeini and S. Wang, “The state of charge balancing techniques for
of the ANN, RNN, etc to predict the optimal PHEV charging. electrical vehicle charging stations with cascaded h-bridge multilevel
converters,” in 2018 IEEE Applied Power Electronics Conference and
Moreover, having a higher epoch number (iterations) can Exposition (APEC), 2018, pp. 637–644.
mitigate the errors of forecasting of the machine learning [4] Z. Liu, F. Wen, and G. Ledwich, “Optimal planning of electric-vehicle
technique as proven in Table IV. charging stations in distribution systems,” IEEE Transactions on Power
Delivery, vol. 28, no. 1, pp. 102–110, 2012.
3) Flexibility: As proven in the simulation results, any [5] P. Zhang, K. Qian, C. Zhou, B. G. Stewart, and D. M. Hepburn, “A
charging techniques that are employed for the modeling of the methodology for optimization of power systems demand due to electric
PHEVs can be trained by using the machine learning tech- vehicle charging load,” IEEE Transactions on Power Systems, vol. 27,
no. 3, pp. 1628–1636, 2012.
niques. Therefore, any complicated load charging condition [6] R.-C. Leou, C.-L. Su, and C.-N. Lu, “Stochastic analyses of electric
can flexibly be forecasted by increasing the iteration numbers vehicle charging impacts on distribution network,” IEEE Transactions
of training of the machine learning techniques. This leads to on Power Systems, vol. 29, no. 3, pp. 1055–1063, 2013.
[7] Y. Zheng, Z. Y. Dong, Y. Xu, K. Meng, J. H. Zhao, and J. Qiu,
have a high accuracy of the PHEV charging load forecasting. “Electric vehicle battery charging/swap stations in distribution systems:
Moreover, using different combinations of nodes and hidden comparison study and optimal planning,” IEEE Transactions on Power
layers of ANN and RNN techniques can enhance the forecast- Systems, vol. 29, no. 1, pp. 221–229, 2013.
ing of the PHEV loads for coordinated, uncoordinated, and [8] T. Liu, X. Hu, W. Hu, and Y. Zou, “A heuristic planning reinforce-
ment learning-based energy management for power-split plug-in hybrid
smart charging. In addition, having more accurate predictions electric vehicles,” IEEE Transactions on Industrial Informatics, vol. 15,
of the PHEV load charging can enhance the ability of the no. 12, pp. 6436–6445, Dec 2019.
Q-learning technique to select an optimal policy between all [9] C. Chen and S. Duan, “Optimal integration of plug-in hybrid electric
vehicles in microgrids,” IEEE Transactions on Industrial Informatics,
available actions. vol. 10, no. 3, pp. 1917–1926, Aug 2014.
[10] J. Kang, R. Yu, X. Huang, S. Maharjan, Y. Zhang, and E. Hossain,
“Enabling localized peer-to-peer electricity trading among plug-in hybrid
VI. C ONCLUSION electric vehicles using consortium blockchains,” IEEE Transactions on
In this paper, a Q-learning technique based on the artificial Industrial Informatics, vol. 13, no. 6, pp. 3154–3164, Dec 2017.
[11] S. Xie, S. Qi, and K. Lang, “A data-driven power management strategy
neural network (ANN) and recurrent neural network (RNN) for plug-in hybrid electric vehicles including optimal battery depth of
was proposed for the plug-in hybrid electric vehicle load discharging,” IEEE Transactions on Industrial Informatics, pp. 1–1,
forecasting. Different kinds of PHEVs, i.e., smart, coordinated, 2019.
and uncoordinated, were used. The results in this paper proved [12] Q. Xing, Z. Chen, Z. Zhang, X. Huang, Z. Leng, K. Sun, Y. Chen,
and H. Wang, “Charging demand forecasting model for electric vehicles
that the Q-learning technique that uses the information of based on online ride-hailing trip data,” IEEE Access, vol. 7, pp. 137 390–
the ANN and RNN can accurately predict the PHEV load 137 409, 2019.
charging. [13] M. Alizadeh, A. Scaglione, J. Davies, and K. S. Kurani, “A scalable
stochastic model for the electricity demand of electric and plug-in hybrid
As proven in simulation results, the Q-learning technique vehicles,” IEEE Transactions on Smart Grid, vol. 5, no. 2, pp. 848–860,
can forecast the load of the PHEVs more precisely than the March 2014.
ANN and RNN techniques. From Table IV, it can conclude [14] F. Koyanagi and Y. Uriu, “A strategy of load leveling by charging and
discharging time control of electric vehicles,” IEEE Transactions on
that under the PHEV charging worst scenario (i.e., smart Power Systems, vol. 13, no. 3, pp. 1179–1184, Aug 1998.
charging), the Q-learning technique has shown more than [15] Q. Dai, T. Cai, S. Duan, and F. Zhao, “Stochastic modeling and
50% improvement compared to conventional ANN and RNN forecasting of load demand for electric bus battery-swap station,” IEEE
Transactions on Power Delivery, vol. 29, no. 4, pp. 1909–1917, Aug
techniques. Contrarily, as proven, using higher number of 2014.
iterations by the ANN technique can increase the accuracy [16] M. A. Ortega-Vazquez, F. Bouffard, and V. Silva, “Electric vehicle
(MSE) of the ANN technique than the RNN technique to aggregator/system operator coordination for charging scheduling and
services procurement,” IEEE Transactions on Power Systems, vol. 28,
predict the PHEV load. no. 2, pp. 1806–1815, May 2013.
Overall, the proposed Q-learning technique can track the [17] Q. Dai, T. Cai, S. Duan, and F. Zhao, “Stochastic modeling and
PHEV load with a higher speed, a higher accuracy, and forecasting of load demand for electric bus battery-swap station,” IEEE
Transactions on Power Delivery, vol. 29, no. 4, pp. 1909–1917, Aug
with a more flexibility than the ANN and RNN techniques. 2014.
Furthermore, as proven in the case studies, different hidden [18] M. C. Kisacikoglu, F. Erden, and N. Erdogan, “Distributed control of
layer numbers, epochs number (iterations), and different nodes pev charging based on energy demand forecast,” IEEE Transactions on
number can improve significantly the forecasting of the PHEV Industrial Informatics, vol. 14, no. 1, pp. 332–341, Jan 2018.
[19] C. Gómez-Quiles, G. Asencio-Cortés, A. Gastalver-Rubio, F. Martı́nez-
charging loads. Álvarez, A. Troncoso, J. Manresa, J. C. Riquelme, and J. M. Riquelme-
In future, the proposed forecasting techniques can be applied Santos, “A novel ensemble method for electric vehicle power consump-
to a large scale power grids with a more complex PHEV load tion forecasting: Application to the spanish system,” IEEE Access, vol. 7,
pp. 120 840–120 856, 2019.
conditions. [20] E. S. Xydas, C. E. Marmaras, L. M. Cipcigan, A. S. Hassan, and N. Jenk-
ins, “Electric vehicle load forecasting using data mining methods,” in
IET Hybrid and Electric Vehicles Conference 2013 (HEVC 2013), Nov
R EFERENCES 2013, pp. 1–6.
[1] E. Taherzadeh, S. Javadi, and M. Dabbaghjamanesh, “New optimal [21] J. Mackenzie, J. F. Roddick, and R. Zito, “An evaluation of htm and
power management strategy for series plug-in hybrid electric vehicles,” lstm for short-term arterial traffic flow prediction,” IEEE Transactions
International Journal of Automotive Technology, vol. 19, no. 6, pp. on Intelligent Transportation Systems, vol. 20, no. 5, pp. 1847–1857,
1061–1069, 2018. May 2019.
[2] A. Moeini, H. Zhao, and S. Wang, “A current-reference-based selective [22] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural
harmonic current mitigation pwm technique to improve the performance computation, vol. 9, no. 8, pp. 1735–1780, 1997.
of cascaded h-bridge multilevel active rectifiers,” IEEE Transactions on [23] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation
Industrial Electronics, vol. 65, no. 1, pp. 727–737, 2018. of gated recurrent neural networks on sequence modeling,” arXiv, 2014.

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2990397, IEEE
Transactions on Industrial Informatics

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 9

[24] Q. Li, F. Wang, J. Wang, and W. Li, “Lstm-based sql injection detection
method for intelligent transportation system,” IEEE Transactions on
Vehicular Technology, vol. 68, no. 5, pp. 4182–4191, May 2019.
[25] J. Schmidhuber, “Deep learning in neural networks: An overview,”
Neural networks, vol. 61, pp. 85–117, 2015.
[26] M.-A. Rostami, A. Kavousi-Fard, and T. Niknam, “Expected cost
minimization of smart grids with plug-in hybrid electric vehicles us-
ing optimal distribution feeder reconfiguration,” IEEE Transactions on
Industrial Informatics, vol. 11, no. 2, pp. 388–397, 2015.
[27] K. Qian, C. Zhou, M. Allan, and Y. Yuan, “Modeling of load demand
due to ev battery charging in distribution systems,” IEEE Transactions
on Power Systems, vol. 26, no. 2, pp. 802–810, 2010.
[28] G. Li and X.-P. Zhang, “Modeling of plug-in hybrid electric vehicle
charging demand in probabilistic power flow calculations,” IEEE Trans-
actions on Smart Grid, vol. 3, no. 1, pp. 492–499, 2012.
[29] J. M. Zurada, Introduction to artificial neural systems. West Publishing
Company St. Paul, 1992, vol. 8.
[30] R. Hecht-Nielsen, “Theory of the backpropagation neural network,” in
Neural networks for perception. Elsevier, 1992, pp. 65–93.
[31] F. Chollet et al., “Keras,” https://keras.io, 2015.

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on May 06,2020 at 20:49:46 UTC from IEEE Xplore. Restrictions apply.