1

Enhancing Smart Grids with Internet of Energy:

Deep Reinforcement Learning and Convolutional
Neural Network
Ali Mohammadi Ruzbahani
University of Calgary
Email: [email protected]
arXiv:2405.13831v2 [cs.CE] 25 May 2024

Abstract—The increasing demand for electricity, coupled with [10], [11]. Furthermore, a comprehensive scheduling program
the rise in greenhouse gas emissions, necessitates the integration is required to increase customers’ monetary profit, making
of Renewable Energy Sources (RESs) into power grids. However, RESs more affordable. Finally, security is also a concern that
the fluctuating nature of RESs introduces new challenges in
energy management. The Internet of Energy (IoE) framework needs to be handled since the growth of advanced information
provides a solution by enabling real-time monitoring, dynamic and communication technologies makes the network more
scheduling, and enhanced energy routing. This paper proposes vulnerable to malicious physical and cyber-attacks [12], [13].
a comprehensive approach to optimizing energy management In a nutshell, the newly emerged challenges need to be dealt
in smart grids using Deep Reinforcement Learning (DRL) and with through novel techniques that are precise and high-speed
Convolutional Neural Networks (CNN). The research focuses on
three main objectives: optimizing operation scheduling, improv- besides self-improvement capability for the upcoming modi-
ing energy routing, and enhancing cyber-physical security. A fications. Conventional Energy Management Systems (EMSs)
DRL-based scheduling algorithm is developed to manage energy are not practical henceforward due to transformation in the
components effectively, while an optimized energy routing algo- network’s topology and load curve, caused by penetration of
rithm ensures efficient electricity flow. Additionally, a security new resources, besides existing massive data generated by
framework utilizing Long Short-Term Memory (LSTM) and
CNN is proposed to detect False Data Injection (FDI) attacks and smart tools. As a complex cyber-physical system, the Internet
electricity theft. The proposed methods aim to improve energy of Energy (IoE) can provide an energy management framework
efficiency, reduce costs, and ensure the security of IoE-enabled that facilitates the accommodation and coordination of RESs,
power systems. This research bridges existing gaps by addressing providing real-time monitoring via embedded bidirectional
the dynamic and complex nature of modern energy networks. The communication systems. IoE is a cloud-based technology that
integration of these advanced technologies promises significant
advancements in the reliability and efficiency of smart grids. integrates the power system with embedded metering, commu-
Ultimately, this work contributes to the development of a sus- nication, sensing, and networking capabilities [14], delivering
tainable and secure energy future. a bidirectional information structure. This is vital in an Energy
Index Terms—Smart Grids, Attack Surface, Physical Interface, Management System (EMS) to distribute electricity safely and
AI, ML, RL. efficiently from RESs to the system or end-users [11], [15].
Operational Scheduling (OS) is one of the main objectives
of the EMSs, which can be fulfilled through IoE. Scheduling
I. I NTRODUCTION the operation, maintenance, and transmission planning plays
Electricity demand is steadily rising since industrial and a leading role in efficient EMS. Optimized OS transforms
economic development heavily relies on electrical energy [1], consumers into prosumers who can participate in the energy
[2]. During the past ten years, electricity demand is increased market by selling their surplus electricity while maximizing
uninterruptedly by an average of 3.1% annually, which resulted the users’ economic profit. It also accelerates the utilization
in growing greenhouse gas (GHG) emissions and increases of RESs, which leads to numerous technical and economic
the need for new energy resources [3], [4]. According to advantages [16], [17]. In addition to facilitating OS, IoE
the Energy Information Administration (EIA), 62% of elec- also offers the opportunity for energy routing optimization
tricity production in the United States supplied by fossil to improve stability and reliability and to increase profit in
energies, and 20% average global growth has happened for the power network. Due to the RESs, multiple energy hubs
these resources. Since fossil fuels are expensive and pollute, are connected via energy routers to build an energy network,
Renewable Energy Sources (RESs) are utilized dramatically which results in the genesis of Virtual Power Plants (VPPs),
[5]–[7]. High penetration of RESs brings up new challenges one of the central concepts for improving energy efficiency
due to supply fluctuation, uncertainties imposed by the nature [8]. Energy router is a fundamental component in EMS that
of RESs and decentralized topology that originates from the regulates the direction and amount of electricity and optimizes
wide geographical distribution of the energy resources. [8], [9]. the energy flow among all devices [9]. Therefore, optimizing
Also, real-time monitoring of energy flow and dealing with big energy routing in IoE needs to be considered as a crucial
data generated by information infrastructures are vital to en- challenge in energy management. Aside from numerous advan-
hance the network’s energy efficiency, reliability, and stability tages of employing IoE in EMS, this technology is vulnerable
 2

to malicious attacks [18]–[20]. The distributed pattern of IoE, hybrid electric vehicles (PHEVs) is introduced. Unfortunately,
which allows users to interact and exchange information and the RESs are ignored in this study, and the utilized Support
energy without central control, leads to many security and Vector Machine (SVM) algorithm can assign a binary function
privacy challenges [21], [22]. Security concerns are not limited to the devices. Furthermore, the centralized decision-making
to cyber layers since the system may be physically manipu- structure that is employed may result in decreasing efficiency
lated/damaged for electricity theft or sabotages. In summary, and arduous recovery after disasters or attacks. A two-stage
improving energy efficiency in the power network via the device scheduling with dynamic programming is proposed
IoE concept is facing three main challenges that should be by [32]. In this study, only major renewable energy stations
addressed: optimal OS, optimal energy routing in the different are considered, with no scheduling at the home level. None
layers, and cyber/physical security concerns. Consequently, of the previous works included comprehensive scheduling
three main goals are defined for this research proposal. First, considering both energy units and home appliances to find
an operation scheduling algorithm is optimized using Deep surplus energy. Almost all of the previous works consider both
Reinforcement Learning (DRL). Next, the same method is consumers and prosumers with the same role in the market as
applied for routing energy at three different layers, including price takers, resulting in a non-transparent market since a large
smart home, Home Area Network (HAN), and grid layers. share of the network belongs to the household users. An intel-
Finally, an attack detection framework is proposed to detect ligent scheduling program would make users prone to act as a
False Data Injection (FDI) attacks and electricity theft using price maker in the electricity market, leading to a competitive
Long Short-Term Memory (LSTM) and Convolutional Neural marketplace. Dealing with the uncertainties of RESs, handling
Network (CNN) [23], [24]. sudden changes in the consuming or generating patterns, and
computation complexity also need to be addressed. After
developing a scheduling algorithm that satisfies all technical
II. P ROBLEM S TATEMENT AND M OTIVATION
and economic goals at the residential level, the next step is
The origination of IoE emanates from the high penetration optimizing the energy route among all devices in a smart home
of RESs across the power grids and can be used for energy and the HANs. In [33] a hierarchical optimization method
management enhancement. IoE provides a decentralized struc- for ERs is proposed to reduce the complexity of centralized
ture that facilitates accommodation and integration of RESs optimal dispatch on large-scale systems. The consideration
besides increasing demand-supply reliability. Real-time mon- level is higher than the sub-distribution network, where there is
itoring that is enabled through bidirectional communication no routing algorithm for the residential section. Also, storage
is another significant advantage of this cloud-based network. and Electric Vehicles (EVs) effects have been ignored. An
Finally, automation controls implemented in an IoE-based energy router design with an optimized algorithm that covers
power network improve energy efficiency, leading to profit smart homes, HANs, and grid levels at the same time is
maximization [19], [25]. The residential sector, including both still absent. Moreover, the desired algorithm that can perform
consumers and prosumers, is known as the most important in online and offline mode has not been studied yet, and
sector in the IoE networks and economic goals are one of islanding mode identification should also be investigated. IoE
the main motivations for this sector [26]. IoE facilitates han- relies on two-way communication with a fully interconnected
dling challenges associated with OS optimization to improve Advanced Metering Infrastructures (AMIs) network, which
energy efficiency, profit maximization, and decreasing GHGs makes the system vulnerable to cyber and physical malicious
emissions. Besides that, IoE can be used to enhance energy activities. For instance, a cyber-attack on the Supervisory
routing optimization. As the core of IoE, an Energy Router Control and Data Acquisition (SCADA) system of a regional
(ER) adjusts the energy route dynamically among the devices electricity distribution company in Ukraine led to a power
considering technical and economic constraints to minimize outage for 225,000 customers, which caused massive technical
energy loss and maximize profit [27], [28]. Consequently, opti- and economic damages [34], [35]. Also, physical tampering
mizing electricity routing is vital, either made for a single user and bypassing are responsible for 20% of the electricity lost
or at the Home Area Network (HAN) level. Despite the above- in India [36], [37]. Therefore, guaranteeing security comes
mentioned advantages, IoEs are vulnerable to cyber-attacks with high priority in IoE-enabled power systems. In [38], a
due to the broad range of bidirectional interconnection among defence mechanism based on an interval state predictor is
installed smart devices. Therefore, to take advantage of this proposed, which can mitigate the effect of malicious attacks.
technology, assuring security against malicious cyber/physical However, the proposed method does not consider the storage,
attacks is required [29]. Several investigations have been EVs and DC busses in the ERs. The security challenges of
conducted on the application of IoE during the last decade, integrating fog computing into the IoE are studied in [22],
but the challenges mentioned above have not been addressed focusing on collision attacks. Still, FDI or physical attacks
well. [30], [31] proposed a real-time power scheduling method like AMIs manipulation and bypassing are not considered.
based on the structural design of the IoE. The suggested Therefore, developing a comprehensive algorithm for cyber
approach has a high computational burden and highly relies and physical attack detection in different layers of IoE is
on regional zones. Also, there is no specific planning for the necessary.
home appliances operation, which is necessary to discover
surplus electricity for market trading and profit maximization.
In [16], a scheduling method for smart homes and plug-in
 3

III. R ESEARCH O BJECTIVES development of a sophisticated energy routing algorithm is the

Three main objectives are specified for this research pro- second key objective of this research proposal. This algorithm
posal, including optimizing OS, designing an optimized ER, will leverage advanced mathematical models and computa-
and improving the cyber-physical security of the IoE. The OS tional techniques to handle the complexity of modern energy
algorithm makes adaption with IoE-based network profitable systems. By integrating real-time data and predictive analytics,
for both prosumers (by maximizing profit) and utilities (by the algorithm can dynamically adjust to changes in demand
minimizing cost). Moreover, consumers are prone to reduce and supply, thereby optimizing performance continuously. En-
the bill amount via flatting the load curve. Energy routing ergy routing optimization is a pivotal component of managing
is the next step for energy management utilizing IoE, which modern energy systems. It requires a nuanced approach that
must be done in both users and grid-level in offline and online balances technical constraints, economic considerations, and
modes. Lastly, cyber-physical security is needed to assure the customer preferences. The proposed research will focus on
system’s security via FDI and physical attacks detection. The creating an advanced energy routing algorithm that enhances
main objectives of this research proposal are as follows: the efficiency and profitability of energy distribution. By
doing so, it will contribute significantly to the development of
smarter, more resilient energy networks capable of meeting the
A. Optimizing operation scheduling of energy components evolving demands of today’s energy landscape. This research
The leading goal of this research is to address the chal- aims to deliver innovative solutions that can be applied in
lenges originated from the aggregation of RESs via IoE for real-world scenarios, providing a substantial impact on the
efficient energy management while reducing the cost and efficiency and sustainability of energy systems.
GHGs emissions and establishing a competitive market. By
developing and maintaining a scheduling algorithm, on the C. Cyber-Physical attack detection
one hand, all consumers make a profit. These earnings come As previously highlighted, Internet of Energy (IoE) net-
from incentives and tariff adjusting (for shifting consumption works are susceptible to cyber-attacks, including false data
to off-peak hours), besides selling the surplus energy to the injection (FDI) attacks, as well as physical tampering. These
market. On the other hand, utilities can reduce the operation, threats can severely impact the performance of scheduling
transmission, and maintenance costs, reducing the need for and routing algorithms, which rely heavily on the accuracy of
new investigations. Ultimately, this research’s first objective is the data they receive. Consequently, ensuring the security of
to develop a scheduling algorithm to increase the customer’s these networks is crucial. The third objective of this research
profit by managing their consumption and trading identified proposal is to conduct a comprehensive security analysis aimed
surplus electricity. at safeguarding data integrity and protecting against cyber-
physical attacks. This objective will involve developing robust
B. Energy routing optimization mechanisms to detect and mitigate potential threats, thereby
enhancing the resilience and reliability of IoE systems [36].
Once the operation times for each energy unit in the
system have been scheduled, the subsequent critical step is
optimizing the energy routing. This process involves using an D. Research Contributions
energy routing algorithm to plan the amount and direction of The ultimate goals of this research are to develop a com-
electricity flow, taking into account the type of power, whether prehensive framework that includes optimized scheduling and
alternating current (AC) or direct current (DC). This planning routing algorithms along with an advanced attack detection
must adhere to both technical and economic constraints to mechanism to identify FDI attacks and electricity theft. To
ensure efficient and cost-effective energy distribution. achieve these objectives, two Deep Reinforcement Learning
The primary goal of the energy routing algorithm is to (DRL)-based algorithms will be developed. These algorithms
optimize the distribution of electricity across the network. This will address the first two objectives by mitigating the un-
involves determining the most efficient pathways for energy certainties associated with Renewable Energy Sources (RES)
to travel from its generation points to its various consumption and reducing the complexity and computational burden of the
points. The algorithm must consider several factors, such as the problem. Given the critical importance of data accuracy in
capacity of transmission lines, the demand at different nodes, ensuring the reliable performance of scheduling and routing al-
and the cost associated with transmitting energy over various gorithms, the research proposes the use of Convolutional Neu-
distances. Additionally, it must address the differences in ral Networks (CNN) and Long Short-Term Memory (LSTM)
handling AC and DC power, which have distinct characteristics networks for detecting FDI attacks. These advanced machine
and requirements. Once the initial routing plan is created, it is learning techniques are well-suited for analyzing complex data
sent back to the scheduling algorithm. Here, alternative routing patterns and identifying anomalies that may indicate a cyber-
and scheduling options are evaluated based on customer pref- attack. Furthermore, a Deep Convolutional Neural Network
erence levels. This iterative process ensures that the final plan (DCNN) model is proposed for detecting electricity theft.
not only meets technical and economic criteria but also aligns The DCNN model will leverage its deep learning capabilities
with customer needs and priorities. The aim is to find the best to accurately identify patterns associated with unauthorized
trade-off between scheduling and routing to maximize overall electricity usage, thereby preventing financial losses and en-
system profit while maintaining reliability and efficiency. The suring the integrity of the energy distribution system. The
 4

contributions of this research are multifaceted and significant. framework represents a significant advancement in energy
By developing optimized scheduling and routing algorithms, management. It not only enhances the efficiency of energy
the research aims to enhance the efficiency and reliability use but also empowers consumers to become active partic-
of energy systems. The introduction of sophisticated attack ipants in the energy market. By optimizing the scheduling
detection mechanisms, using cutting-edge machine learning of energy resources and shifting loads to off-peak hours, the
techniques, will ensure the security and integrity of the data, framework reduces costs, increases the utilization of renewable
safeguarding IoE networks from both cyber and physical energy, and supports the development of a more resilient
threats. This comprehensive approach will not only improve and sustainable energy system. This research aims to provide
the performance of energy systems but also contribute to the a robust solution that addresses the current challenges in
development of more resilient and secure IoE infrastructures, energy management and contributes to the evolution of smart,
capable of meeting the evolving challenges of the modern en- integrated energy networks.
ergy landscape. The research contribution can be summarized
as follows: A. Developing an energy routing algorithm
via DRL considering technical and economic constraints
IV. D EVELOPING A DRL- BASED SCHEDULING Routing optimization extremely relies on OS since the amount
FRAMEWORK FOR TECHNO - ECONOMIC IMPROVEMENT IN and direction of energy that needs to be sent or received
THE POWER NETWORK among devices is planned based on the scheduling algorithm’s
As the first major contribution, this research proposal in- results. Also, the transmission constraints that are notified
troduces a Q-Learning based framework, an off-policy Re- by utilities must be satisfied. As the second contribution, a
inforcement Learning (RL) algorithm, to optimize operation DRL based algorithm is proposed to deal with the complexity
schedules within diverse environments and under various con- of the problem and minimize the negative impact of power
straints. These environments include home appliances, energy generation and consumption uncertainty at both user-level
resources (such as generation and storage units), and the tech- and grid levels. Furthermore, the proposed DRL-based OS
nical, economic, and communication layers that interconnect algorithm is capable of handling islanding mode challenges
them. The proposed framework aims to create an optimal besides offline working ability.
scheduling program that transforms traditional consumers into
prosumers by shifting the operation of flexible loads to off- B. Cyber-physical attack detection guarantying information
peak hours. This shift not only reduces energy costs for correctness
households but also fosters a competitive market where even To ensure the security of the IoE, an LSTM algorithm is
those without their own energy generation units can participate proposed to detect FDI attacks in time series data, combined
effectively. The scheduling of Renewable Energy Sources with a CNN for feature extraction and correlation identification
(RESs), fixed storage systems, and Electric Vehicles (EVs) among different types of data to deal with uncertainties.
is contingent on their specific limitations and the overall con- Additionally, an Ensemble Deep Convolutional Neural Net-
ditions of the interconnected environments. These conditions work (EDCNN) algorithm is proposed for Electricity Theft
include the dynamic variations in energy supply and demand, Detection (ETD) caused by AMIs’ physical manipulation. As
as well as the technical and economic constraints that can fluc- the first layer of the model, a random under bagging technique
tuate over time. By leveraging Deep Reinforcement Learning is applied to deal with the imbalance data, and then deep
(DRL), the proposed framework enables consumers to act as CNNs are utilized on each subset. Finally, a voting system
prosumers. This is achieved by strategically shifting flexible is embedded in the last part.
loads to off-peak periods, thereby lowering energy bills and
allowing for the trading of surplus energy. In practical terms,
the Q-Learning based framework optimizes the scheduling C. Significance of the Study
process by continuously learning and adapting to changes Electrical energy plays a crucial role in this era and as a
in the environment. It considers the real-time status and high demand type of energy. Utilizing RESs is vital due to
capabilities of home appliances, energy generation and storage global warming caused by GHGs emissions originated from
units, and the overall energy market conditions. By doing so, it using fossil fuels to generate electricity. High penetration of
identifies the most cost-effective and efficient times to operate RESs requires modern energy management systems to deal
various devices and systems, ensuring that energy consumption with technical and economic constraints. IoE is the enabler
is aligned with periods of lower demand and higher availability for modern energy management, which provides novel energy
of renewable energy. Moreover, the framework’s adaptability trading features besides real-time monitoring and dynamic
to varying technical and economic constraints ensures that scheduling. After reviewing all the related studies, this pro-
it remains effective even as external conditions change. This posal determined the research gaps of application of IoE for
adaptability is crucial in a dynamic energy landscape where energy management, including optimal OS, optimal energy
supply and demand can be highly variable. The use of DRL routing, security, and energy trading. Consequently, a com-
techniques allows the system to handle these complexities prehensive scheduling program is proposed to optimize the
and make informed decisions that maximize both individual energy components’ operation scheduling considering techno-
and collective benefits. The Q-Learning based optimization economic constraints besides achieving the best energy routing
 5

plan. Also, a reliable platform is required to ensure the off-peak hours, households can reduce energy costs and par-
correctness of the information used in the scheduling and ticipate in energy trading markets, even without their own
routing algorithms. Therefore, an attack detection algorithm energy generation units. This shift not only benefits individual
is developed detecting FDI attacks and electricity theft [19], households by lowering bills but also contributes to a more
[39]. balanced and efficient energy grid. The IoE concept, supported
by RL approaches, offers a powerful framework for improving
energy efficiency through optimized scheduling and real-time
V. P ROPSOED M ETHODOLOGY
management. By leveraging the strengths of RL algorithms,
After reviewing the various studies and reports, along this research aims to enhance the operational efficiency of
with industrial and market demand, the problem statements smart homes and integrate them more effectively into the
have been recognized. A comprehensive literature review then broader energy system. This approach promises to deliver
demonstrated the research gaps and challenges that have led significant economic and environmental benefits, paving the
to defining research objectives and contributions. Next, the way for a more sustainable and resilient energy future.
best solution has been suggested for every goal where several
datasets have been collected to train, evaluate, and test the
proposed algorithms. The proposed solutions are as follows:

A. Deep Reinforcement Learning based scheduling algorithm

The Internet of Energy (IoE) is a concept that enhances en-
ergy management by integrating renewable energy sources and
distributed generators (DGs). It facilitates a bidirectional flow
of information and real-time monitoring, leading to significant
improvements in energy efficiency. The operation scheduling
of energy units in smart homes is a critical component of
IoE because the residential sector accounts for a substantial
portion of global energy consumption. Effective scheduling
can optimize energy use, reduce costs, and enhance the overall
efficiency of energy systems. Reinforcement Learning (RL)
approaches are particularly well-suited for this task due to
their ability to generalize and resist bias. These algorithms can
handle incomplete information by extracting useful data from
Fig. 1. Reinforcement Learning algorithm Design
the available inputs. Moreover, RL algorithms typically have a
fast convergence rate to approximate solutions, which reduces
their complexity and computational burdens compared to other
machine learning algorithms. This efficiency makes RL an Aside from their advantages, RL algorithms suffer from
attractive choice for managing the dynamic and often uncertain dimensionality problems similar to other ML algorithms [4].
environments associated with energy systems. As illustrated in Besides that, assume a Markovian property for the environ-
Figure 6, an RL algorithm comprises two main components: ment, which is not the case all the time. To address these
an agent and an environment that interact frequently. The drawbacks, DRL is introduced. Utilizing DRL improves the
agent selects actions from an action space and sends signals control strategy and optimization process via feature extrac-
to the environment, learning how to behave in each state. tion and alleviate the dimensionality problem by ignoring
Each time an action is taken, the environment generates a redundant information due to the uncertainties. This research
reward value. This reward, along with the new state, is sent presents a DRL based framework to optimize scheduling in
back to the agent for the next iteration. The reward indicates the IoE-network under different environment and limitations.
whether an action is positive or negative, while the new state As Figure 7 demonstrates, the environments include the ap-
provides the agent with the necessary information for future plication layer (home appliances, energy resources), technical
decisions. RL algorithms excel in handling complex problems layer (e.g. stability, reliability, power quality, ampacity, etc.),
with uncertainties because they can iteratively correct errors economic layer (e.g. generation and transmission costs, energy
during the training process. This adaptability is crucial in price, tariff adjustment, etc.), and communication layers (e.g.
energy management, where conditions can change rapidly Automated meter reading, wired and wireless networks). An
and unpredictably. By continuously learning and adjusting, optimal scheduling program can transform customers into
RL algorithms can optimize energy scheduling, ensuring that prosumers by shifting the flexible load operation time to off-
energy consumption aligns with periods of low demand and peak hours, which leads to a competitive market. Scheduling
high availability of renewable resources. In the context of of resources, fixed storages, and EVs depend on their intrinsic
smart homes, RL-based scheduling can transform traditional limitations and rely on the overall condition of other environ-
energy consumers into prosumers—entities that both consume ments. Also, technical, and economic constraints may vary,
and produce energy. By shifting flexible load operations to considering dynamic changes in energy supply and demand.
 6

algorithm must route electricity among energy units at the

home level and at HAN and grid levels.

C. Joint LSTM and CNN for attack detection

The proposed attack detection in this research has two
levels, including cyber and physical malicious activities dis-
covery. For the cyber layer, considering the sequential nature
of the data, an LSTM model is proposed to detect FDI attacks
in the network to ensure the correctness of the data used
in the scheduling and routing algorithms. Also, to detect
electricity theft, as one of the major physical attacks, CNN
is proposed. The information used for ETD can be considered
a multidimensional dataset since various interpretation takes
place based on the different points of view. As a deep learning
algorithm, CNN can extract high-level features capturing the
spatial and temporal dependencies. As figure 9 shows, the pro-
posed framework has three main stages: 1) data preprocessing
to classify datasets and extract meaningful features, 2) training
the model based on the datasets, and 3) detecting the malicious
activity as FDI attack to electricity theft.

VI. R ESEARCH C ONTRIBUTIONS

This research’s ultimate goals are to propose a framework,
including optimum scheduling and routing algorithms, besides
an attack detection mechanism to discover FDI attacks and
electricity theft. To achieve the objectives as mentioned earlier,
Fig. 2. The structure of the Proposed Method two DRL based algorithms will be developed for the first two
objectives to mitigate the effect of RES’s uncertainties and re-
duce the complexity of the problem and computational burden.
B. Q-learning based routing algorithm Considering the importance of data accuracy, CNN and LSTM
As an off-policy RL algorithm, Q-learning intends to dis- algorithms are proposed for FDI attack detection, and Deep
cover the best action to take based on the current state while Convolutional Neural Network (DCNN) model is proposed
learning the best value function maximizing the total reward. for electricity theft detection. The research contribution can
In the first step, the algorithm observes the environment and be summarized as follows:
decides how to act based on the current state. Then, a reward or
penalty is assigned, while the algorithm is trained based on the A. Developing a DRL-based scheduling framework for
last experiment. Finally, after several iterations, the algorithm techno-economic improvement in the power network
reaches an optimal strategy. As figure 8 shows, three different
The first major contribution of this research proposal is
routing levels include nano-grid, microgrid, and grid levels.
the introduction of a Q-Learning based framework, an off-
policy Reinforcement Learning (RL) algorithm, designed to
optimize operation schedules within various environments and
constraints. These environments encompass home appliances,
energy resources (including generation and storage units),
and the technical, economic, and communication layers that
interconnect them. The proposed framework aims to create an
optimal scheduling program that transforms traditional energy
consumers into prosumers by shifting the operation of flexible
loads to off-peak hours. This shift reduces energy costs for
households and fosters a competitive market where even those
without their own energy generation units can participate
effectively. By participating in this market, households can
Fig. 3. Schematic of different levels in the routing algorithm trade surplus energy, contributing to the overall efficiency
and sustainability of the energy system. The scheduling of
Considering the number of energy units and different lev- Renewable Energy Sources (RESs), fixed storage systems,
els of considerations, the Q-learning well situs method for and Electric Vehicles (EVs) within this framework depends
electricity routing in the IoE-based power systems since the on their specific limitations and the overall conditions of
 7

Fig. 4. Attack detection Mechanism

the interconnected environments. These conditions include of the problem and minimize the negative impact of power
dynamic variations in energy supply and demand, as well as generation and consumption uncertainty at both user-level
the technical and economic constraints that can fluctuate over and grid levels. Furthermore, the proposed DRL-based OS
time. By leveraging Deep Reinforcement Learning (DRL), the algorithm is capable of handling islanding mode challenges
proposed framework enables consumers to act as prosumers. besides offline working ability.
This is achieved by strategically shifting flexible loads to
off-peak periods, thereby lowering energy bills and allowing
for the trading of surplus energy. In practical terms, the Q- C. Cyber-physical attack detection guarantying information
Learning based framework optimizes the scheduling process correctness
by continuously learning and adapting to changes in the
environment. It considers the real-time status and capabilities To ensure the security of the IoE, an LSTM algorithm is
of home appliances, energy generation and storage units, and proposed to detect FDI attacks in time series data, combined
the overall energy market conditions. By doing so, it identifies with a CNN for feature extraction and correlation identification
the most cost-effective and efficient times to operate various among different types of data to deal with uncertainties.
devices and systems, ensuring that energy consumption is Additionally, an Ensemble Deep Convolutional Neural Net-
aligned with periods of lower demand and higher availability work (EDCNN) algorithm is proposed for Electricity Theft
of renewable energy. Moreover, the framework’s adaptability Detection (ETD) caused by AMIs’ physical manipulation. As
to varying technical and economic constraints ensures that the first layer of the model, a random under bagging technique
it remains effective even as external conditions change. This is applied to deal with the imbalance data, and then deep
adaptability is crucial in a dynamic energy landscape where CNNs are utilized on each subset. Finally, a voting system
supply and demand can be highly variable. The use of DRL is embedded in the last part.
techniques allows the system to handle these complexities
and make informed decisions that maximize both individual
and collective benefits. The Q-Learning based optimization VII. C ONCLUSION
framework represents a significant advancement in energy
management. It not only enhances the efficiency of energy In conclusion, this research addresses the critical challenges
use but also empowers consumers to become active partic- of energy management in IoE-enabled smart grids by devel-
ipants in the energy market. By optimizing the scheduling oping advanced algorithms for operation scheduling, energy
of energy resources and shifting loads to off-peak hours, the routing, and cyber-physical security. The DRL-based schedul-
framework reduces costs, increases the utilization of renewable ing algorithm optimizes the use of RESs, reducing costs and
energy, and supports the development of a more resilient emissions while maximizing economic benefits for prosumers.
and sustainable energy system. This research aims to provide The energy routing algorithm ensures efficient electricity flow,
a robust solution that addresses the current challenges in adapting to technical and economic constraints. Furthermore,
energy management and contributes to the evolution of smart, the proposed security framework enhances the resilience of
integrated energy networks. IoE networks against cyber and physical attacks, ensuring
[27]. reliable operation. The contributions of this research provide
a robust foundation for future developments in smart grid
technology, promoting the integration of renewable energy
B. Developing an energy routing algorithm and enhancing overall energy management efficiency. These
via DRL considering technical and economic constraints advancements are essential for meeting the growing energy
Routing optimization extremely relies on OS since the amount demands in an environmentally sustainable manner. By lever-
and direction of energy that needs to be sent or received aging cutting-edge AI technologies, this research paves the
among devices is planned based on the scheduling algorithm’s way for smarter, more resilient power systems. The findings
results. Also, the transmission constraints that are notified highlight the potential for IoE to revolutionize energy man-
by utilities must be satisfied. As the second contribution, a agement, making it a cornerstone for future innovations in the
DRL based algorithm is proposed to deal with the complexity field.
 8

R EFERENCES [20] A. Yazdinejad, A. Bohlooli, and K. Jamshidi, “P4 to sdnet: Automatic

generation of an efficient protocol-independent packet parser on recon-
figurable hardware,” in 2018 8th International Conference on Computer
[1] D. Namakshenas, A. Yazdinejad, A. Dehghantanha, and G. Srivastava,
and Knowledge Engineering (ICCKE). IEEE, 2018, pp. 159–164.
“Federated quantum-based privacy-preserving threat detection model
[21] ——, “Performance improvement and hardware implementation of open
for consumer internet of things,” IEEE Transactions on Consumer
flow switch using fpga,” in 2019 5th Conference on Knowledge Based
Electronics, 2024.
Engineering and Innovation (KBEI). IEEE, 2019, pp. 515–520.
[2] H. M. Rouzbahani, H. Karimipour, and L. Lei, “A review on virtual [22] Z. Chen, J. Liu, Y. Shen, M. Simsek, B. Kantarci, H. T. Mouftah,
power plant for energy management,” Sustainable energy technologies and P. Djukic, “Machine learning-enabled iot security: Open issues and
and assessments, vol. 47, p. 101370, 2021. challenges under advanced persistent threats,” ACM Computing Surveys,
[3] A. Yazdinejad, R. M. Parizi, A. Dehghantanha, Q. Zhang, and K.-K. R. vol. 55, no. 5, pp. 1–37, 2022.
Choo, “An energy-efficient sdn controller architecture for iot networks [23] H. M. Rouzbahani, H. Karimipour, and L. Lei, “Optimizing scheduling
with blockchain-based security,” IEEE Transactions on Services Com- policy in smart grids using probabilistic delayed double deep q-learning
puting, vol. 13, no. 4, pp. 625–638, 2020. (p3dql) algorithm,” Sustainable Energy Technologies and Assessments,
[4] H. M. Ruzbahani, A. Rahimnejad, and H. Karimipour, “Smart house- vol. 53, p. 102712, 2022.
holds demand response management with micro grid,” in 2019 IEEE [24] Y. Meidan, M. Bohadana, Y. Mathov, Y. Mirsky, A. Shabtai, D. Breiten-
Power & Energy Society Innovative Smart Grid Technologies Conference bacher, and Y. Elovici, “N-baiot—network-based detection of iot botnet
(ISGT). IEEE, 2019, pp. 1–5. attacks using deep autoencoders,” IEEE Pervasive Computing, vol. 17,
[5] A. Yazdinejad, R. M. Parizi, A. Dehghantanha, and K.-K. R. Choo, no. 3, pp. 12–22, 2018.
“Blockchain-enabled authentication handover with efficient privacy pro- [25] H. M. Rouzbahani, H. Karimipour, E. Fraser, A. Dehghantanha, E. Dun-
tection in sdn-based 5g networks,” IEEE Transactions on Network can, A. Green, and C. Russell, “Communication layer security in
Science and Engineering, vol. 8, no. 2, pp. 1120–1132, 2019. smart farming: A survey on wireless technologies,” arXiv preprint
[6] H. Mohammadi Rouzbahani, H. Karimipour, A. Rahimnejad, A. De- arXiv:2203.06013, 2022.
hghantanha, and G. Srivastava, “Anomaly detection in cyber-physical [26] H. M. Rouzbahani, A. Dehghantanha, and K.-K. R. Choo, “Big data
systems using machine learning,” Handbook of big data privacy, pp. analytics and forensics: An overview,” Handbook of Big Data Analytics
219–235, 2020. and Forensics, pp. 1–5, 2022.
[7] H. M. Rouzbahani, H. Karimipour, and L. Lei, “An ensemble deep [27] P. W. Shor, “Algorithms for quantum computation: discrete logarithms
convolutional neural network model for electricity theft detection in and factoring,” in Proceedings 35th annual symposium on foundations
smart grids,” in 2020 IEEE International Conference on Systems, Man, of computer science. Ieee, 1994, pp. 124–134.
and Cybernetics (SMC). IEEE, 2020, pp. 3637–3642. [28] A. Yazdinejad, A. Dehghantanha, R. M. Parizi, and G. Epiphaniou, “An
[8] A. Yazdinejad, R. M. Parizi, A. Dehghantanha, H. Karimipour, G. Sri- optimized fuzzy deep learning model for data classification based on
vastava, and M. Aledhari, “Enabling drones in the internet of things nsga-ii,” Neurocomputing, vol. 522, pp. 116–128, 2023.
with decentralized blockchain-based security,” IEEE Internet of Things [29] A. H. Bahrami and H. M. Rouzbahani, “Cyber security of smart
Journal, vol. 8, no. 8, pp. 6406–6415, 2020. manufacturing execution systems: a bibliometric analysis,” AI-Enabled
[9] H. M. Ruzbahani and H. Karimipour, “Optimal incentive-based demand Threat Detection and Security Analysis for Industrial IoT, pp. 105–119,
response management of smart households,” in 2018 IEEE/IAS 54th In- 2021.
dustrial and Commercial Power Systems Technical Conference (I&CPS). [30] A. Yazdinejad, A. Dehghantanha, and G. Srivastava, “Ap2fl: auditable
IEEE, 2018, pp. 1–7. privacy-preserving federated learning framework for electronics in
[10] A. Yazdinejad, A. Dehghantanha, R. M. Parizi, G. Srivastava, and healthcare,” IEEE Transactions on Consumer Electronics, 2023.
H. Karimipour, “Secure intelligent fuzzy blockchain framework: Effec- [31] M. A. Ferrag, O. Friha, D. Hamouda, L. Maglaras, and H. Janicke,
tive threat detection in iot networks,” Computers in Industry, vol. 144, “Edge-iiotset: A new comprehensive realistic cyber security dataset of
p. 103801, 2023. iot and iiot applications for centralized and federated learning,” IEEE
[11] H. M. Rouzbahani, H. Karimipour, and L. Lei, “Multi-layer defense Access, vol. 10, pp. 40 281–40 306, 2022.
algorithm against deep reinforcement learning-based intruders in smart [32] A. Yazdinejad, E. Rabieinejad, T. Hasani, and G. Srivastava, “A bert-
grids,” International Journal of Electrical Power & Energy Systems, vol. based recommender system for secure blockchain-based cyber physical
146, p. 108798, 2023. drug supply chain management,” Cluster Computing, vol. 26, no. 6, pp.
[12] G. K. Jagarlamudi, A. Yazdinejad, R. M. Parizi, and S. Pouriyeh, 3389–3403, 2023.
“Exploring privacy measurement in federated learning,” The Journal of [33] H. M. Rouzbahani, H. Karimipour, and G. Srivastava, “Big data appli-
Supercomputing, pp. 1–41, 2023. cation for renewable energy resource security,” Handbook of Big Data
and Privacy, 2019.
[13] A. Yazdinejad, A. Dehghantanha, R. M. Parizi, M. Hammoudeh,
[34] A. Yazdinejad, M. Kazemi, R. M. Parizi, A. Dehghantanha, and
H. Karimipour, and G. Srivastava, “Block hunter: Federated learning
H. Karimipour, “An ensemble deep learning model for cyber threat
for cyber threat hunting in blockchain-based iiot networks,” IEEE
hunting in industrial internet of things,” Digital Communications and
Transactions on Industrial Informatics, vol. 18, no. 11, pp. 8356–8366,
Networks, vol. 9, no. 1, pp. 101–110, 2023.
2022.
[35] Y. E. Oktian, B. Stanley, and S.-G. Lee, “Building trusted federated
[14] H. M. Rouzbahani, H. Karimipour, A. Dehghantanha, and R. M. Parizi, learning on blockchain,” Symmetry, vol. 14, no. 7, p. 1407, 2022.
“Blockchain applications in power systems: A bibliometric analysis,” [36] A. Yazdinejadna, R. M. Parizi, A. Dehghantanha, and M. S. Khan,
Blockchain Cybersecurity, Trust and Privacy, pp. 129–145, 2020. “A kangaroo-based intrusion detection system on software-defined net-
[15] T. Crawford, S. Duong, R. Fueston, A. Lawani, S. Owoade, A. Uzoka, works,” Computer Networks, vol. 184, p. 107688, 2021.
R. M. Parizi, and A. Yazdinejad, “Ai in software engineering: A survey [37] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas,
on project management applications,” arXiv preprint arXiv:2307.15224, “Communication-efficient learning of deep networks from decentralized
2023. data,” in Artificial intelligence and statistics. PMLR, 2017, pp. 1273–
[16] A. Yazdinejad, A. Bohlooli, and K. Jamshidi, “Efficient design and 1282.
hardware implementation of the openflow v1. 3 switch on the virtex-6 [38] H. Mohammadi Rouzbahani, “Application of internet of energy in smart
fpga ml605,” The Journal of Supercomputing, vol. 74, pp. 1299–1320, grids using deep reinforcement learning,” 2022.
2018. [39] B. Zolfaghari, A. Yazdinejad, A. Dehghantanha, J. Krzciok, and
[17] H. M. Rouzbahani, A. H. Bahrami, and H. Karimipour, “A snapshot K. Bibak, “The dichotomy of cloud and iot: Cloud-assisted iot from
ensemble deep neural network model for attack detection in industrial a security perspective,” arXiv preprint arXiv:2207.01590, 2022.
internet of things,” AI-Enabled Threat Detection and Security Analysis
for Industrial IoT, pp. 181–194, 2021.
[18] A. Yazdinejad, B. Zolfaghari, A. Dehghantanha, H. Karimipour, G. Sri-
vastava, and R. M. Parizi, “Accurate threat hunting in industrial internet
of things edge devices,” Digital Communications and Networks, vol. 9,
no. 5, pp. 1123–1130, 2023.
[19] D. Sheridan, J. Harris, F. Wear, J. Cowell Jr, E. Wong, and A. Yazdinejad,
“Web3 challenges and opportunities for the market,” arXiv preprint
arXiv:2209.02446, 2022.