IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 27, NO.

8, AUGUST 2016 1643

Dynamic Energy Management System

for a Smart Microgrid
Ganesh Kumar Venayagamoorthy, Senior Member, IEEE, Ratnesh K. Sharma, Member, IEEE,
Prajwal K. Gautam, Associate Member, IEEE, and Afshin Ahmadi, Student Member, IEEE

Abstract— This paper presents the development of an intelli- Battery_cap(t) Energy stored in the battery at time t.
gent dynamic energy management system (I-DEMS) for a smart Battery_capsp Energy stored in the battery at SOC
microgrid. An evolutionary adaptive dynamic programming and
reinforcement learning framework is introduced for evolving the set point.
I-DEMS online. The I-DEMS is an optimal or near-optimal CL Critical load.
DEMS capable of performing grid-connected and islanded micro- DG Diesel generation.
grid operations. The primary sources of energy are sustainable, DT Decision tree.
green, and environmentally friendly renewable energy systems
(RESs), e.g., wind and solar; however, these forms of energy are Nchr (t) Continuous number of states the
uncertain and nondispatchable. Backup battery energy storage battery is charging.
and thermal generation were used to overcome these challenges. NCL Noncritical load (controllable load).
Using the I-DEMS to schedule dispatches allowed the RESs Ndschr (t) Continuous number of states the
and energy storage devices to be utilized to their maximum
in order to supply the critical load at all times. Based on the battery is discharging.
microgrid’s system states, the I-DEMS generates energy dispatch P(t) Power dispatch set point.
control signals, while a forward-looking network evaluates the PB (t)P(t) For charging and discharging battery.
dispatched control signals over time. Typical results are presented PCL_D (t) Power demands of the CL at time t.
for varying generation and load profiles, and the performance
of I-DEMS is compared with that of a decision tree approach- PCL_S (t) Total P(t) supplied to CL.
based DEMS (D-DEMS). The robust performance of the PCL(PV/ W ) (t)P(t) Supplied to CL by PV and wind.
I-DEMS was illustrated by examining microgrid operations PCL(B)(t)P(t) Supplied to CL by battery.
under different battery energy storage conditions. PCL(DG) (t)P(t) Supplied to CL by DG.
Index Terms— Adaptive dynamic programming, dynamic PCL(G) (t)P(t) Supplied to CL from grid.
energy management system (DEMS), evolutionary computing,
microgrid, neural networks, reinforcement learning, renewable
PDG (t)P(t) By diesel generator.
energy. PDGmax Maximum available DG.
N OMENCLATURE PG (t)P(t) By grid.
B Battery. PGmax_import (t) Maximum power that the grid can
Battery_capinitial Initial value of energy stored in the import at time t.
battery. PGmax_export (t) Maximum power that the grid can
Battery_capmax Maximum value of energy stored in the export at time t.
battery. PNCL_D (t) Power demands of the NCL at time t.
Manuscript received February 20, 2014; revised May 13, 2015 and PNCL_S (t) Total P(t) supplied to NCL.
December 20, 2015; accepted December 20, 2015. Date of publication PNCL(PV/ W ) (t)P(t) Supplied to NCL by PV and wind.
January 20, 2016; date of current version July 15, 2016. This work was sup- PNCL(B) (t)P(t) Supplied to NCL by battery.
ported in part by NEC Laboratories America Inc., and in part by the National
Science Foundation under Grant ECCS 1232070, Grant ECCS 1308192, and PPV (t) Power available from PV system at
Grant IIP 1312260. time t.
G. K. Venayagamoorthy is with the Real-Time Power and Intelligent
Systems Laboratory, Clemson University, Clemson, SC 29634 USA, and also PW (t) Power available from wind system at
with the Eskom Centre of Excellence in HVDC Engineering, University of time t.
KwaZulu–Natal, Durban 4041, South Africa (e-mail: [email protected]). PV Photovoltaic.
R. K. Sharma is with the Energy Management Department,
NEC Laboratories America Inc., Cupertino, CA 95014 USA (e-mail: SOC(t) State of charge of battery at time t.
[email protected]). SOCinitial SOC initial value.
P. K. Gautam was with the Real-Time Power and Intelligent Systems
Laboratory, Clemson University, Clemson, SC 29634 USA. He is now with
SOCmin SOC minimum value.
Spirae, Inc., Fort Collins, CO 80524 USA (e-mail: [email protected]). SOCmin_reserve SOC minimum value with reserve.
A. Ahmadi was with the Real-Time Power and Intelligent Systems SOCmax SOC maximum value.
Laboratory, Clemson University, Clemson, SC 29634 USA. He is now with
the Department of Electrical and Computer Engineering, Clemson University, SOCsp SOC set point.
Clemson, SC 29634 USA (e-mail: [email protected]). U (t) Total utility for evaluating microgrid
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
operation at time t.
Digital Object Identifier 10.1109/TNNLS.2016.2514358 U B (t) Utility for evaluating SOC(t).

2162-237X © 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted,
but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1644 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 27, NO. 8, AUGUST 2016

UBCY (t) Utility for evaluating Nchr (t) D-DEMS supplies the entire CL and NCL requirements with-
and Ndschr (t). out assigning any priority, and will charge the battery only
UCL (t) Utility for evaluating PCL_S (t) to meet if more energy is available than load [2]. If energy sources
PCL_D (t). are not available, the battery supplies full-load requirements
UDG (t) Utility for evaluating PDG (t) to meet until it discharges fully. This minimizes the lifespan of energy
PCL_D (t). storage through random and abrupt charging and discharging
UG (t) Utility for evaluating PG (t). decisions, and also reduces the security and reliability of
UNCL (t) Utility for evaluating PNCL_S (t) to electric power supply by not considering load priorities (no
meet PNCL_D (t). temporary load shedding), which may result in a total power
outage. Expert systems-based energy management schemes
for battery storage have been used [3], [4] to incorporate
I. I NTRODUCTION operational constraints, such as state of charge (SOC) limit and

M ICROGRIDS integrate modular distributed energy

sources, such as wind, solar, and fuel cells, with
storage devices and controllable loads to form a low-voltage
charge/discharge current limits, for dispatching renewables.
Computational intelligence methods are widely employed
in EMSs [5], [6]. Research in this area has been active.
distribution system. A microgrid can be defined as a small- In [7], real-time particle swarm optimization (PSO)-based
scale, self-supporting network driven by on-site generation energy management of a stand-alone hybrid wind and micro-
sources with the ability to separate from an external grid turbine energy system was presented. Dynamic programming-
for sustainability or energy security purposes. They improve based power management optimization was compared with
grid reliability and supply sustainable and quality electric simple rule-based management for a grid-connected PV with
power. Microgrids can be connected to a main power network batteries in [8]. An EMS for a microgrid using a fuzzy
or operated autonomously, similar to the power systems of logic expert system to minimize the operation’s cost and
physical islands [1]. emission levels was developed in [9]. A smart EMS to achieve
Smart microgrids promise a new approach for electric power optimal microgrid operation costs was presented using genetic
generation through the clusters of small distributed on-site algorithms in [10]. The nonlinear regression technique was
generators. There may be numerous advantages in developing used for real-time energy management in microgrids in [11].
microgrids, including the following: A fuzzy logic control EMS was presented in [12] to satisfy the
1) to manage growing demand without overloading existing energy load demand and maintain the SOC of the battery and
electricity infrastructure or expanding capacity; the hydrogen tank level between certain target margins, while
2) to reduce frequency and duration of grid disruption trying to optimize the utilization cost and lifetime of the energy
through distributed energy resource management system storage system. An expert energy system was proposed in [13]
and self-healing functionality; to simultaneously minimize the total operation cost and the net
3) to ensure energy security through self-sustainability; emission by finding the optimal set points of distributed energy
4) to address climate change by utilizing clean energy resources and storage devices. An adaptive training algorithm
resources; based on genetic algorithms, fuzzy clustering, and neuron-by-
5) to supply electric power to areas where local utility neuron algorithms was used for real-time microgrid operations
is unable to provide reliable service or have access to in [14]. Generation and load forecast models were combined
customers. to create a new a microgrid EMS in [15]. To optimize energy
Because the output of renewable resources fluctuates dispatch, an optimized fuzzy logic controller (FLC)-based
depending on the weather condition and time of day, the PV energy dispatch controller was implemented using
majority of renewable energies cannot guarantee a continuous a PSO algorithm [16], in which an FLC was developed to
and steady amount of power generation. Besides, electricity assign energy dispatch priority to CLs, then to the battery, and
demand may be partly unpredictable in microgrids, which finally to NCLs. An FLC membership function and rule set
adds another dimension of complexity to the control of was optimized by PSO, such that the optimized FLC could
power system. Grids that want to allow for the integration maximize energy to the system loads while maintaining a
of renewable energy sources (RESs) have to consider these higher than average battery SOC.
destabilizing effects due to such variable energy inflow and Action-dependent heuristic dynamic programm-
outflow. As variable power sources (such as wind and solar ing (ADHDP), a type of adaptive critic design (ACD)-
power) reach high levels of grid penetration, energy storage based controller, was implemented in a PV system in [17]
devices and intelligent energy management that can handle and in a battery system [18], in which two neural networks
variability and uncertainty become essential. For a microgrid were used to derive an optimal control strategy. The intelligent
to provide reliable power supply, advanced control algorithms methods in [16] and [17], however, were limited to energy
to manage energy dispatch and maximize its performance management between PV and the energy storage system and
become crucial. did not demonstrate dynamic energy dispatch considering
Conventional dynamic energy management systems imple- grid-connected mode and additional generation, such as wind
mented using decision trees DTs (D-DEMS) have been devel- turbines and diesel generators.
oped, but are highly inefficient. They dispatch energy based on The microgrid considered in this paper consists of wind
available power and the state of energy storage. In addition, the and solar power, a diesel generator, and a battery energy
VENAYAGAMOORTHY et al.: DYNAMIC ENERGY MANAGEMENT SYSTEM FOR A SMART MICROGRID 1645

storage system. For maximum utilization of renewable energy

sources, an intelligent DEMS (I-DEMS) was developed, which
optimizes the microgrid system operations on a minute-by-
minute time scale using an optimal energy dispatch strategy.
In this paper, backup battery energy storage and thermal
generation were used to overcome the uncertainty and non-
dispatchability challenges in a microgrid with RESs.
The primary contributions of this paper include the
following.
1) Development of an adaptive I-DEMS using an ADHDP
approach. ADHDP is based on combined concepts
of adaptive dynamic programming and reinforcement
learning concepts. The ADHDP framework employs
two neural networks.
2) Dynamic optimization of the I-DEMS using an evolu-
tionary strategy to improve its dispatch solutions over
time. To the best of our knowledge, this is the first
paper to present the introduction of evolutionary learning
approach to empower adaptive dynamic programming
to solve dynamic optimization problems such EMS
optimally in a faster manner.
3) Development of an I-DEMS from data (from a loca-
tion’s PV solar and wind power profiles, and customers’ Fig. 1. Microgrid system showing the interface to the I-DEMS and
CL and NCL profiles). microgrid operator. The maximum size in kilowatt is shown for each microgrid
4) An I-DEMS framework to accomplish multiple objec- component.
tives. In this paper, these objectives include max-
imizing reliability, self-sustainability, environmental
friendliness, extended battery life, and maximize
customer satisfaction.
5) Development of a performance index (PI) to compare
I-DEMS with D-DEMS. This is an applicable and useful
metric to evaluate other DEMS approaches.
Using the I-DEMS based on an evolutionary ADHDP
framework allowed the RESs and energy storage devices to
Fig. 2. Renewable energy [solar, PPV (t) and wind, PW (t)] profiles for
be utilized to their maximum in order to supply the CL at all a period of two days (a total of 2880 min).
times. The I-DEMS used the microgrid’s system states to gen-
erate energy dispatch control signals, while a forward-looking
network (FLN) evaluated the dispatched control signals over
time. A penalty and reward concept was introduced in util-
ity function formulations to be utilized in the evolutionary
ADHDP approach. The integration of evolutionary learning
into the I-DEMS framework allowed for fast online dynamic
optimization of the I-DEMS performance.
The remainder of this paper is organized as follows.
Section II describes the microgrid model and the wind and
solar energy and load profiles used in this paper. The develop- Fig. 3. Critical (PCL_D (t)) and controllable (PNCL_D (t)) load profiles for
a period of two days.
ment of DEMS based on the DT-based approach is presented
in Section III. Section IV describes the I-DEMS frame-
work, the dynamic utility formulation, and their development. II. M ICROGRID S TRUCTURE
Section V presents the dynamic optimization of the I-DEMS The microgrid system in Fig. 1 consisted of hybrid energy
with evolutionary learning to yield fast and enhanced energy sources, namely, 40-kW solar PV generation, 30-kW wind
dispatch solution. Section VI presents typical results with generation, a 10-kW diesel generator, a battery energy storage
the I-DEMS and discusses its performance in comparison system, and CL and controllable (noncritical) load. The solar
with that of the D-DEMS for integrated grid-connected and and wind power profiles chosen for the study are shown
islanded operations. The robust performance of the I-DEMS by in Fig. 2. Fig. 3 shows the CL and controllable load profiles.
examining microgrid operations under different battery energy The battery size in Fig. 1 was designed to power CLs for
storage conditions is also presented in this section. Finally, the at least 3 h. The assets considered in the microgrid system
conclusion is drawn in Section VII. (CL, NCL, PV, wind generation, DG, and battery) were
1646 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 27, NO. 8, AUGUST 2016

Fig. 4. DT approach-based DEMS (D-DEMS) for the microgrid in Fig. 1.

aggregated or lumped representations of assets; thus, all these 1) Case I: If sum of energy from RESs, (PPV (t) + PW (t))
assets were scalable not only in terms of kilowatt/megawatt is greater than the CL demand PCL_D (t).
size, but in number of different types of components as well. a) The CL is met first.
This will allow the development of different sized microgrid b) The surplus energy will be supplied to the bat-
clusters, which can operate independently or in parallel with tery, PB (t), to increase its SOC(t) to the set
the broader utility grid and perform under a centralized EMS. point, SOCsp , if determined necessary.
The maximum CL and controllable load were c) At this point, the remaining energy, (PPV (t) +
20.6 and 26.1 kW, respectively. Day 1 data (1440 min) PW (t) − PCL_D (t) − PB (t)), will be used to meet
was used to develop I-DEMS. Day 2 data (a second set NCL PNCL_D (t) in part or full as determined.
of 1440 min) was used to evaluate the I-DEMS’s performance d) After satisfying the above three steps, any excess
on unseen data. The microgrid operation DEMSs were energy, (PPV (t) + PW (t) − PCL_D (t) − PB (t) −
implemented in MATLAB in this paper. PNCL_D (t)), will be exported to the grid, upon
receiving a request from the utility network
operator. Prior to supplying any PG (t) to the grid,
III. D ECISION T REE A PPROACH -BASED DEMS
PNCL_D (t) must be supplied in full.
The DT approach-based DEMS is a deterministic energy 2) Case II: If energy from RESs, (PPV (t) + PW (t)) is
dispatch manager that evaluates the system states of a micro- insufficient to meet CL PCL_D (t).
grid based on set rules and, then, computes the respec- a) Dispatches from battery, PB (t), diesel generator,
tive energy dispatches. The operation of the D-DEMS is PDG (t), and grid, PG (t), will strictly be utilized in
shown in Fig. 4 (flowchart). In developing the rules for the this sequence to meet CL PCL_D (t).
D-DEMS, the time varying CL was given the highest priority, b) The NCL PNCL_D (t) will never be met
the load that must be met at all times. In a nut shell, the using DG, PDG (t), or grid import, PG (t).
D-DEMS operation strictly implements the following dispatch c) Under this condition, the battery will not be
steps sequentially in two possible cases. charged and no grid export is possible.
VENAYAGAMOORTHY et al.: DYNAMIC ENERGY MANAGEMENT SYSTEM FOR A SMART MICROGRID 1647

In the case of an islanded microgrid operation, when the

battery energy storage system and the diesel generator were
utilized to their maximum (SOCmin and PDGmax ) but fell short
of meeting the full CL demand, a concept of battery storage
reserve capacity was introduced to supply the unmet CL.
The energy storage unit was allowed to discharge to a new Fig. 5. Forward path of HDP architecture.
minimum SOC level (SOCmin_reserve ), thus meeting 100% of
CL at all times.
The following sign convention is utilized in this paper:
1) a positive PB (t) represents discharging; 2) a negative PB (t)
represents charging; 3) a positive PG (t) represents import; and
4) a negative PG (t) represents export.

IV. I NTELLIGENT DYNAMIC E NERGY

M ANAGEMENT S YSTEM
The multiple objectives of the I-DEMS in this paper (Fig. 1) Fig. 6. ADHDP network receiving feedback for learning from the external
environment/plant through a primary reinforcement signal, U (t).
are as follows.
1) Supply the power requirements of the CLs, PCL_D (t),
at all times. This provides 100% reliability with regard was developed to approximate the cost-to-go function J of
to power supply to CLs. dynamic programming [given in (1)] by using a J -function
2) Maintain the battery SOC at an optimal level (defined as approximator, such as a neural network [19]. Inputs to this
the operator through a set point for SOC). This ensures approximator are measurements of known/predicted system
and supports meeting the reliability goal in 1). outputs and control inputs. This approach was first introduced
3) Maximize controllable load dispatch PNCL_D (t). This in [21] and was later called ACD. ACD seeks to minimize
means more customer satisfaction and it creates oppor- the expected value of the cost function with respect to the
tunities for demand-response capability. control, conditioned on knowledge of the system dynamics,
4) Maximize the utilization of renewable energy resources, its state, and the probability distributions of uncertainties [22].
and minimize the use of DG and import/export from Several synonyms for ACD can be found in the litera-
the grid. This means more environmental friendly and ture, including approximate dynamic programming [19], [23],
sustainable operation. adaptive dynamic programming [24], [25], HDP [26], [27],
5) Increase battery life by maximizing battery charging neural dynamic programming [28], and reinforcement
or discharging for a continuous number of states (each learning [29].
state is the dispatch instant, every minute in this paper) As shown in Fig. 5, this ACD architecture (HDP) consists
and thus enhanced sustainability by reducing the rate of of the critic, model, and action networks. These three networks
replenishing batteries. perform the function of evaluation, prediction/estimation, and
These objectives ensure the microgrid operations to be self- control, respectively. The model-independent HDP architec-
sustainable, reliable, environmental friendly, and technology tures are referred to as ADHDP [19].
ready for smart grid functionality. The cost of operation of ACD-based designs yield nonlinear EMS, whereas classical
microgrids, hour-to-hour, will not be a dominant factor in optimal designs typically yield linear EMS. Moreover,
the future, as communities invest in microgrid assets and as classical methods rely on the linear model of the sys-
I-DEMS technology advances. Therefore, the emphasis here tem, whereas ACD approaches can be a measurement-based
did not include economics in the cost function to be optimized design [30]. The ADHDP can use the approximation capa-
dynamically. The I-DEMS framework is developed based on bilities of neural networks to develop optimal DEMS from
an ADHDP approach. The ADHDP approach is based on measurements of available system inputs and outputs, feed-
the combined concepts of adaptive dynamic programming and back control, and reinforcement signals received, as shown
reinforcement learning [19]. in Fig. 6.
In this paper, ADHDP uses two neural networks, an action
A. Adaptive Dynamic Programming network (I-DEMS), which provides the control signals, and
and Reinforcement Learning a critic network, also referred to as FLN, which criticizes
Dynamic programming is a very useful tool in solving the I-DEMS performances. These two networks solve the
optimization problems. In particular, it can easily be applied Hamilton–Jacobi–Bellman equation of optimal control [19].
to optimal control of nonlinear systems with or without con- The FLN network approximates the cost-to-go function J of
straints on the control and state variables. However, it is often Bellman’s equation of dynamic programming, which is
computationally difficult to run exact dynamic programming given by
due to the backward numerical process required to find the ∞

optimal solution, as a result of the curse of dimensionality [20]. J (t) = γ i × U (t + i ) (1)
To circumvent this problem, adaptive dynamic programming i=0
1648 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 27, NO. 8, AUGUST 2016

Fig. 7. I-DEMS controller development framework based on the ADHDP

approach.

Fig. 8. Forward-looking neural network implementation (critic network) for

where γ is the discount factor in the range [0, 1]. The providing performance feedback to the I-DEMS (action network) dispatch
controller.
utility, U (t), guides the FLN in evaluating the I-DEMS per-
formance. The I-DEMS network provides optimal control to
minimize or maximize the cost-to-go function J . given by
The utility function formulation, variable scaling, dis-
count factor selection, learning rates, and convergence U (t) = UCL + UNCL (t)+ U B (t)+ UDG (t)+ UG (t)+ UBCY (t)
of the ADHDP controller are explained in [19]. The = w1 (t) × f (PCL_S (t)) + w2 (t) × f (PNCL_S (t))
ADHDP-based I-DEMS optimizes power dispatch to and from + w3 (t) × f (SOC(t)) + w4 (t) × f (PDG (t))
CL and controllable load, battery, grid, and diesel generator + w5 (t) × f (PG (t)) + w6 × f (Nchr (t) + Ndschr (t))
to achieve the multiobjectives mentioned above. The system
states are the actual wind power, solar power, battery SOC (2)
and load demand values, and the grid status, as shown where weights w1 (t), w2 (t), w3 (t), w4 (t), w5 (t), and w6 are
in Fig. 1. based on prioritizing the power dispatch to the CL, controllable
Fig. 7 shows the ADHDP architecture-based I-DEMS for load, battery, diesel generator, grid, and the sum of the
a smart microgrid. The I-DEMS drives the microgrid system continuous number of states that the battery is charging and
to the desired states, and the FLN provides dynamic perfor- discharging. The weights were carefully selected to handle the
mance feedback to the I-DEMS with respect to achieving the objectives of I-DEMS: 1) self-sustainability; 2) reliability; and
desired states. I-DEMS learns a policy function, whereas the 3) carbon emission reduction. This allows the development of
FLN learns a value function given by Bellman’s equation of smart microgrid clusters with improved reliability, efficiency,
dynamic programming. economics, and sustainability of the electricity generation
and distribution. If necessary, the weights can be selected to
B. Forward-Looking Network (Critic Neural Network) prioritize power dispatch by considering the generation cost of
each unit. However, cost consideration is not within the scope
The FLN was implemented using a multilayer percep-
of this paper.
tron (MLP) feedforward neural network, as shown in Fig. 8.
The utility function is very important, because it guides
The inputs to the FLN consisted of the inputs (states) and
the critic network to improve the performance of the action
outputs (power dispatches) of the I-DEMS at time instances
network. The feedback loop allows the action network to
t, t − 1, and t − 2, as well as a bias term. The five
improve behavior over time. Algorithms 1–5 illustrate how
system states were the CL power requirements, NCL power
to compute the subutilities in (2). Algorithm 6 illustrates
requirements, SOC of the battery, PV, and wind power (a total
the computation of the U (t). U (t) was used to evaluate the
of 15, including the states at the time delayed values). The
microgrid operation at time t with the objective of reducing
power dispatches were the CL and NCL and energy to the
the amount of unmet loads, if any, reducing the use of DG
battery, and from the diesel generator and the grid (a total
and grid generation as much as possible, and enhancing the
of 15, including the dispatches at the time delayed values).
lifecycle of the battery [31]. The operational and maintenance
The input, hidden, and output layers consisted of 31 linear
costs of the battery were assumed to be proportional to the
neurons, 30 sigmoidal neurons, and one linear neuron, respec-
number of charging and discharging cycles [32]. The utility
tively. The output of the FLN in the ADHDP framework
formulation was based on the concept of penalty and reward
was the approximated J , given by (1). As a rule of thumb,
allocation. The I-DEMS was rewarded for optimal dispatches
the number of neurons in the hidden layer should at least
and penalized for nonoptimal dispatches.
be the size of the inputs. The optimization of the neural
The utility for evaluating supplied energy to CL to meet
network structure is not within the scope of this paper.
CL demand is given in
The utility function, U (t), provides the FLN with immediate
performance measure of the I-DEMS dispatch signals and is UCL (t) = w1 (t) × abs(1 − PCL_S (t)/PCL_D (t)) (3)
VENAYAGAMOORTHY et al.: DYNAMIC ENERGY MANAGEMENT SYSTEM FOR A SMART MICROGRID 1649

Algorithm 1 Utilities (U B (t) and UNCL (t)) Evaluating Algorithm 3 Utility UBCY (t) Evaluating Battery Lifecycle
SOC(t) and E NCL (t) to Meet PNCL (t) Impacts [Nchr (t) and Ndschr (t)]
1: Initialize w2ini and w3ini 1: Initialize Ndschr (t) = Nchr (t) = 0 and w5
2: Initialize S OCmin = S OCsp − S OCmin 2: if E B (t − 2 : t) > 0
3: Initialize S OCmax = S OCmax − S OCsp 3: Ndschr (t) = 1
4: S OC(t) = abs(S OC(t) − S OCsp ) 4: end
5: if S OC(t) >= SOCsp 5: if E B (t − 2 : t) < 0
6: w2 (t) = (SOC(t)/SOCmax ) × w2ini + w2ini 6: Nchr (t) = 1
7: w3 (t) = abs(w3ini − (SOC(t)/SOCmax ) × w3ini ) 7: end
8: U NC L (t) = w2 (t) × (1 − E NC L (t)/PNC L (t)) 8: U BCY (t) = w5 × (1 − (Nchr (t) + Ndschr (t))
9: U B (t) = w3 (t) × (Battery_capmax − Battery_cap(t))/
Batter y_capmax
10: end Algorithm 4 Utility UG (t) Evaluating Grid Export PG (t)
11: if SOC_NN_caps < SOC_sp 1: if PP V (t) + PW (t) − PC L_S (t) > 0
12: w3 (t) = (S OC(t)/S OCmin ) × w3ini + w3ini 2: if PC L_S (t) = PC L D (t)
13: w2 (t) = abs(w2ini − (S OC(t)/S OCmin ) × w2ini ) 3: if PNC L_S (t) = PNC L_D (t)
14: U NC L (t) = w2 (t) × (1 − E NC L (t)/PNC L (t)) 4: if S OC(t) <= S OCmin
15: U B (t) = w3 (t) × abs((Batter y_capsp − 5: if PDG (t) = 0
Batter y_cap(t))/Batter y_capsp ) 6: Surplus Pren (t) = PC L_S (t) + PNC L (t) − Ppv (t)
16: end −Pw (t)
7: if Surplus Pren (t) = PG (t)
8: UG (t) = 0
Algorithm 2 Utility UDG (t) Evaluating SOC(t) and E CL (t) 9: else
to Meet PCL (t) 10: W6 (t) = W6ini
1: Initialize w4ini 11: if Surplus Pren (t) < PG (t)
2: if E C L (t) − PP V (t) − PW (t) > 0 12: UG (t) = w6 (t) × abs(1 − PG (t)/
3: if SOC(t) <= SOCmin Surplus Pren (t))
4: if E C L (t) − PP V (t) − PW (t) <= PDGmax 13: if Surplus Pren (t) < PG (t)
5: unmet PC L (t) = EC L (t) − PP V (t) − PW (t) 14: UG (t) = w6 (t) × abs(PG (t)/Surplus Pren (t))
6: if unmet PC L (t) >= E DG (t) 15: end
7: w4 (t) = (unmet PC L (t) − E DG (t))/ 16: end
(unmet PC L (t))) × w4ini 17: else
8: U DG (t) = w4 (t) × (1 − E DG (t)/PDGmax ) 18: W6 (t) = W6ini
9: else 19: UG (t) = w6 (t) × abs(PG (t)/Surplus Pren (t))
10: w4 (t) = (EDG (t) − unmetPC L (t))/(EDG (t)) 20: end
×w4ini
11: U DG (t) = w4 (t) × (E DG (t)/PDGmax )
12: end based on system states and dispatches, as outlined
13: else in Algorithms 1–5.
The development of FLN in the ADHDP framework began
14: w4 (t) = (PDGmax − E DG (t))/(PDGmax ) × w4ini
with a discount factor of zero, where J (t) was trained to
15: U DG (t) = w4 (t) × (1 − E x DG (t)/PDGmax )
meet U (t) and then gradually increased to a value lower
16: end
than one. The horizon was determined by the discount factor.
17: else
During training, the objective of FLN development was to
18: w4 (t) = w4ini minimize (5) and update the FLN weights. The weight update
19: U DG (t) = w4 (t) × (E DG (t)/PDGmax ) algorithm used was the standard backpropagation
20: end ∞
21: else 
2
E FLN (t) (5)
22: w4 (t) = w4ini t =0
23: U DG (t) = w4 (t) × (E DG (t)/PDGmax )
where
24: End
E FLN (t) = U (t) + γ · J (t) − J (t − 1). (6)

where C. I-DEMS (Action Neural Network)

The I-DEMS was implemented using an MLP feedforward
w1 (t) = 1 − (w2 (t) + w3 (t) + w4 (t) + w5 (t) + w6 (t)). (4)
neural network, as shown in Fig. 9. I-DEMS learned the policy
The weights w1 (t), w2 (t), w3 (t), w4 (t), w5 (t), and w6 were iteration using the feedback from FLN, which was used to
initialized to some values and then updated dynamically update the weights of the MLP. The number of neurons in the
1650 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 27, NO. 8, AUGUST 2016

Algorithm 5 Utility UG (t) Evaluating Grid Import PG (t)

1: if PCl (t) − PW (t) − PP V (t) > 0
2: if PDG (t) = PDGmax (t)
3: if S OC(t) <= S OCmin (t)
4: if PNC L_S (t) = 0
5: Unmet PC L D (t) = PC L_S (t) − PP V (t) − PW (t)
−PDGmax (t)
6: if Unmet PC L D (t) = PG (t)
7: UG (t) = 0
8: else
9: W6 (t) = W6ini
10: if Unmet PC L D (t) > PG (t)
11: UG (t) = w6 (t) × abs(1 − PG (t)/PGmax_import (t))
Fig. 9. Action network implemented using a neural network.
12: if Surplus PC L D (t) < PG (t)
13: UG (t) = w6 (t) × abs(PG (t)/PGmax_import (t))
14: end
15: end
16: else
17: W6 (t) = W6ini
18: UG (t) = w6 (t) × abs(PG (t)/PGmax_import (t))
19: end

Algorithm 6 Total Utility Function Evaluating Microgrid

Operation at Time t
1: Start
2: Initialize w1ini, w2ini, w3ini, w4ini, w5, w6ini
3: Collect: Fig. 10. The initial development of the I-DEMS using supervised learning
PC L_S (t), PNC L_S (t), PDG (t), PC L D (t), PNC L_ D (t), approach based on the D-DEMS’s dispatch signals.
P DG max , PP V (t), PW (t), S OC(t), S OCsp , S OCmin ,
S OCmax , Batter y_Capmax , Batter y_Cap(t),
Batter y_Capsp , PG max_import (t), PG max_ex port (t) the sources dispatched. The power dispatched to the loads was
4: for i = 1Z : t the sum of CL and NCL. For an islanded microgrid, the power
5: Compute: UC L (t), U NCl (t), U DG (t), U B (t), U BCY (t), dispatched from the sources was the sum of RESs, battery, and
UG (t) DG, while for the grid-connected microgrid, that value was
6: U (t) = UC L (t)+U NCl (t)+U DG (t)+U B (t)+U BCY (t) the sum of RESs, battery, diesel, and import from the grid.
+UG (t) Any imbalance was addressed by implementing the following
7: end checks at the output of the I-DEMS.
8: End 1) The power dispatched to the CL and NCL does not
exceed the CL and NCL demands, respectively.
2) The power dispatched to charge the battery does not
input layer is equal to the number of input features. The inputs exceed the sum of energy from RESs, and the power
to the I-DEMS were composed of five system states at time dispatched to discharge the battery does not exceed the
instances t, t − 1, and t − 2 (a total of 15 including the states total load demand.
at the time delayed values); five power dispatch set points at 3) The SOC of the battery has a minimum and maximum
time instance t − 1 and P(t − 1); and a bias term. threshold, SOCmin and SOCmax , respectively.
The five system states were the CL power requirements, 4) The power dispatched by the DG does not exceed its
NCL power requirements, SOC of the battery, PV power, maximum capacity and is not negative.
and wind power. The input, hidden, and output layers of the The microgrid studied herein operates in islanded and grid-
I-DEMS consisted of 21 linear neurons and 50 sigmoidal connected modes. The power balance rules for each of these
neurons. The output of I-DEMS (five linear output neurons) modes are described as follows.
is the power dispatches to the CLs, NCLs and battery, and 1) Power balance for islanded microgrid operation is as
power dispatched from the diesel generator and the grid. follows.
The initial development of the I-DEMS is based on mimick- a) If the sum of energy from RESs exceeds the total
ing the rule set of the D-DEMS using the popular supervised load dispatched, diesel operation is restricted.
learning, as shown in Fig. 10 (block diagram). b) If the total load dispatched exceeds the sum of
The power dispatched by the I-DEMS may result in a small energy from RESs, battery discharging and DG
degree of power unbalance between the load dispatched and are strictly utilized in this sequence. The battery
VENAYAGAMOORTHY et al.: DYNAMIC ENERGY MANAGEMENT SYSTEM FOR A SMART MICROGRID 1651

supplies the imbalance if its SOC(t) is greater

than SOCmin .
c) With the introduction of the battery reserve
capacity, the battery attains a minimum SOC of
SOCmin_reserve only if the energy storage device
and the diesel generator are utilized to SOCmin
and PDGmax , respectively, and cannot meet the
CL power demand.
d) If the sum of energy from RESs exceeds the
total load dispatched, battery charging occurs. The
battery consumes any imbalance if SOC(t) is less
than SOCmax .
2) Power balance for grid-connected microgrid operation is
as follows.
a) If the sum of RESs exceeds the total load dis-
patched, grid import and diesel operation are Fig. 11. Enhanced and fast I-DEMS framework to achieve near-optimal
restricted. solution. This framework uses a modified evolutionary learning algorithm.
b) Power exported to the grid does not exceed the sum
of RESs, and power imported from the grid does
not exceed the total load demand. a modified evolutionary algorithm was combined with
c) If the load dispatched exceeds the sum of RESs, ADHDP. The enhanced I-DEMS framework consisted of mul-
grid export is restricted. tiple I-DEMSs, as shown in Fig. 11; this architecture enhances
d) If the total load dispatched exceeds the sum of the speed of convergence and dispatches near-optimal controls.
energy from RESs, battery discharging, DG, and In order to implement the evolutionary I-DEMS, the selection
grid import are strictly utilized in this sequence. operator used in conventional evolutionary computing was
The battery supplies any imbalance if its SOC(t) borrowed, but the mutation operator was replaced by the
is greater than SOCmin . ADHDP FLN feedback to produce offspring.
e) If the sum of energy from RESs exceeds the Three parent I-DEMS controllers (including default
total load dispatched, battery charging and grid I-DEMS) were initialized, in which each parent produced
export occur strictly in this sequence. The bat- an offspring I-DEMS using the feedback from the FLN
tery consumes any imbalance if SOC(t) is less (Figs. 7 and 9). The parent default DEMS controller is the con-
than SOCmax . troller that resulted from the offline ADHDP process discussed
in Section IV. The learning process still continues, while
V. DYNAMIC O PTIMIZATION OF THE I-DEMS I-DEMS is operating online, however, in the online evolu-
The development of the optimal energy dispatch controller tionary optimization of DEMS (Fig. 11); the default DEMS is
in the ADHDP framework was an iterative process in which always present to make sure that the power dispatch is near-
the utility and cost-to-go functions decreased over time, thus optimal and meet the CL demand. In other words, reliable
improving the performance of the I-DEMS. In this ongoing controller is present at all times. At time instant t = 3, the
dynamic optimization process, if the total utility [given by (2)] I-DEMS yielding the lowest utility value (of now a total of
for evaluating the microgrid’s operation increased, then the six I-DEMS) was selected to dispatch optimal energy to the
I-DEMS weight updates were rejected, and the previous best microgrid system and to update the FLN. The population
weights were retained. Always, the best I-DEMS weights of parent I-DEMSs was reduced to three before the next
were used for power dispatches. The objective of the dynamic dispatch instance (one default I-DEMS and two others with
optimization of the I-DEMS was to minimize J (t) overall, lowest utility value selected from the evolutionary process).
thereby minimizing the sum of all U (t) values over the Two additional checks were incorporated into the operation.
horizon of the operation. The constant d J (t)/d J (t) = 1 was First, it was likely that the best two I-DEMSs would have
backpropagated through the FLN to obtain d J (t)/d P(t) in converged to the same weight set, resulting in equal utility
order to adjust the dispatch, P(t), so as to minimize J (t). values. Therefore, one of the controller’s weights was per-
A feedback error from the FLN to the I-DEMS was turbed by adding a random number between [−0.01, 0.01].
computed using (7), and the I-DEMS weights were This approach tries to overcome any local minimum problem
updated that may occur. The second check verified that the selection
of the dispatch adhered to the objective of the I-DEMS,
∂ J (t)
Err D E M S (t) = . (7) which was to ensure that power supplied to CL met 100%
∂ P(t) of the CL power demand. However, when the total utility,
The dynamic optimization of the I-DEMS based on U (t), was reduced but the CL dispatch did not meet total
ADHDP develops one optimal DEMS over time. In order to CL demand, the parent I-DEMS and the critic network weight
enhance the speed of convergence for finding the optimal or updates were rejected, and the previous best weights were
near-optimal policy at every dispatch time, the concept of retained.
1652 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 27, NO. 8, AUGUST 2016

TABLE I
C OMPARISON OF D-DEMS AND I-DEMS C ONTROLLERS ’ P ERFORMANCE
ON THE S ECOND D AY OF O PERATION W ITH A 35% B ATTERY SOC initial

Fig. 12. Dynamic changes of w4 (t) for determining the subutility function
UDG (t). all of NCL is met. Thus, the maximum and ideally desirable
value of (8) is 1. The higher the value of the PI, the better
the DEMS is in its dispatch. For easiness of comparison, a
VI. R ESULTS AND D ISCUSSION
normalized PI is obtained by dividing with the PIs of I-DEMS
A. Initialization of Parameters and D-DEMS with the PI of the D-DEMS

In this section, the performance of I-DEMS is compared  ( P PV (t ) +PW (t ))
( PPV (t ) +PW (t ) +PDG (t ) +PB_dschr (t ) +PG_import (t ))
with that of the D-DEMS. In this paper, the initial I-DEMS was PI =  . (8)
developed using the first day’s data and an initial battery SOC ( PNCL_D (t ))
( PNCL_S (t ))
of 35%, and the DEMS performances were evaluated using the
second day’s data (Figs. 2 and 3) and under different battery This PI evaluates two aspects. First, an indication of how
conditions. The maximum, minimum, and set point SOC of much of renewable energy was utilized to meet the load
the battery were 100%, 30%, and 65%, respectively. The initial demand. Second, an indication of how much of the control-
weights w1ini , w2ini , w3ini , w4ini , w5ini , and w6ini in the utility lable load was met.
functions were 0.3, 0.05, 0.05, 0.25, 0.25, and 0.1, respectively, To consider battery life performance, a modified perfor-
and were dynamically updated during microgrid operation. mance index considering battery life (PI-LF), given in (9),
An example of dynamic change in the weight w4 (t) for given was developed to introduce reward to the I-DEMS depending
set of operating conditions is shown in Fig. 12. The variation upon the charge and discharge performance of the battery

of w4 (t) changed the equation for computing UDG (t), as  ( P PV (t ) +PW (t ))
( P PV (t )+PW (t ) +PDG (t ) + α· P B_dschr (t ) +PG_import (t ))
shown in Algorithm 2. As it can be observed, from 12:00 A . M . PI-LF =  (9)
to 8:30 A . M ., many times the initial value of w4 (t) (0.25) ( PNCL_D (t ))
( PNCL_S (t ))
dropped to zero. At these instances, the power from renewable
where
resources and battery was unable to meet the CL demand.
Therefore, going through Algorithm 2, the value of w4 (t) was Nchr−DDEMS + Ndschr−DDEMS
α= (10)
changed to zero to relax the utility function and allow for Nchr−IDEMS + Ndschr−IDEMS
maximum dispatch of diesel generator to meet CL. where Ndschr and Nchr are the number of discharges and
The number of charges and discharges equaled one only charges, respectively.
when the battery discharged or charged continuously for
three successive dispatches, respectively. The look-ahead hori-
C. Case Studies
zon of ten steps, which was equivalent to 10 min in this paper,
was determined using the discount factor of 0.8. Microgrid operations were altered between grid-connected
and islanded modes every 5 min. This study was carried out
for a 4000-kW-min battery with initial SOC levels at 35%,
B. Performance Index 60%, and 90%. The performance of the I-DEMS on its second
To the best of our knowledge, there is no standard PI day of operation was tabulated, as shown in Tables I–III.
for evaluating the dispatch of a microgrid DEMS. Therefore, The total PV and wind generation on the second day were
a PI in (8) was developed to compare the I-DEMS and 206.88 and 208.75 kWh, respectively. The total CL and
D-DEMS according to the objectives of this paper. The PI’s controllable load requirements were 303.41 and 245.37 kWh,
numerator is the ratio of renewable energy utilization to the respectively.
total energy dispatched from all generation sources. It equals From Tables I–III, both DEMSs were able to meet 100%
one when all of the CL and controllable load requirements are of the CL power requirements from RESs, thus satisfying
met without the use energy from the diesel generator, grid, or the most important objective of supplying the CL completely.
the battery. The denominator of the PI is the ratio of NCL Moreover, I-DEMS outperformed the D-DEMS in terms of
demand to the NCL met. The denominator equals one when the power dispatched to the controllable load, as well as the
VENAYAGAMOORTHY et al.: DYNAMIC ENERGY MANAGEMENT SYSTEM FOR A SMART MICROGRID 1653

TABLE II TABLE IV
C OMPARISON OF D-DEMS AND I-DEMS C ONTROLLERS ’ P ERFORMANCE C OMPARISON OF P ERFORMANCE ( FOR 1 min OF D ISPATCH ) OF THE
FOR THE S ECOND D AY OF O PERATION W ITH I-DEMS C ONTROLLER W ITH AND W ITHOUT THE M ODIFIED
A 60% B ATTERY SOC initial E VOLUTIONARY A LGORITHM -BASED L EARNING AND
THE D-DEMS C ONTROLLER

TABLE III
C OMPARISON OF D-DEMS AND I-DEMS C ONTROLLERS ’ P ERFORMANCE
FOR THE S ECOND D AY OF O PERATION W ITH
A 90% B ATTERY SOC initial

Fig. 13. Comparison of load dispatch of the D-DEMS and I-DEMS

controllers. (a) D-DEMS (blue line) and I-DEMS (red line) meet the
CL demand for the entire day. (b) I-DEMS has a smooth switching behavior of
the controllable load, whereas the D-DEMS is drastic in its switching behavior
performance of the battery storage device, which means more of the controllable load.
customer satisfaction. Increments of 5%–7% in supplying
controllable load were observed with the I-DEMS, at the
expense of a small amount of diesel and grid dispatches. implement the I-DEMS in real time. With a foreseen imple-
Furthermore, I-DEMS was able to minimize the utilization mentation on a graphic processing unit platform, it is expected
of DG and grid generation as well, thus meeting the goal of that performance can be significantly improved. This remains
self-sustainability. On the other hand, the D-DEMS used less to be investigated as part of future work.
diesel and grid energy at the expense of reduced battery life The energy dispatched by the D-DEMS and I-DEMS for an
and lower satisfaction of the controllable load requirement. initial SOC of 35% on the second day of operation is shown
As it can be observed, in all cases, the dynamically opti- in Figs. 13–15.
mized I-DEMS performed superior, meeting 100% of the vary- Fig. 15 shows that the battery power dispatched with
ing CL demand from renewable energy sources, meeting more the I-DEMS was less than 25 kW at any time, compared
of the controllable load, maintaining minimal transactions with with over 35-kW battery power dispatched by the D-DEMS.
the diesel generator and the grid, as well as a maintaining mean Furthermore, the battery life cycle has clearly been improved
SOC level, while extending the battery life. in I-DEMS due to a fewer number of charges and discharges,
Table IV shows that performance of the I-DEMS on its thus reducing operation and maintenance costs.
minute dispatches, with and without the evolutionary strategy. The performance of both DEMSs for a 3000-kW-min bat-
The evolutionary strategy was able to enhance the performance tery with various initial SOCs, from 35% to 95%, was studied
by 0.11%, at an increased computational cost of 80 ms. and shown in Fig. 16. As it can be observed, I-DEMS always
Thus, it is expected that the evolutionary algorithm with performed better than the conventional D-DEMS. I-DEMS
three parents will find a better I-DEMS over a longer period of was able to smoothly and continuously accomplish higher
time than with a single I-DEMS. Furthermore, knowing that performance, while initial SOC was increased. However,
the scheduled dispatches are for the following minute and the D-DEMS did not perform well until the battery SOC level
necessary computation time is <1 s, it is possible to increase reached a certain percentage. Moreover, I-DEMS has achieved
the population size of the evolutionary strategy and still a better performance (PI-LF) in terms of battery life cycle,
1654 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 27, NO. 8, AUGUST 2016

Fig. 17. Normalized PI [PI using (8)] for different battery SOCs and sizes.

VII. C ONCLUSION
Fig. 14. Comparison of generation dispatch of the D-DEMS (blue line)
and I-DEMS (red line) controllers. (a) Both controllers have a similar diesel The development of an I-DEMS for smart microgrid oper-
generator usage. (b) I-DEMS has a slightly greater usage (positive kilowatts ation has been presented. The ADHDP approach, based on
represent import) of grid power. combined concepts of adaptive dynamic programming and
reinforcement learning, was utilized to evolve an optimal
control policy and an approximate cost-to-go function for
microgrid operation, with variable and uncertain renewable
energy generation and varying CL and controllable load pro-
files. A modified evolutionary computing learning approach
was introduced to speed up the convergence for finding near-
optimal control policy and cost-to-go function during online
operation as it becomes necessary to enhance performance.
The performance of the I-DEMS was compared with that
Fig. 15. Battery dispatch comparison between D-DEMS and I-DEMS. of a DEMS developed using a DT-based approach under
seen and unseen operating conditions. The I-DEMS, while
satisfying the primary goal of meeting 100% of the CL demand
requirements, still managed to improve the energy dispatched
to controllable loads, and its dispatch strategy extended the
lifecycle of the battery. This means that microgrids of the
future can be managed intelligently to be self-sustainable,
reliable, and environmental friendly.
Future work will focus on extending the I-DEMS framework
to include dynamic state prediction, and on carrying out a real-
time implementation to coordinate the set-point active power
dispatches with reactive power controls.

R EFERENCES
[1] A. Dimeas and N. Hatziargyriou, “A multiagent system for microgrids,”
in Proc. IEEE Power Eng. Soc. General Meeting, vol. 1. Jun. 2004,
Fig. 16. Performance comparison of DEMSs (3000 kW-min). pp. 55–58.
[2] G. P. Henze and R. H. Dodier, “Adaptive optimal control of a grid-
independent photovoltaic system,” J. Solar Energy Eng., vol. 125, no. 1,
pp. 34–42, 2003.
[3] S. Teleke, M. E. Baran, S. Bhattacharya, and A. Q. Huang, “Rule-based
which allows the battery to last longer and avoid additional control of battery energy storage for dispatching intermittent renewable
operation and maintenance cost. sources,” IEEE Trans. Sustain. Energy, vol. 1, no. 3, pp. 117–124,
The normalized PI was used to compare the performance Oct. 2010.
of I-DEMS with D-DEMS. Fig. 17 shows the normalized PI [4] D. Velasco de la Fuente, C. L. T. Rodríguez, G. Garcerá, E. Figueres,
and R. O. González, “Photovoltaic power system with battery backup
for different battery sizes and initial SOCs. It is observed for with grid-connection and islanded operation capabilities,” IEEE Trans.
all possible combinations I-DEMS resulted in a greater than Ind. Electron., vol. 60, no. 4, pp. 1571–1581, Apr. 2013.
one per-unit normalized PI. In other words, I-DEMS provides [5] C. M. Colson, M. H. Nehrir, and S. A. Pourmousavi, “Towards real-
time microgrid power management using computational intelligence
better performance under different operating conditions inde- methods,” in Proc. IEEE Power Energy Soc. General Meeting, Jul. 2010.
pendent of battery size and initial SOC. pp. 1–8.
VENAYAGAMOORTHY et al.: DYNAMIC ENERGY MANAGEMENT SYSTEM FOR A SMART MICROGRID 1655

[6] M. Tekin, D. Hissel, M.-C. Pera, and J. M. Kauffmann, [30] S. Ray, G. K. Venayagamoorthy, B. Chaudhuri, and R. Majumder,
“Energy-management strategy for embedded fuel-cell systems using “Comparison of adaptive critic-based and classical wide-area controllers
fuzzy logic,” IEEE Trans. Ind. Electron., vol. 54, no. 1, pp. 595–603, for power systems,” IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38,
Feb. 2007. no. 4, pp. 1002–1007, Aug. 2008.
[7] S. A. Pourmousavi, M. H. Nehrir, C. M. Colson, and C. Wang, “Real- [31] D. Tran and A. M. Khambadkone, “Energy management for lifetime
time energy management of a stand-alone hybrid wind-microturbine extension of energy storage system in micro-grid applications,” IEEE
energy system using particle swarm optimization,” IEEE Trans. Sustain. Trans. Smart Grid, vol. 4, no. 3, pp. 1289–1296, Sep. 2013.
Energy, vol. 1, no. 3, pp. 193–201, Oct. 2010. [32] M. C. Bozchalui and R. Sharma, “Optimal operation of commercial
[8] Y. Riffonneau, S. Bacha, F. Barruel, and S. Ploix, “Optimal power flow building microgrids using multi-objective optimization to achieve emis-
management for grid connected PV systems with batteries,” IEEE Trans. sions and efficiency targets,” in Proc. IEEE Power Energy Soc. General
Sustain. Energy, vol. 2, no. 3, pp. 309–320, Jul. 2011. Meeting, Jul. 2012, pp. 1–8.
[9] A. Chaouachi, R. M. Kamel, R. Andoulsi, and K. Nagasaka, “Multi-
objective intelligent energy management for a microgrid,” IEEE Trans.
Ind. Electron., vol. 60, no. 4, pp. 1688–1699, Apr. 2013.
[10] C. Chen, S. Duan, B. Liu, and G. Hu, “Smart energy management system
for optimal microgrid economic operation,” IET Renew. Power Generat.,
vol. 5, no. 3, pp. 258–267, 2011.
[11] A. Mohamed, V. Salehi, and O. Mohammed, “Real-time energy man-
agement algorithm for mitigation of pulse loads in hybrid microgrids,”
IEEE Trans. Smart Grid, vol. 3, no. 4, pp. 1911–1922,
Dec. 2012. Ganesh Kumar Venayagamoorthy (S’91–M’97–
[12] P. García, J. P. Torreglosa, L. M. Fernández, and F. Jurado, SM’02) was a Professor of Electrical and Com-
“Optimal energy management system for stand-alone wind tur- puter Engineering with the Missouri University of
bine/photovoltaic/hydrogen/battery hybrid system with supervisory con- Science and Technology, Rolla, MO, USA, from
trol based on fuzzy logic,” Int. J. Hydrogen Energy, vol. 38, no. 33, 2002 to 2011. He is currently the Duke Energy
pp. 14146–14158, 2013. Distinguished Professor of Power Engineering and
[13] M. Motevasel and A. R. Seifi, “Expert energy management of a micro- a Professor of Electrical and Computer Engineering
grid considering wind energy uncertainty,” Energy Convers. Manage., and Automotive Engineering with Clemson Univer-
vol. 83, pp. 58–72, Jul. 2014. sity, Clemson, SC, USA. He is also the Founder
[14] P. Siano, “Real time operation of smart grids via FCN networks and Director of the Real-Time Power and Intelli-
and optimal power flow,” IEEE Trans. Ind. Informat., vol. 8, no. 4, gent Systems Laboratory with Clemson University.
pp. 944–952, Nov. 2012. He is an Honorary Professor with the School of Engineering, University of
[15] R. Palma-Behnke et al., “A microgrid energy management system based KwaZulu–Natal, Durban, South Africa. He is the top author of the College
on the rolling horizon strategy,” IEEE Trans. Smart Grid, vol. 4, no. 2, of Engineering and Science at Clemson University, according to the Elsevier
pp. 996–1006, Jun. 2013. Expert Portal, and was the Most Cited Researcher of Clemson University by
[16] R. L. Welch and G. K. Venayagamoorthy, “Energy dispatch fuzzy ResearchGate in November and December of 2015. He has been involved in
controller for a grid-independent photovoltaic system,” Energy Convers. over 65 sponsored projects in excess of U.S. $10 million. He has authored over
Manage., vol. 51, no. 5, pp. 928–937, 2010. 475 refereed technical articles. His publications are cited about 10 000 times
[17] R. L. Welch and G. K. Venayagamoorthy, “Energy dispatch controllers with an h-index of 51. His current research interests include the development
for a photovoltaic system,” Eng. Appl. Artif. Intell., vol. 23, no. 2, and innovation of advanced computational methods for smart grid operations,
pp. 249–261, 2010. including intelligent sensing and monitoring, power system optimization,
[18] T. Huang and D. Liu, “Residential energy system control and manage- stability and control, and signal processing.
ment using adaptive dynamic programming,” in Proc. Int. Joint Conf. Dr. Venayagamoorthy is a fellow of the Institution of Engineering and Tech-
Neural Netw. (IJCNN), Jul./Aug. 2011, pp. 119–124. nology (IET), U.K., and the South African Institute of Electrical Engineers.
He received the U.K. IET Generation, Transmission and Distribution Premier
[19] J. Si, A. G. Barto, W. B. Powell, and D. Wunsch, Eds., Handbook
Award for the best research paper published in 2010/2011 for the paper titled
of Learning and Approximate Dynamic Programming. New York, NY,
Wide Area Control for Improving Stability of a Power System With Plug-In
USA: Wiley, 2004.
Electric Vehicles in 2012. He is involved in the leadership and organization of
[20] R. E. Bellman, Dynamic Programming. Princeton, NJ, USA: many conferences, including the General Chair of the Annual Power System
Princeton Univ. Press, 1957. Conference in Clemson since 2013, and the Pioneer and Chair/Co-Chair of the
[21] P. J. Werbos, “Advanced forecasting methods for global crisis warning IEEE Symposium of Computational Intelligence Applications in Smart Grid
and models of intelligence,” General Syst. Yearbook, vol. 22, pp. 25–38, since 2011. He is the Chair of the IEEE Power & Energy Society Working
1977. Group on Intelligent Control Systems, and the Founder and Chair of the IEEE
[22] R. F. Stengel, Optimal Control and Estimation. New York, NY, USA: Computational Intelligence Society Task Force on Smart Grid. He has served
Dover, 1994. as an Editor/Guest Editor of several IEEE and Elsevier journals.
[23] J. Si and Y.-T. Wang, “Online learning control by association and
reinforcement,” IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264–276,
Mar. 2002.
[24] J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, “Adaptive dynamic
programming,” IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 32,
no. 2, pp. 140–153, May 2002.
[25] J. J. Murray, C. J. Cox, and R. E. Saeks, “The adaptive dynamic
programming theorem,” in Stability and Control of Dynamical
Systems With Applications. Boston, MA, USA: Birkhäuser, 2003, Ratnesh K. Sharma (M’11) received the B.Tech.
pp. 379–394. (Hons.) degree from IIT Kharagpur, Kharagpur,
[26] G. G. Lendaris and C. Paintz, “Training strategies for critic and action India, and the Ph.D. degree from the University of
neural networks in dual heuristic programming method,” in Proc. IEEE Colorado Boulder, Boulder, CO, USA.
Int. Conf. Neural Netw., vol. 2. Jun. 1997, pp. 712–717. He leads the Energy Management Department with
[27] P. J. Werbos, “Consistency of HDP applied to a simple reinforcement NEC Laboratories America Inc., Cupertino, CA,
learning problem,” Neural Netw., vol. 3, no. 2, pp. 179–189, 1990. USA. He has authored over 300 papers/technical
[28] L. Yang, R. Enns, Y.-T. Wang, and J. Si, “Direct neural dynamic reports and holds over 100 U.S. patents. His current
programming,” in Stability and Control of Dynamical Systems With research interests include sustainable energy man-
Applications. Boston, MA, USA: Birkhäuser, 2003, pp. 193–214. agement in electricity, buildings, and transportation
[29] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. sectors, including energy conversion, power systems,
Cambridge, MA, USA: MIT Press, 1998. communications, and analytics.
1656 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 27, NO. 8, AUGUST 2016

Prajwal K. Gautam (A’14) received the bachelor’s Afshin Ahmadi (S’14) received the B.Sc. degree
degree in electrical engineering from the Institute in electrical engineering from Islamic Azad Uni-
of Engineering, Tribhuvan University, Nepal, India, versity, Tehran, Iran, in 2006, the M.B.A. degree
in 2006, and the M.S. degree in electrical engineer- from Silliman University, Dumaguete, Philippines,
ing from Clemson University, Clemson, SC, USA, in 2009, and the M.S. degree in electrical engi-
in 2014. neering from the University of the Philippines,
He was a Graduate Research Assistant with the Diliman, Philippines, in 2012. He is currently pursu-
Real-Time Power and Intelligent Systems Labora- ing the Ph.D. degree with the Department of Electri-
tory, Clemson University, from 2011 to 2013, under cal and Computer Engineering, Clemson University,
the guidance of Dr. Venayagamoorthy. He is cur- Clemson, SC, USA.
rently a Power System Engineer with Spirae, Inc., He was a Graduate Research Assistant with the
Fort Collins, CO, USA. His current research interests include distributed Real-Time Power and Intelligent Systems Laboratory, Clemson University,
energy resources management system, microgrid, power systems modeling, from 2014 to 2015, under the guidance of Dr. Venayagamoorthy. His current
and optimization. research interests include smart grid, intelligent system applications to power
systems, renewable energy integration, and power system optimization and
economics.