Received 1 July 2024, accepted 15 July 2024, date of publication 22 July 2024, date of current version 31 July 2024.

Digital Object Identifier 10.1109/ACCESS.2024.3432120

Optimal Energy Management of Data Center

Micro-Grid Considering Computing
Workloads Shift
LUYAO LIU 1,2 , XINWEI SHEN 2, (Senior Member, IEEE), ZHIGANG CHEN1 , QIE SUN3,4 ,
AND RONALD WENNERSTEN4
1 China Energy Engineering Group Guangdong Electric Power Design Institute Company Ltd., Guangzhou, Guangdong 510700, China
2 Shenzhen International Graduate School, Tsinghua University, Shenzhen, Guangdong 518055, China
3 Institute of Thermal Science and Technology, Shandong University, Jinan, Shandong 250061, China
4 Institute for Advanced Science and Technology, Shandong University, Jinan, Shandong 250061, China

Corresponding author: Luyao Liu ([email protected]; [email protected])

This work was supported by the Postdoctoral Research Project of Guangdong Electric Power Design Institute under Grant EV10961W.

ABSTRACT Data centers have been experiencing increasingly significant challenges in electricity con-
sumption and carbon emissions with the fast-development of artificial intelligence (AI). In the context of
carbon neutrality, the integration of data centers with renewable green energy has become a prevailing
trend. To effectively integrate renewable energy, it is imperative to thoroughly explore the data center’s
operation flexibility. Delay-tolerant computational workloads have been considered as one of the most
promising flexible resources for power regulation within data center micro-grids (DCMs). This paper first
analyzes the working characteristics of three kinds of typical delay-tolerant computing workloads, i.e. short-
running deferrable workloads, long-running continuous workload, long-running interruptible workload, and
then clarifies the time-shifting mechanisms for each. Next, the corresponding time-shifting models of the
delay-tolerant workloads are established. Finally, considering the time attributes of workloads and system
settings, the day-ahead optimization scheduling framework of DCM incorporating the time-shifting models
of multiple workloads are formulated, with the aim of minimizing the operation cost of DCM and renewable
power curtailment. Application of the power management scheme in a data center case study confirms its
effectiveness in improving the operational economy of data center and increasing green energy utilization.

INDEX TERMS Data center micro-grid, renewable energy, delay-tolerant workloads, time shift.

I. INTRODUCTION of electricity, comprising nearly 2% of the global electricity

Data centers are key facilities for storing, transmitting, and demand [2]. In the US, data centers consume 200TWh of
processing data, providing important support for emerging electricity in 2022, accounting for about 4% of the total
technologies such as cloud computing and artificial intelli- national electricity consumption. In China, the power usage
gence (AI). They are the cornerstone of today’s information of data centers surpassed 216.6 TWh in 2021, representing
society [1]. Recent years have witnessed a rapid advance roughly 2.6% of the total societal electricity consumption.
in the AI industry, whose increasing demand for computing The CO2 emissions from data centers reach approximately
power has promoted the further development of data cen- 135 million tons, constituting approximately 1.14% of the
ters. However, power consumption and carbon emissions of national CO2 emissions [3]. With the continuously expand-
data centers are significant problems. According to statistics, ing of energy-intensive AI generation models training and
data centers worldwide consume approximately 460 TWh cryptocurrency mining, the power consumption and carbon
emissions of data centers can get more aggravated [4].
The associate editor coordinating the review of this manuscript and To addressee these challenges, the integration of data
approving it for publication was Reinaldo Tonkoski . centers with renewable green energy sources has become a
2024 The Authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/
VOLUME 12, 2024 102061
 L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

prominent trend [5], [6]. The policy of ‘Channels computing In existing studies, scholars have studied utilizing the spa-
resources from the east to the west’ proposed in 2021 by tial adjustability of computing workloads to promote renew-
China could promote the layout of data centers in the west- able energy consumption. Chen et al. [12] simplified the
ern regions where own abundant renewable wind and solar topology of computing power networks from an energy per-
energy sources, and thereby increase the penetration of green spective. They regarded the computing power loads of geo-
energy [7]. To effectively incorporate the intermittent renew- graphically dispersed data centers as concise electricity load
able energy, it is essential to extensively investigate the oper- nodes and computing power flow as virtual electricity power
ation flexibility of data centers [8]. Fig.1 presents a structural flow to describe the spatial coupling of computing power
overview of a green data center micro-grid (DCM). The IT loads, and then employed price leverage to guide the spa-
equipment offers computing resource for the interactive and tial shift of power loads among interconnected data centers,
batch processing workloads, while the refrigeration system aiming to boost renewable energy consumption and enhance
provides cooling resources to extract the heat generated by the grid resilience. Yang et al. [13] proposed a spatial migration
IT equipment. The power infrastructure connects the wind, mechanism of workloads based on the spatiotemporal dis-
photovoltaic (PV) power system, along with the main-grid on tribution complementarity of renewable energy in multiple
the supply side, and the IT equipment, refrigeration system on regions around the world, and conducted simulations using
the power demand side. Google’s interconnected multiple data centers to minimize
carbon emissions. The literature, exemplified by the afore-
mentioned research primarily focuses on the spatial transfer-
ence of workloads among multiple data centers, which pays
little attention to temporal shift of workloads within single
data centers.
Studies regarding the temporal transferability modeling of
time-shiftable delay-tolerant workloads within a single data
center have been conducted. Cupelli et al. [14] considered the
delay-tolerant workloads’ arrival, queuing, execution process
and simulated their power consumption. Through collabora-
tive regulation with cooling units and energy storage devices,
the energy costs in data centers was reduced. Wang et al. [15]
also modelled the delay-tolerant workloads considering their
FIGURE 1. Structure overview of a data center micro-grid. arrival time, execution time, and deadline of different tasks.
The operation scheduling optimization model of an integrated
The computing workloads within data centers are energy system with the computing task transfer is proposed
considered having significant spatiotemporal adjustable char- to minimize the operating costs of data center and maximize
acteristics. According to the length of response delay, the the user satisfaction level of computing tasks. Cao et al. [16]
computing workloads can be categorized into delay-sensitive collected real workload data from Alibaba data center clusters
and delay-tolerant types [9]. Delay-sensitive workloads and considered a kind of delay-tolerant workload that cannot
mainly refer to ‘online tasks’, or ‘interactive workloads’. be manually interrupted during execution. They formulated
Common delay-sensitive tasks include online transactions, the time shift model of the delay-tolerant workload with
web browsing, and video live streaming, etc. This type of consideration of relative time parameters. The model was
workloads cannot be postponed in the time dimension, but embedded into a power scheduling model of a renewable
can be transferred to other geographical locations for pro- data center energy system, aiming to minimize the opera-
cessing in the spatial dimension using high-speed fiber optic tion cost of the system and start-up/down cost of fossil-fuel
technology. By adjusting the workloads’ processing location, generators.
the computing and electricity demand within data centers Research represented by the above studies have demon-
can be changed accordingly, allowing for flexible regulation strated the flexible regulation potential of delay-tolerant
of the power [10]. Delay-tolerant workloads mainly refer to workloads within data centers. However, existing literature
‘offline tasks’ or ‘batch processing workloads’, which are only conducted plain modelling of delay-tolerant workloads
executed based on a batch input through a computer program with simple time shift patterns, and hasn’t comprehensively
without human intervention. Its maximum response time investigated multiple working characteristics and time-shift
can reach several minutes to days. Delay-tolerant workloads mechanisms of different delay-tolerant workloads. The
mainly include image processing, AI training and inference, research of complete time-shift modeling of delay-tolerant
scientific research simulation, and large-scale data analysis, workloads with various migration mechanisms, as well as
etc. Within the response deadline, by dynamically adjusting thorough investigation of embedding these models into data
the start time of execution and the computing amount during center power dispatching models is lack, which hinders the
each time period, the power consumption of workloads could cognition and further utilization of the flexible resources of
be controlled [11]. computing workloads in data centers.

102062 VOLUME 12, 2024

L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

In light of the deficiencies, our research aims to provide computing tasks, when combined, can effectively help to mit-
a power scheduling framework for data center micro-grid igate fluctuations in renewable energy. Analysis also shows
(DCM) that incorporates the time-shift models of various that long-running workloads generally occupy a small portion
delay-tolerant workloads. To this aim, this work first sort of computing workloads but consume a significant amount
out several types of delay-tolerant batch processing work- of processor resources [20]. These energy-intensive long-
loads with different working characteristics and time-shift running workloads therefore present a notable potential for
mechanisms. Then, the key time shift parameters are iden- power regulation.
tified and the shift process models are established for each
kind of delay-tolerant workload. Last, the day-ahead power 1) SHORT-RUNNING DEFERRABLE WORKLOADS
scheduling optimization model of a renewable DCM incorpo- Some workloads like scheduled data backups, log clean-
rating the power regulation models of computing workloads ing, and email sending only require brief execution time,
is constructed, with the aim of maximizing operation econ- but don’t need to execute immediately, which is catego-
omy and utilization level of renewable energy. To verify the rized as ‘short-running deferrable workload’. The differ-
effectiveness of model and assess the benefits of workloads ence between short-running deferrable workload and interac-
shift, the proposed power management model of DCM is tive workload is that the former doesn’t ask for immediate
applied in a data center case and implemented under various response, and can be delayed by a few minutes to several
scenarios. hours. Users may convert some interactive workloads to
The remainder of this paper is structured as fol- short-running deferrable workloads by changing the response
lows: Section II analyzes the working features of various deadline.
delay-tolerant workloads and establishes the time shift mod- The short-running deferrable workload has the character-
els for them. This section also describes the relation between istics of short duration and large quantity. A large number
computing power and electricity power, with which the of short-running workloads assemble together forming the
time shift models can convert to power regulation models. short-running workloads cluster, which could provide power
Section III provides the details of day-ahead power schedul- regulation potential. Thus, we take the short-running work-
ing optimization model of DCM considering workloads shift. loads cluster as the research objective rather than a single
Section IV reports the results after power scheduling opti- short-running workload. The time shifting mechanism for
mization and analyzes the benefits of workloads shift. Finally, short-running deferrable workloads is to shift these work-
Section V summarizes the paper and provides concluding loads, either wholly or partially, from their original time
remarks. periods to subsequent time periods.

II. MODELLLING OF DELAY-TOLERENT WORKLOADS FOR 2) LONG-RUNNING CONTINUOUS WORKLOAD

POWER REGULATION For some workloads, such as test suites, compilation jobs,
This section clarifies time shift mechanisms of delay-tolerant database migration, and backups, they cannot be paused
workloads with distinct working characteristics, i.e. short- or interrupted during their execution and must be executed
running deferrable, long-running continuous, long-running within a continuous period of time. This type of workload
interruptible workloads, and formulates time shift mod- typically requires a long period to ensure stable system oper-
els for them. It also introduces the conversion expression ation. Workloads with this working characteristic are called
between computing power and electricity power, enabling the ‘long-running continuous workload.’
time shift models to be transformed into power regulation The temporal mitigation pattern for the workload is to shift
models. the overall workload to a certain time range while keeping the
original shape of the computing power load curve unchanged.
A. DELAY-TOLERANT WORKLOADS WITH VARIOUS
WORKING PROPOTIES AND SHIFT MECHANISMS 3) LONG-RUNNING INTERRUPTIBLE WORKLOAD
The most important properties for determining a delay- For certain workloads, e.g. machine learning training, block-
tolerant workload’s shifting capability are their time con- chain mining, protein folding, or scientific simulations, they
straints. The time-relative properties of duration and inter- can be paused or interrupted at some point during their work-
ruptibility have a substantial impact on whether and how a ing process, and resumed at an appropriate time. Workloads
workload can be shifted in time [17]. This section catego- with this working characteristic are referred to as ‘long-
rizes delay-tolerant workloads based on these characteristics. running interruptible workloads.’
Distinguished by duration time length, time-shiftable delay- According to its time-shift characteristics, long-running
tolerant workload tasks are categorized into short-running interruptible workloads can be viewed as a holistic task com-
and long-running workloads. Considering the interruptibility, prised of multiple sequential subtasks. The shift strategy for
the long-running workloads can further classified into contin- managing such workloads is adjusting each subtask forward
uous and interruptible workloads [18]. or backward to a certain time point within a specified time
Analysis has revealed that 90% of batch jobs in Alibaba’s frame, while ensuing the sequential order dependency among
data cluster ran for less than 15 minutes [19]. Short-running the subtasks.

VOLUME 12, 2024 102063

 L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

B. SHORT-RUNNING DEFERRABLE WORKLOADS SHIFT Dori

A . Eq. (5) indicates that the transpose of the sum of ele-
MODELLING ments in all the columns of matrix UA equals to the computing
shift
Introduce Dori
A to denote the original computing power power demand of workloads after shift DA . In Eq. (6),
mat
VA is a constant matrix with the same dimension as the
demand time sequence of short-running deferrable work-
shift
loads, and DA to denote the computing power demand time variable matrix UA . To illustrate the specific form of VAmat ,
sequence of short-running workloads after shifting. Dori
A and
we first introduce a constant value H to represent the max-
shift imum deferrable time periods of short-running workloads.
DA are represented as follows:
h i The short-running workloads of the original time points are
0
A = dA , dA , . . .dA , . . . ,dA
1 2
Dori k
(1) allowed to be fully or partially transferred to the 1st to H th
shift 0 time point thereafter. In VAmat , when k = 1, 2, . . . , 0 − H , the
DA 1
= [dA∗ , dA∗
2
, . . . dA∗
t
, . . . , dA∗ ] (2)
elements from ‘column k, row k,’ to ‘column k, row k + H ’;
where DoriA is a constant vector with a dimension of 1 0. and when k = 0−H+1, 0 −H +2, . . . , 0, the elements from
∗

In this paper, the day-ahead scheduling period is set to be ‘column k, row k’ to ‘column k, row 0’, and from ‘column k,
24h, and the scheduling time step and scheduling periods row 1’ to ‘column k, row H − 0 + k’, are filled with the value
per scheduling day is recorded as 1t and 0. dA1 , dA2 , dAk of 1, while the elements at the remaining positions are filled
are the values of computing power demand of short-running with value of 0. Set the maximum deferrable time periods H
workloads at original status at time point of 1, 2, and k. to be 11, the VAmat can be specifically written as Eq. (7). VAmat
shift
DA is a variable with a dimension of 1∗ 0. dA∗ 1 , d 2 , d t are
A∗ A∗ changes as the parameter H varies. One can construct the
the values of computing power demand of the short-running matrix VAmat according to the concrete value of parameter H.
workloads after shifting at the time period of 1, 2, and t.
... ... ...
 
The time shifting pattern for short-running workloads is 1 0 1 1
shifting all or part of the tasks from the original time periods  1 1 ... ... ... 1 1 
... ... ... ... ... ... ...
 
to next several time periods. The key parameter for the time
... ... ...
 
shift modelling of short-running workloads is the time shift  1 1 1 1 
... ... ...
 
matrix variable of the inflow and outflow amount of comput-  1 1 0 1 
VAmat ... ... ...
 
ing power demand at each time point.  1
= 1 0 0 
 (7)
Use ut,k
A to represent the amount of computing power
 0 1 ... ... ... 0 0 
... ... ...
 
demand of short-running workloads that is transferred from  0 0 0 0 
... ... ... ... ... ... ...
 
the original time point k to another time point of t. The time
... ... ...
 
shift matrix variable of the inflow and outflow amount of  0 0 1 0 
computing power demand UA with a dimension of 0 ∗ 0 is 0 0 ... ... ... 1 1
formed, which is expressed as:
 1,1 1,2 1,k 1,0−1 1,0
 M is a constant value, equal to the peak computing power
uA uA ... uA . . . uA uA
 2,1 demand dAmax of the short-running workloads. Eq. (6) indi-
 uA u2,2 ... u2,k . . . u2,0−1 u2,0

A A A A  cates that the transferred computing power demand from the
 ... ... ... ... ... ... ... 
 
 t,1 original time point to the new time period should be less than
ut,2 ···... ut,k ... ut,0−1 ut,0 dAmax and greater than 0.

 uA A A A A

 
 ut+1,1 ut+1,2 ... ut+1,k . . . ut+1,0−1 ut+1,0  To prevent invalid shifting, a penalty cost function Cdata
 A A A A A
is established, as shown in Eq. (8). WAmat is the penalty cost

UA =  ut+2,1 ut+2,2 ... ut+2,k . . . ut+2,0−1 t+2,0 
 (3)

a u
 At+3,1 A A A
coefficient matrix with a dimension of 0 ∗ 0, as expressed by
ut+3,2 . . . · · · ut+3,k . . . ut+3,0−1 ut+3,0

u 
 A A A A A  Eq. (9).
 ut+4,1 t+4,2
uA . . . uA t+4,k t+4,0−1 t+4,0 
. . . uA uA
 A 
 ... ... ... ... ... ... ...  0 X
0

 0−1,1
 X UA . ∗ W mat
u0−1,2 u0−1,k . . . u0−1,0−1 u0−1,0 cpu ∗ 1t
∗ Pmax
A
... Cdata = (8)

uA A A A A Dmax

u0,1 u0,2 0,k 0,0−1 0,0 cpu
A A ... uA . . . uA uA t=1 k=1
... ... ...
 
0 0 0.05 0.05
UA should satisfies the following constraints:  0.05 0 ... ... ... 0.05 0.05 
0  ... ... ... ... ... ... ... 
 
X
Dori
A = UA (4) 
 0.05 0.05 ... ... ... 0.05

0.05 
... ... ...
t=1
 
 0.05 0.05 0 0.05 
0 T
WAmat ... ... ...
 
shift
X  0.05
= 0.05 0 0  (9)
DA =[ UA ] (5) ... ... ...

 0 0.05 0 0 
... ... ...
 
k=1  0 0 0 0 
0 ≤ UA ≤ VAmat ∗ M  ... ... ... ... ... ... ... 
 
(6)
... ... ...
 
 0 0 0 0 
Eq. (4) states that the sum of elements in all the rows of
matrix UA equals to the original computing power demand 0 0 ... ... ... 0.05 0

102064 VOLUME 12, 2024

L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

In Eq.(9), Dmaxcpu represents the maximum computing power of 1. UBindex represents the time period in which the start node
of the CPU chip, which is 1600 GFLOPS in this paper, is positioned, as expressed by Eq.(13). In Eq. (14), te and tl
as indicated in Table 1. Pmaxcpu represents the maximum power are the earliest start time point and the latest start time point
consumption of the CPU used in this study, which is 0.15kW, of the long-running continuous workload.
as indicated in Table 1. Setting the unit penalty cost coeffi- The relationship between the computing power demand
cient for the transfer amount of computing power demand time series vector variable of the long-running continuous
shift
at all possible time points to be 0.05 /kWh, when k = workload after shift DB and the time-shift vector of the start
1, 2, . . . , 0 − H , the elements in positions of WAmat from the node of workload UB is expressed as:
‘column k, row k + 1’ to ‘column k, row k + H ’; and when shift
k = 0 − H + 1, 0 − H + 2, . . . , 0 − 1, elements from DB = UB ∗ MBmat (15)
the ‘column k, row k +1’ to ‘column k, row 0’ and from shift
where the dimension of DB is 0 ∗ 0;MBmat is a constructed
‘column k, row 1,’ to ‘column k, row H − (0 − k)’; and when
auxiliary constant matrix, with a dimension of 0 ∗ 0, which is
k = 0, elements from the ‘column k, row 1’ to ‘column k,
constructed based on Dmat mat
B . MB is expressed as Eq.(16):
row H ’ are filled with 0.05; while the other elements are set to
be 0, as expressed in Eq. (9). One can adjust the matrix VAmat ... ...
 1 2 K

dB dB dB 0 0 0
according to concrete parameter value of the unit penalty cost  0 dB1 ... dBK −1 dBK ... 0 0 
coefficient.
 
 0
 0 ... ... ···... ... 0 0 
 0 0 ... ... ... ... 0 0 
C. LONG-RUNNING CONTINUOUS WORKLOAD SHIFT MBmat = ...
 
(16)
... ... ... ... ... ... ... 
MODELING  
 0 0 ... ... ... ... 0 0 
Use Dmat
B to denote the sequence of the long-running con-  
 0 0 ... dB1 dB2 . . . dBK 0 
 
tinuous workload. Dmat
B is expressed as Eq. (10), which is a
constant vector. 0 0 ... 0 dB1 . . . dBK −1 dBK

B = [dB , dB , . . . dB , . . . , dB ]
1 2
Dmat k K
(10) In MBmat , the elements from ‘column k, row k’ to ‘row k,
where K represents the processing time length of the column k + K -1’ (k = 1,2,. . . , 0+1-K ) are filled with the
long-running continuous workload, which is a constant value computing power demand values corresponding to the start
shorter than 0. dB1 represents the computing power demand node and subsequent nodes of the long-running continuous
value at the starting node of the long-running continu- workload, and the elements at the remaining positions are 0.
ous workload; dB2 , dBk , dBK represents the computing power
demand value at the 2nd , k th , K th time points following the D. LONG-RUNNING INTERRUPTIBLE WORKLOAD SHIFT
start node of the workload. MODELING
The mitigation pattern for the long-running continuous According to the time-shift characteristics of long-running
workload is to shift the overall workload over the time axis. interruptible workload, this study considers the long-running
Once the start time of the workload is determined, the com- interruptible workload as a holistic task formed by multiple
puting power demand for each subsequent time point can also subtasks in sequence. The duration of each subtask is consis-
be uniquely ascertained. Therefore, the pivotal parameter for tent with the scheduling time step 1t. The processing time
the time-shift modelling of long-running continuous work- length of the long-running interruptible workload is shorter
load is the time-shift vector variable of its start node. than 0. Use a constant value K to represent the processing
Use UB to represent whether the start node of the time length, and the long-running interruptible workload can
long-running continuous workload is located at a certain time be regarded as a holistic task formed by K subtasks. Using dC1 ,
point. UB is a variable vector with a dimension of 1∗ 0 which dC2 , ,. . . ,dCk ,and dCK to denote the computing power demand
is represented as Eq.(11), and has the following constraints of value of the 1st , 2nd , . . . , k th and K th subtask, the sequence
Eq.(12)-(14): h of the long-running interruptible workload Dmat C can be con-
UB = u1B , u2B , u3B . . . , utB , . . . , u0−1
i
0 structed as Eq. (17), which is a constant vector.
B , uB (11)
h i
0 Dmat = d 1
, d 2
, . . . , d k
, . . . , d K
(17)
X C C C C C
utB = 1 (12)
t=1 The mitigation pattern for the long-running interruptible
UBindex = [1, 2, 3, . . . , 0] ∗ UB
T
(13) workload is scheduling each subtask to a new time point
within a specified time frame, while maintaining the sequen-
te ≤ UBindex ≤ tl (14)
tial order unchanged among the subtasks. Once the exe-
In Eq. (11), when the element utB
is 1, it signifies that the cution time of each subtask is determined, the computing
start node of the workload is situated within the t time point. power demand for the workload after shift can be uniquely
Conversely, if utB is 0, it implies that the start node of the ascertained. Thus, the key parameter for the time-shift mod-
workload is not positioned within that time period. Eq. (12) elling of long-running interruptible workload is the time-shift
means that only one element in the vector can be the value matrix variable of its subtasks.

VOLUME 12, 2024 102065

 L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

Use a binary variable uk,t th

C to indicate whether the k sub- The servers in the data center studied in this article utilize
task is moved to time point t. The time shift matrix variable CPU chips. The computing power capacity of the data center
UC with a dimension of 0 ∗ K is formed, which is represented Dmax
dc is expresses as:
as Eq. (18). UC has the constraints of Eq.(19)-(22).
Dmax max
dc = Nrack ∗ Nserver ∗NCPU ∗ Dcpu (25)
 1,1
u1,2 . . . u1,k . . . u1,K

uC C C C where Nrack is the number of racks in the data center, Nserver
 2,1
u2,2 . . . u2,k . . . u2,K

 uC C C C
 is the number of servers per rack, NCPU is the number of CPU
chips per server, Dmax
 
 u3,1 uC3,2
. . . uC 3,k 3,K 
. . . uC  cpu is the maximum computing power of
 C
 ... ... ... ... ... ...  the CPU chip studied in this paper.
UC =   (18)
 t,1
 uC uCt,2
. . . uC t,k
. . . ut,K
 Based on the assumption that: (1) all servers in the data
C 
 ...

... ... ... ... ... 
 center are homogeneous; (2) all servers in the computer room
 0−1,1 0−1,2 are turned on. The power consumption of servers can be
. . . u0−1,k . . . u0−1,K

 uC uC C C

modelled as:
u0,1
C u0,2
C . . . u 0,k
C . . . u 0,K
C

0
X Pdata = Pidle + Ppeak − Pidle ∗ ucpu (26)
ut,k
C = 1 (k = 1, 2, . . . , K ) (19)
where ucpu is the processor utilization rate. Pidle represents
t=1
h i the power of servers when ucpu is zero, Ppeak represents the
UCindex = uindex,1
C , uindex,2
C , . . . , uindex,k
C , . . . , uindex,K
C power of servers when ucpu reaches 100%.
(20) The formula for calculating CPU utilization is:
UCindex = [1, 2, 3, . . . , 0] ∗ UC (21) ucpu =
Ddata
(27)
Dmax
uindex,k+1
C − uCindex,k
≥ 1 (k = 1, 2, . . . , K ) (22) dc
where Ddata is the computing power demand of workloads.
In Eq. (19), for each column in matrix UC , the sum of the This model provides a way to estimate the electricity power
element values in all rows is 1, implying that a subtask can consumption of workloads based on their computing power
only transfer from one time point to another, not to multiple demand. Through this model, the time shift models of work-
time points. In Eq. (20), UCindex represents the time point of loads could be transformed into power regulation models.
each subtask after the time shift, which is a variable with a
dimension of 1 ∗ K, which is calculated through Eq. (21). III. DAY-AHEAD OPTIMIZATION SCHEDULING MODEL
Eq.(22) enforces that the time position at which the k+1th FOR DATA CENTER MICRO-GRID
subtask moves to should be greater than the time position at In this section, the day-ahead optimization scheduling model
which the kth subtask moves to. for DCM incorporating the workloads shift is established.
The time series vector variable of computing power
demand after the movement of each subtask is denoted as A. TIME STEP OF OPTIMIZATION SCHEDULING, DATA
shift
DC with a dimension of 1∗ 0, which is obtained by: MEASUREMENT AND WORKLOADS MOVEMENT
The time step of day-ahead scheduling 1t is set as 15 min-
shift T T
DC = [Uc ∗ (Dmat
c ) ] (23) utes, which is consistent with sampling interval of power
supply and demand. On the one hand, the fluctuations trend
E. CONVERSION RELATION BETWEEN COMPUTING of wind and PV power throughout a day typically exhibit
POWER AND ELECTRICITY POWER periodic changes. The 15-min interval allows for capturing
Computing power refers to the calculating ability of a device the primary fluctuation characteristics of wind and PV power,
to generate particular outputs through data processing. The while mitigating the computational burden caused by overly
computing power is implemented through various computing frequent data points. On the other hand, the workloads power
chips such as CPU, GPU, FPGA, ASIC, which are carried by consumption is relatively steady throughout the day. Using
computers, servers, and other systems. The standard measure- 15 minutes as the interval for the measuring of power con-
ment for computing power is the number of FLoating-point sumption and as the minimum unit for its movement, offers
Operations executed Per Second (FLOPS). sufficient accuracy to illustrate the load shift mechanism
The computing power of a chip is determined by three while simplifying data processing.
key factors, which are the number of computing cores of
the chip Ncore , the core frequency fcore , and the double- B. OPTIMIZATION OBJECTIVE
precision floating-point operands per clock cycle Zcore of the The day-ahead scheduling operates on a 24-hour optimization
core. The computing power Dmax chip of the chip follows the scale with 15mins as the time step. Based on the day-ahead
formula: forecasts of renewable power and computing power load,
with the goal of minimizing the daily operating cost, the
Dmax
chip = Ncore ∗ fcore ∗ Zcore (24) scheduling plan for the next 24 hours will be formulated.

102066 VOLUME 12, 2024

L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

The day-ahead optimization scheduling is designed to min- demand are denoted as Dori
MG . The specific values of original
imize operational costs within the dispatching day, expressed computing power demand of various workloads are provided
as: in Fig. 2.
OBJ = minCexch + Ccur + Cdata (28) 2) REFRIGERATION CONSTRAINT
The optimization function comprises the net cost of elec- The relationship of the electricity power consumption of
tricity transactions with the main-grid Cexch ( ), penalties for refrigeration system Ptrefri and refrigeration power Qtrefri is:
wind and PV power curtailment Ccur , which are expressed Qtrefri
as Eq. (29) and (30). The penalty costs for the computing Ptrefri = (33)
ηrefri
workloads scheduling Cdata ( ) is given in Eq. (8).
X0 where ηrefri is the comprehensive refrigeration performance
Cexch = Kbuy Ptbuy 1t − Ksell Ptsell 1t (29) coefficient
t=1
X0 Ptrefri should meet the upper and lower limits constraints:
Ccur = Kcur Ptcur 1t (30) Pmin t max
(34)
t=1 refri ≤ Prefri ≤ Prefri
where Kbuy /Ksell is the unit electricity purchase/sale price In this research, Pmin max
refri is 0, Preri is 143kW. Detailed analy-
from/to the main grid, which is 0.8 /kWh and 0.4 /kWh, sis can be found in Section IV.
respectively; Ptbuy /Ptsell is the purchased/sold electricity The expression of Ptrefri is taken from [21] and [22], written
power during time period t; Ptcur is the curtailment power of as:
wind and PV in time period t; Kcur is the unit penalty price
Qtrefri = Ptheat
for renewable power curtailment. To promote the utilization  
θint+1 − θint ∗ Cin ∗ (R1 + R2) + θint − θout t


of renewable power, we set a higher penalty coefficient,
which is 2 /kWh for the renewable power curtailment in − (35)
(R1 + R2)
the objective function.
By considering the system operation constraints and com- where Ptheat is the heat generation of the servers at time t,
bining the day-ahead forecasts of wind and PV power gen- where equals to Ptdata [23]; Cin is the equivalent heat capacity
eration and electricity consumption of computing work- of indoor air; R1 is the equivalent thermal resistance of indoor
loads, this model optimizes the time shift plan of various air and the inner side of the wall; R2 is the equivalent thermal
delay-tolerant workloads, refrigeration output plan, power resistance of the outer wall and outdoor air; θin is the indoor
exchange plan, wind and PV power utilization plan under the temperature which is set as a constant value of 25◦ ; θout is the
goal of minimizing the operating costs of DCM during the outdoor temperature, which are constant values and provided
scheduling day. in Appendix.

C. RELATIVE CONSTRAINTS 3) UPPER AND LOWER LIMIT OF RENEWABLE POWER

UTILIZATION
Constraints regarding the delay-tolerant workloads are
detailed in Section II. The other system constraints are listed The utilized wind and PV power at time point t Ptwind and Ptpv
as follows. have the constraints as below.
0 ≤ Ptwind ≤ Ptwind,max (36)
1) COMPUTING POWER DEMAND BALANCE CONSTRAINT 0 ≤ Ptpv ≤ Ptpv,max (37)
This study considers short-running workloads (A), two
where Ptwind,max
and Ptpv,max
is the forecasts of maximum
long-running continuous workloads (B1 and B2), three
power generation of wind and solar power at time t. The
long-running interruptible workloads (C1, C2, C3), and
values of Pwind,max and Ppv,max are provided in Section IV.
delay-sensitive workloads (MG). The total computing power
demand of all the workloads at the original state Dori data and
shift 4) POWER EXCHANGE CONSTRAINT
after shift Ddata should follow the below constraints.
The day-ahead planned purchased power and sold power Ptbuy
Dori ori ori ori ori ori ori ori
data = DA + DB1 + DB2 + DC1 + DC2 + DC3 + DMG
and Ptsell , as well as the state of purchase power and sold
(31) power Btbuy and Btsell have the following constraint.
shift
Ddata = DA
shift shift shift shift
+ DB1 + DB2 + DC1 + DC2 + DC3
shift shift Pmin t t max t
buy ∗ Bbuy ≤ Pbuy ≤ Pbuy ∗ Bbuy (38)
+ Dori
MG (32) Pmin t
sell ∗ Bsell ≤Ptsell ≤ Pmax
sell ∗ Btsell (39)
t t
Bbuy + Bsell =1 (40)
where Dori ori
A , . . . , DC3 are the computing power demand of
delay-tolerant workloads A, . . . , C3 at the original state. where Pmin min max
buy and Psell are both 0kW, Pbuy is 360kW. This study
shift shift
DA , . . . , DC3 are the computing power demand of max
set the power sale limit Psell be 0kW, 50 kW, 100kW, 200kW
delay-tolerant workloads A, . . . , C3 after shift. Since the and 400kW as the boundary conditions to test the sensitivity
delay-sensitive workloads don’t shift, their computing power of the model under different scenarios.

VOLUME 12, 2024 102067

 L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

5) POWER SUPPLY AND DEMAND BALANCE CONSTRAINT

The total power at supply side of data center equals to the
power on demand side:
Ptwind + Ptpv + Ptbuy = Ptdata + Ptrefri + Ptsell (41)

IV. CASE ANALYSIS

A. BOUNDARY CONDITONS DATA
1) PARAMETERS OF CPU, SERVER, RACK AND DATA CENTER
This study selected a medium-sized data center room in
northern China as the example [24], [25]. The data center
room occupies an area of 200m2 , and is equipped with 60
42U racks, each housing 10 4-way 4U servers. Detailed
parameters for the CPU chip, server and rack are outlined in
Table 1.

TABLE 1. Parameters for CPU, server, and rack, and computer room.
FIGURE 2. The original computing power load time series curves of the
delay-sensitive and delay-tolerant workloads.

long-running interruptible workloads (C1, C2 and C3) are

given in Table 2.

TABLE 2. Time constraints parameters for each delay-tolerant workloads.

Based on Table 1, in this paper, Dmax

cpu is 1600GFLOPS, 3) CAPACITY PARAMETER OF REFRIGERATION AND
max
Ddc is 3.84∗ 10^6 GFLOPS, Ppeak is 3.6∗ 10^5 W. The Pidle is RENEWABLE POWER SYSTEM
2.16∗ 10^5 W, which is calculated by multiplying Ppeak with The heat load of data center mainly stems from the heat gen-
a coefficient of 0.6 [23]. erated by servers and environmental maintenance structure.
The rated refrigeration capacity of refrigeration system Qrated
refri
2) PARAMETERS OF DELAY-SENSITIVE AND is determined based on the server equipment power and the
DELAY-TOLERANT WORKLOADS area of data center room, as described by the formula:
The study considers short-running workloads (A), two
refri = Pheat + βSa
Qrate max
(42)
long-running continuous loads (B1 and B2), three long-
running interruptible workloads (C1, C2 and C3). This study where Pmax represents the maximum heat generation of the
heat
set the ratio of delay-tolerant workloads to total workloads server cluster, which is equal to Ppeak . Sa denotes the area of
to be 0.3, 0.5 and 0.7 as the boundary conditions to test the computer room, set as 200m2 for this study. The empirical
the sensitivity of the model under different scenarios. For coefficient β is assigned with a value of 0.7 [27]. Hence, the
illustration purpose, the original computing power demand rated refrigeration capacity Qrated
refri is determined to be 500kW
time series curves for the delay-sensitive interactive work- for the computer room. With a comprehensive refrigeration
load and delay-tolerant batch processing workloads when performance coefficient ηrefri of 3.5, the corresponding rated
the ratio of delay-tolerant workloads is 0.5 are depicted in electricity consumption of the refrigeration system Prated
refri is
Fig. 2. calculated to be 143kW.
The maximum allowed deferrable time periods H for The installed capacity of wind and PV power system,
short-running workloads (A), the earliest start time period denoted as Pmax
vre , is determined by:
te , the latest start time period tl for long-running contin-
vre ∗ ηvre = Prefri + Pdata
Pmax rated max
uous workloads (B1 and B2), and shifting time range for (43)

102068 VOLUME 12, 2024

L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

In this formula, the average power generation utilization where the workloads don’t shift over time. In the simple
coefficient of renewable energy ηvre is assumed to be 0.5, and scheduling scenario, the power balance results of utilized
correspondingly Pmaxvre is calculated to be 1000kW. wind power Pwind# , utilized PV power PPV # , purchased
This study focuses on developing a scheduling method power Pbuy# , server power consumption Pdata# , refrigera-
considering the time-shiftable delay-tolerant workloads to tion electricity consumption Prefri# and sold power Psell#
facilitate the renewable power utilization, while the economic are illustrated in Fig. 4. As for the calculation method, for
analysis of allocating renewable energy is not the main con- each time period, Pdata# is calculated through Eq. (26), (27)
cern. This study set the wind and PV power capacity be and (31), and Prefri# is calculated through Eq. (33) and (35).
300 kW and 700 kW; 500kW and 500 kW; 700kW and And by combining the 24h-ahead forecasts of wind and PV
300kW, respectively, as the boundary conditions to test the power generation, i.e. Pwind,max and PPV ,max , and taking into
sensitivity of the model under different scenarios. account the system operation constraints of purchased/sold
power limits, power balance, etc., the daily operation plan
in this scenario is obtained. Fig. 4 indicates that in this
scenario, electricity sales mainly occur during the 31st -
61st time periods, while electricity purchases predominantly
occur during the 1st - 30th and 62nd - 96th time periods.
Considering the purchase and sale electricity prices, as well
as the penalties for wind and PV power curtailment, the daily
operating cost in this simple scheduling scenario amounts
to 4482.0. The planned utilized renewable wind and solar
power is 6802.9 MWh, while the planned curtailment of wind
and solar power is 1060.6 kWh.
In the day-ahead optimization scheduling scenario, where
the time-shiftable workloads are involved, the power bal-
ance results of utilized wind power Pwind , utilized PV power
PPV , purchased power Pbuy , server power consumption Pdata ,
refrigeration electricity consumption Prefri , and sold power
FIGURE 3. Normalized values of 24h-ahead forecasts of wind and PV
Psell for the scheduling day are presented in Fig. 5. Compar-
power generation. isons of renewable power curtailment and power exchange
between the day-ahead simple scheduling scenario and the
Wind and PV power generation data used in the study are optimization scheduling scenario are depicted in Fig. 6
sourced from a reliable database [28], as depicted in Fig. 3. and 7. The computing power demand time series of the
To align with the configured capacities of wind and PV power delay-tolerant workloads at the original state and after the
systems, the data is scaled to generate 24h-ahead forecasts of temporal shift are shown in Fig. 8 and 9, respectively. Upon
wind and PV power generation used in this study. observing Fig. 8 and 9, it is evident that after optimiza-
tion, long-running continuous workload B1 and B2, long-
B. RESULTS ANALYSIS running interruptible workload C1-C3 all moves to the 33rd -
The optimization algorithm is implemented using Gurobi 59th time periods, and short-running workloads A partially
solver on MATLAB R2024a. This section reports the oper- shift to these time periods. Consequently, the wind and PV
ation costs and renewable power curtailment results of DCM power curtailment during the 33rd - 59th periods is reduced,
after day-ahead scheduling optimization. The time sequences while the electricity purchases during the 1st - 30th periods
of workloads at original status and after shifting is also ana- and the 62nd - 96th periods decreases. The optimized oper-
lyzed and compared. In this part, results under a specific sce- ating cost plan is 3329.3 for the scheduling day, while
nario is first presented. Further, additional results are obtained the planned wind and PV power utilization and curtail-
under different scenarios by changing the renewable power ment is 7215.1 MWh and 648.3 kWh. Compared to simple
capacities, ratio of delay-tolerant workloads and interconnec- scheduling scenario, the planned operation cost decreases by
tion capacity to validate the effectiveness of the proposed 25.7%, and the renewable power curtailment is also down
model. by 412.3kWh (5.2% of the total daily renewable power gen-
eration). These comparison results prove that considering
1) SCHEDULING RESULTS AFTER OPTIMZITION SCHDULING workload time-shift for the day-ahead optimization schedul-
Results under a specific scenario, where the wind and ing is advantageous in increasing the economic benefits of
PV power capacities are 500kW & 500kW, the ratio of DCM operation and promoting the consumption of renewable
delay-tolerant workloads is 0.5, and the power sale limit is power.
100kW are reported and analyzed in detail. It’s worth noting that, the increased utilization of renewable
To quantitatively verify the effect of considering workloads energy in data centers holds important implications as it’s
time-shift, this paper set up a simple scheduling scenario beneficial to the reduction of carbon emissions, which is also

VOLUME 12, 2024 102069

 L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

FIGURE 4. The power balance results in the day-ahead simple scheduling FIGURE 6. Comparison of renewable power curtailment in the day-ahead
scenario. simple scheduling scenario and the optimization scheduling scenario.

FIGURE 7. Comparison of power exchange in the day-ahead simple

FIGURE 5. The power balance results in the day-ahead optimization scheduling scenario and the optimization scheduling scenario.
scheduling scenario.

now and here we don’t temporarily discuss. Later, we plan

of interest to data center. By considering that 1kWh renewable to incorporate the carbon emission trading into the planning
power could reduce 0.872kg CO2 [29], renewable energy process of data center micro-grid with consideration of work-
could provide 6802.9 MWh of electricity for data center loads shift.
when not considering the workloads shift in the scheduling,
which corresponds to CO2 reduction of 5932.1t. Moreover, 2) SHIFT RESULTS OF EACH WORKLOAD AFTER
when workloads shift is taken into account in the scheduling OPTIMZITION SCHEDULING
optimization, the renewable power utilization can increase to The temporal shift details for the various delay-tolerant work-
7215.1 MWh a day, leading to a significant carbon emission loads are visualized in Fig. 10-15. Specifically, Fig. 10 depicts
reduction of 6291.6t. Therefore, the integration of renewable the computing power demand of short-running workloads (A)
energy into data center has the potential to yield significant before and after shifting. The results indicate a redistribu-
reduction in carbon emissions, while the incorporation of tion of computing power demand as short-running workloads
workloads shift into the scheduling could further enhance this transfer from initial time points to new time points, revealing
effect. In the future, as carbon trading mechanisms evolve, a decrease in demand during the initial time periods and
the reduction in CO2 emissions will not only contribute to an increase at the new time periods. The total computing
environmental benefits but also potentially generate cost sav- power demand stays unchanged before and after the shift-
ings and revenue. However, the mechanisms are not mature ing process. Fig. 11 and 12 display the computing power

102070 VOLUME 12, 2024

L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

FIGURE 8. The computing power demand time series of delay-tolerant FIGURE 10. The computing power demand curve of short-running
workloads at the original state. workloads A at original state and after moving.

FIGURE 9. The computing power demand time series of delay-tolerant FIGURE 11. The computing power demand curve of long-running
workloads after temporal shift. continuous workload B1 at original state and after moving.

demand of long-running continuous workloads (B1 and B2)

before and after shifting, where the shape of the computing workloads after the day-ahead optimization scheduling with-
power load curve keeps unchanged. Additionally, Fig. 13-15 out the penalty coefficient is shown in Fig. 16.
illustrate the computing power demand of long-running inter- (1) In the scenario with the penalty coefficient setting, most
ruptible workloads (C1, C2, and C3), demonstrating that the of the short-running workloads (A) are reallocated to the
sequential order of subtasks at different time points and the 32nd - 56th time periods, where the surplus wind and solar
total computing power demand remain consistent before and power is utilized. The reason of this behavior is that, the
after shifting. These findings provide evidence supporting cost of reducing the wind and PV power sales volume
the feasibility of workloads’ time shifting, which proves the (0.4+0.05=0.45 /kWh) is lower than the cost of purchasing
efficiency of the workloads scheduling strategy. electricity from other periods (1st – 30th and 62nd - 96th peri-
Among the three kinds of time-shiftable workloads, the ods) (0.8 /kWh). For another, this behavior may also lead
short-running workloads have a special setting where there to electricity revenue (2-0.05=1.95 /kWh) due to reducing
is a penalty coefficient in their time shift model. This paper penalties for wind and PV power curtailment. Hence, oper-
discusses the rationale behind this setting by comparing ators are willing to pay lower costs or gain profits by using
the scenarios with and without the penalty coefficient. The wind and PV power, rather than purchasing electricity from
time shift results of the various time-shiftable delay-tolerant the main grid.

VOLUME 12, 2024 102071

 L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

FIGURE 12. The computing power demand curve of long-running FIGURE 14. The computing power demand curve of long-running
continuous workload B2 at original state and after moving. interruptible workload C2 at original state and after moving.

FIGURE 13. The computing power demand curve of long-running FIGURE 15. The computing power demand curve of long-running
interruptible workload C1 at original state and after moving. interruptible workload C3 at original state and after moving.

(2) In the scenario without the penalty coefficient setting renewable power curtailment diminution of workloads’ tem-
for short-running workloads shift, there is still an inflow poral shift under different scenarios. By setting the capac-
and aggregation of short-running workloads in the 34th -55th ities of wind and PV power capacity be 300kW&700
time periods. But there are also some workloads transferring kW; 500kW&500 kW; 700kW&300kW, setting the ratio of
from the 62nd - 96th period to the 1st -30th time period. delay-tolerant workloads be 0.3, 0.5 and 0.7, and setting the
The mutual transfer of short-running workloads between interconnection capacity be 400kW, 200kW, 100kW, 50kW,
the 1st -30th and the 62nd -96th time periods is an invalid 0, a total of 45 scenarios are created. The various results of
time shift, since the unit price of power purchase in these renewable power curtailment and operation cost of day-ahead
two time frames is the same. Thus, the comparison indi- optimization scheduling and simple scheduling, as well as the
cates that setting the penalty coefficient for the time-shift benefits of workloads’ shift under different renewable power
model of short-running workloads can avoid invalid capacities, delay-tolerant workloads ratios, and interconnec-
movement. tion capacities, are reported in Table 3.
From Table 3, it’s observed that:
3) BENEFITS UNDER DIFFERENT SCENARIOS • When the renewable energy capacity and interconnec-
To validate the robustness of the proposed model, the study tion capacity remain constant, an increased proportion
also examines the effect on operation cost reduction and of delay-tolerant workloads results in more significant

102072 VOLUME 12, 2024

L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

TABLE 3. Results of operation cost and renewable energy curtailment and benefits of workloads shift under different scenarios.

benefits in reducing operational costs and curtailing • For every renewable power capacity, despite the change
renewable power by workloads shift as compared to of delay-tolerant workloads ratio and interconnection
simple scheduling. capacity, the workloads shift have reduction effect on
• When the renewable power capacity and the ratio of renewable power curtailment and operation cost.
delay-tolerant workloads are held constant, a lower • When the wind and PV power capacity is 300kW&
interconnection capacity isn’t positive or negative rela- 700kW; 500kW&500kW; 700kW&300kW, the reduc-
tive to the enhanced reduction effect on renewable power tion effect on renewable power curtailment and oper-
curtailment and operation costs. The greatest reduction ation costs can reach up to 500.6kWh and 22.3%;
effect is achieved at a certain intermediate value of 585.4 kWh and 30.9%; and 565.1kWh and 45.4%, when
interconnection capacity. the delay-tolerant workloads ratio and interconnection

VOLUME 12, 2024 102073

 L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

economy of data center micro-grid and increasing the utiliza-

tion rate of renewable energy. The proposed time shift theory
and power regulation models can be applied into further
planning and operation of other form of data center energy
systems, such as data center integrated energy system, which
is expected to increase the system benefits and adoption
of renewable energy, ultimately accelerating the low-carbon
transition of the data center industry.

APPENDIX
The data of a 24h time series of outdoor temperature is shown
in Fig. 17.

FIGURE 16. The computing power demand of delay-tolerant workloads

after shifting in the scenario without the penalty coefficient setting for
short-running workloads shift.

capacity be 0.7, 0kW and 0.7, 200kW; 0.7, 0 kW and

0.7, 200kW; 0.7, 50kW and 0.7, 200 kW. These results
are marked using gray color in Table 3.
In summary, for each renewable energy capacity, work-
loads transfer through the timeline can achieve varying
degrees of reduction on renewable power curtailment and
operation costs under different delay-tolerant workloads pro-
portion and interconnection capacity settings. The control
strategy and benefits of workloads shift remain valid when
the capacity ratios of wind and PV, the ratio of delay-tolerant
workloads, and interconnection capacity change. Therefore, FIGURE 17. Outdoor temperature during the scheduling day.
managing the time shiftable workloads in power scheduling
is conductive to improving the economic efficiency of DCM REFERENCES
operations and promoting the consumption of renewable [1] E. Masanet, A. Shehabi, N. Lei, S. Smith, and J. Koomey, ‘‘Recalibrating
energy. global data center energy-use estimates,’’ Science, vol. 367, no. 6481,
pp. 984–986, Feb. 2020.
[2] Electricity, Int. Energy Agency, Paris, France, 2024.
V. CONCLUSION [3] Y. Cao and S. Zhang, ‘‘Facilitating the provision of load flexibility to the
The power consumption and carbon emission problems of power system by data centers: A hybrid research method applied to China,’’
Utilities Policy, vol. 84, Oct. 2023, Art. no. 101636.
data center has attracted worldwide attention with the rapid
[4] Z. Cao, X. Zhou, H. Hu, Z. Wang, and Y. Wen, ‘‘Toward a systematic survey
advance of AI technologies and digital economy, and the inte- for carbon neutral data centers,’’ IEEE Commun. Surveys Tuts., vol. 24,
gration of data centers with low-carbon renewable wind and no. 2, pp. 895–936, 2nd Quart., 2022.
solar energy is becoming a promising strategy. Aiming at the [5] G. Rostirolla, L. Grange, T. Minh-Thuyen, P. Stolf, J. M. Pierson,
G. Da Costa, G. Baudic, M. Haddad, A. Kassab, J. M. Nicod, L. Philippe,
volatilities of renewable power, the delay-tolerant workloads V. Rehn-Sonigo, R. Roche, B. Celik, S. Caux, and J. Lecuivre, ‘‘A survey of
within data centers are considered as one of most potential challenges and solutions for the integration of renewable energy in datacen-
sources for power regulation. In light of the deficiencies of ters,’’ Renew. Sustain. Energy Rev., vol. 155, Mar. 2022, Art. no. 111787.
[6] U. Hussan, M. Hassan, M. A. Ayub, J. Peng, H. Rasheed, H. Jiang,
systematic analysis to delay-tolerant workloads in existing and F. Asghar, ‘‘Smooth and uninterrupted operation of standalone DC
studies, the paper clarifies the temporal shift mechanisms microgrid under high and low penetration of RESs,’’ IEEE Access, vol. 12,
of three typical delay-tolerant workloads, i.e. short-running pp. 48620–48629, 2024.
[7] National Development and Reformation Commission. Accessed: May 12,
deferrable, long-running continuous, long-running interrupt- 2024. [Online]. Available: https://www.ndrc.gov.cn/xwdt/ztzl/dsxs/
ible workloads, and builds the temporal shift models for them [8] H. Zhu, D. Zhang, H. H. Goh, S. Wang, T. Ahmad, D. Mao, T. Liu, H. Zhao,
to participate in the day-ahead power scheduling of data and T. Wu, ‘‘Future data center energy-conservation and emission-
reduction technologies in the context of smart and low-carbon city con-
center micro-grid. struction,’’ Sustain. Cities Soc., vol. 89, Feb. 2023, Art. no. 104322.
By applying the developed models into a data center case [9] Y. Wang, Y. Han, J. Shen, K. Han, J. Li, W. Ji, H. Li, R. Jing, H. Hua,
across various scenarios, results confirm the effectiveness of D. Wang, J. Han, S. Shao, J. Lin, J. Zhao, Q. Zhang, Y. Zhang, H. Zhang,
Q. Yang, and J. Yan, ‘‘Data center integrated energy system for sus-
the developed energy management scheme and the efficiency tainability: Generalization, approaches, methods, techniques, and future
of the time-shiftable workloads in improving the operation perspectives,’’ Innov. Energy, vol. 1, no. 1, 2024, Art. no. 100014.

102074 VOLUME 12, 2024

L. Liu et al.: Optimal Energy Management of DCM Considering Computing Workloads Shift

[10] W. Zhang and V. M. Zavala, ‘‘Remunerating space–time, load-shifting LUYAO LIU received the B.E. and Ph.D.
flexibility from data centers in electricity markets,’’ Appl. Energy, vol. 326, degrees from Shandong University, in 2016 and
Nov. 2022, Art. no. 119930. 2022, respectively. She is currently working as
[11] Z. Liu, Y. Chen, C. Bash, A. Wierman, D. Gmach, Z. Wang, M. Marwah, a Postdoctoral with China Energy Engineer-
and C. Hyser, ‘‘Renewable and cooling aware workload manage- ing Group Guangdong Electric Power Design
ment for sustainable data centers,’’ in Proc. 12th ACM SIGMET- Institute Company Ltd., and as a Research
RICS/PERFORMANCE Joint Int. Conf. Meas. Model. Comput. Syst., Associate with Shenzhen International Gradu-
London, U.K., Jun. 2012, pp. 175–186.
ate School, Tsinghua University. Her research
[12] M. Chen, C. Gao, M. Shahidehpour, Z. Li, S. Chen, and D. Li, ‘‘Internet
interests include load and renewable power fore-
data center load modeling for demand response considering the coupling
of multiple regulation methods,’’ IEEE Trans. Smart Grid, vol. 12, no. 3,
cast and integrated energy system coordinated
pp. 2060–2076, May 2021. optimization.
[13] T. Yang, H. Jiang, Y. Hou, and Y. Geng, ‘‘Carbon management of multi-
datacenter based on spatio-temporal task migration,’’ IEEE Trans. Cloud
Comput., vol. 11, no. 1, pp. 1078–1090, Jan. 2023.
[14] L. Cupelli, T. Schütz, P. Jahangiri, M. Fuchs, A. Monti, and D. Müller,
‘‘Data center control strategy for participation in demand response pro-
grams,’’ IEEE Trans. Ind. Informat., vol. 14, no. 11, pp. 5087–5099,
Nov. 2018.
[15] J. Wang, H. Deng, Y. Liu, Z. Guo, and Y. Wang, ‘‘Coordinated optimal XINWEI SHEN (Senior Member, IEEE) received
scheduling of integrated energy system for data center based on computing the Ph.D. degree from Tsinghua University,
load shifting,’’ Energy, vol. 267, Mar. 2023, Art. no. 126585. in 2016. He is currently an Assistant Professor and
[16] Y. Cao, M. Cheng, S. Zhang, H. Mao, P. Wang, C. Li, Y. Feng, and Z. Ding, a Principle Investigator with Shenzhen Interna-
‘‘Data-driven flexibility assessment for internet data center towards peri- tional Graduate School, Tsinghua University. His
odic batch workloads,’’ Appl. Energy, vol. 324, Oct. 2022, Art. no. 119665. research interests include power systems, energy
[17] H. He, H. Shen, Q. Hao, and H. Tian, ‘‘Online delay-guaranteed workload internet, and integrated energy systems.
scheduling to minimize power cost in cloud data centers using renewable
energy,’’ J. Parallel Distrib. Comput., vol. 159, pp. 51–64, Jan. 2022.
[18] P. Wiesner, I. Behnke, D. Scheinert, K. Gontarska, and L. Thamsen, ‘‘Let’s
wait awhile: How temporal workload shifting can reduce carbon emissions
in the cloud,’’ 2021, arXiv:2110.13234.
[19] C. Lu, K. Ye, G. Xu, C.-Z. Xu, and T. Bai, ‘‘Imbalance in the cloud: An
analysis on Alibaba cluster trace,’’ in Proc. IEEE Int. Conf. Big Data,
Boston, MA, USA, Dec. 2017, pp. 2884–2892.
[20] C. Reiss and A. Tumanov, ‘‘Heterogeneity and dynamicity of clouds
at scale,’’ in Proc. ACM Symp. Cloud Comput., San Jose, CA, USA,
Nov. 2021, pp. 1–13. ZHIGANG CHEN is currently an Assistant Chief Engineer and a Senior
[21] J. Zhu, ‘‘Evaluation of adjustable load capacity of electric heating and Engineer with China Energy Engineering Group Guangdong Electric Power
research on cluster control strategy,’’ M.S. thesis, Dept. Electron. Eng., Design Institute Company Ltd. His research interests include electric power
Northeast Electric Power Univ., Changchun, China, 2019. systems and integrated energy systems.
[22] Chinese Society of Electrical Engineering. Topic a of the 15th CSEE
National College Students Electrical Mathematics Modelling Competition.
Accessed: Feb. 28, 2024. [Online]. Available: http://shumo.neepu.edu.cn
[23] C. Jin, X. Bai, C. Yang, W. Mao, and X. Xu, ‘‘A review of power consump-
tion models of servers in data centers,’’ Appl. Energy, vol. 265, May 2020,
Art. no. 114806.
[24] Q. Huang, ‘‘Understanding of several important parameters in ‘code for
design of electronic information system room’ GB50174–2008,’’ Comput. QIE SUN received the Ph.D. degree from the Royal Institute of Technology,
Room Technol. Manag., vol. 5, pp. 6–9, Oct. 2010. in 2011. He is currently a Professor with the Institute for Advanced Science
[25] J. Zhu, S. Yang, and L. Yu, ‘‘Modeling and temperature control of real-time and Technology, Shandong University. His research interests include sustain-
consumption of renewable energy in data center refrigeration systems,’’
able energy systems and advanced energy storage.
Autom. Electr. Power Syst., vol. 46, no. 20, pp. 13–22, Oct. 2022.
[26] Second-Generation Intelligent Intel® Xeon® Scalable Processor.
Accessed: Feb. 28, 2024. [Online]. Available: https://ark.intel.
com/content/www/cn/zh/ark/products/series/192283/2nd-generation-
intel-xeon-scalable-processors.html
[27] X. Ding, ‘‘Calculation analysis and optimization ideas for heat load and
air conditioning cooling capacity in professional data center computer
rooms,’’ UPS Appl., vol. 12, pp. 11–13, Dec. 2017.
[28] European Commission Science Hub. Accessed: Feb. 28, 2024. [Online]. RONALD WENNERSTEN is currently a Professor with the Institute
Available: https://joint-research-centre.ec.europa.eu/index_en for Advanced Science and Technology, Shandong University. His cur-
[29] Y. Wang, S. Zhou, Z. Yao, and X. Ou, ‘‘Life cycle modeling analysis of rent research interests include energy transition and energy technology
the interaction between carbon dioxide and air pollutant emissions of coal development.
power in China,’’ Electric Power, vol. 54, no. 8, pp. 128–135, 2021.

VOLUME 12, 2024 102075