This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 1

Burstiness-Aware Resource Reservation for

Server Consolidation in Computing Clouds
Sheng Zhang, Member, IEEE, Zhuzhong Qian, Member, IEEE, Zhaoyi Luo, Jie Wu, Fellow, IEEE, and
Sanglu Lu, Member, IEEE

Abstract—In computing clouds, burstiness of a virtual machine (VM) workload widely exists in real applications, where spikes usually
occur aperiodically with low frequency and short duration. This could be effectively handled through dynamically scaling up/down in a
virtualization-based computing cloud; however, to minimize energy consumption, VMs are often highly consolidated with the minimum
number of physical machines (PMs) used. In this case, to meet the dynamic runtime resource demands of VMs in a PM, some VMs have
to be migrated to some other PMs, which may cause potential performance degradation. In this paper, we investigate the burstiness-
aware server consolidation problem from the perspective of resource reservation, i.e., reserving a certain amount of extra resources on
each PM to avoid live migrations, and propose a novel server consolidation algorithm, QUEUE. We first model the resource requirement
pattern of each VM as a two-state Markov chain to capture burstiness, then we design a resource reservation strategy for each PM
based on the stationary distribution of a Markov chain. Finally, we present QUEUE, a complete server consolidation algorithm with
a reasonable time complexity. We also show how to cope with heterogenous spikes and provide remarks on several extensions.
Simulation and testbed results show that, QUEUE improves the consolidation ratio by up to 45% with large spike size and around
30% with normal spike size compared with the strategy that provisions for peak workload, and achieves a better balance between
performance and energy consumption in comparison with other commonly-used consolidation algorithms.

Index Terms—Bursty workload, Markov chain, resource reservation, server consolidation, stationary distribution

1 I NTRODUCTION We observed that the variability and burstiness of VM

workload widely exists in modern computing clouds,
C LOUD computing has been gaining more and more
traction in the past few years, and it is changing
the way we access and retrieve information [1]. The
as evidenced in prior studies [4], [6], [7], [8], [9]. Take
a typical web server for example, burstiness may be
caused by flash crowed with bursty incoming requests.
recent emergence of virtual desktop [2] has further el-
We all know that VMs should be provisioned with
evated the importance of computing clouds. As a cru-
resources commensurate with their workload require-
cial technique in modern computing clouds, virtualiza-
ments [10], which becomes more complex when con-
tion enables one physical machine (PM) to host many
sidering workload variation. As shown in Fig. 1, two
performance-isolated virtual machines (VMs). It greatly
kinds of resource provisioning strategies are commonly
benefits a computing cloud where VMs running various
used to deal with workload burstiness—provisioning for
applications are aggregated together to improve resource
peak workload and provisioning for normal workload.
utilization. It has been shown in previous work [3]
Provisioning for peak workload is favourable to VM
that, the cost of energy consumption, e.g., power supply,
performance guarantee, but it undermines the advantage
and cooling, occupies a significant fraction of the total
of elasticity from virtualization and may lead to low
operating costs in a cloud. Therefore, making optimal
resource utilization [1], [8], [9].
utilization of underlying resources to reduce the energy
In contrast, provisioning for normal workload makes
consumption is becoming an important issue [4], [5]. To
use of elasticity in cloud computing. In this case, to
cut back the energy consumption in clouds, server con-
meet the dynamic resource requirements of VMs, local
solidation is proposed to tightly pack VMs to reduce the
resizing and live migration are the two pervasively-used
number of running PMs; however, VMs’ performance
methods. Local resizing adaptively adjusts VM configu-
may be seriously affected if VMs are not appropriately
ration according to the real-time resource requirement
placed, especially in a highly consolidated cloud.
with negligible time and computing overheads [11]. On
the other hand, live migration moves some VM(s) to a
• S. Zhang, Z.Z. Qian, and S.L. Lu are with the State Key Laboratory for
Novel Software Technology, Nanjing University, Nanjing, 210023, China.
relatively idle PM, when local resizing is not able to
E-mail: {sheng, qzz, sanglu}@nju.edu.cn. allocate enough resources. However, in a highly con-
• Z.Y. Luo is with the David Cheriton School of Computer Science, Univer- solidated computing cloud where resource contention
sity of Waterloo, Waterloo, N2L3G1, Canada.
E-mail: [email protected]
is generally prominent among VMs, live migration may
• J. Wu is with the Department of Computer and Information Sciences, cause significant service downtime; furthermore, it also
Temple University, Philadelphia, PA 19122, USA. incurs noticeable CPU usage on the host PM [12], which
E-mail: [email protected].
probably degrades the co-located VMs’ performance.

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 2

adlo
kor (1) To the best of our knowledge, we are the first
W )R ) to quantify the amount of reserved resources with
(e
e
R(
p

)R ikp adol consideration of workload burstiness. We propose

(d
b
S kor
aol to use the two-state Markov chain model to capture
rko W
W ake workload burstiness, and we present a formal prob-
la P lem description and its NP-completeness.
mro
N Time (2) We develop a novel algorithm, QUEUE, for
burstiness-aware resource reservation, based on the
Fig. 1. An example of workload with bursty spikes. stationary distribution of a Markov chain. We also
show how to cope with heterogeneous spikes to
further improve the performance of QUEUE.
In this paper, we propose to reserve some extra (3) Extensive simulations and testbed experiments are
resources on each PM to accommodate bursty work- conducted to validate the effectiveness and advan-
load [13]. In doing so, when a resource spike occurs, tages of QUEUE.
VMs can be quickly reconfigured to the new level of We now continue by presenting related work in Sec-
resource requirement through local resizing with mini- tion 2 and our model in Section 3. Problem formulation is
mal overheads, instead of being migrated to some other provided in Section 4. We show the details of QUEUE in
PMs. Hence, the number of live migrations could be Section 5. Heterogeneous spikes are handled in Section 6.
reduced considerably and the overall performance of a Evaluation results are presented in Section 7. Before
computing cloud could be improved. concluding the paper in Section 9, we discuss known
Specifically, we investigate the problem of minimizing issues and extensions of QUEUE in Section 8.
the amount of extra resources reserved on each PM
during server consolidation while the overall perfor-
mance is probabilistically guaranteed. By “probabilisti- 2 R ELATED W ORK
cally guaranteed”, we mean that, the fraction of time Most of prior studies [3], [15], [16] on server consolida-
within which the aggregated workloads of a PM exceed tion focused on minimizing the number of active PMs
its physical capacity is not larger than a threshold. from the perspective of bin packing. A heterogeneity-
Imposing such a threshold rather than conducting live aware resource management system for dynamic capac-
migration upon PM’s capacity overflow is a way to ity provisioning in clouds was developed in [17]. Stable
tolerate minor fluctuations of resource usage (like the resource allocation in geographically-distributed clouds
case of CPU usage) and to break the tradeoff between was considered in [18]. Network-aware virtual machine
utilization and performance. Then, our problem can be placement was considered in [19]. Scalable virtual net-
formulated as an optimization, wherein the goal is to work models were designed in [8], [20] to allow cloud
minimize the amount of resource reserved on each PM, tenants to explicitly specify computing and networking
and the constraint is that the capacity violation ratio of requirements to achieve predictable performance.
every PM is not larger than a predetermined threshold. In a computing cloud, burstiness of workload widely
We use a two-state Markov chain to capture the bursti- exists in real applications, which becomes an inevitable
ness of workload [7], and also shows how to learn the characteristic in server consolidation [1], [4], [6], [7], [21].
chain parameters. Inspired by the serving windows in Some recent works [22], [23] used stochastic bin-packing
queueing theory [14], we abstract the resources reserved (SBP) techniques to deal with variable workloads, where
on each PM for workload spikes as blocks. Denoting workload is modeled as random variable. Some other re-
by θ(t) the number of busy blocks at time t on a PM, search [10], [24], [25] studied the SBP problem assuming
we show that a sequence of θ(0), θ(1), θ(2), ... has the VM workload follows normal distribution. Several other
Markov property, namely that, the next state only de- studies [26], [27] focused on workload prediction while
pends on the current state and not on the past sequence the application runs. Different from them, in our model a
of states. Then we develop a novel server consolidation lower limit of provisioning is set at the normal workload
algorithm, QUEUE, based on the stationary distribution level which effectively prevents VM interference caused
of this Markov chain. We also show how to further by unpredictable behaviors from co-located VMs.
improve the effectiveness of QUEUE with more careful Markov chain was used to inject burstiness into a
treatment of heterogenous workload spikes. Simulation traditional benchmark in [7]. Several works [5], [28], [29]
and testbed results show that, QUEUE improves the studied modeling and dynamic provisioning of bursty
consolidation ratio by up to 45% with large spike size workload in cloud computing. A previous study [30]
and around 30% with normal spike size compared with proposed to reserve a constant level of hardware re-
the strategy that provisions for peak workload, and source on each PM to tolerate workload fluctuation; but
achieves a better balance between performance and en- how much resource should be reserved was not given.
ergy consumption in comparison with other commonly- To the best of our knowledge, we are the first to quantify
used consolidation algorithms. The contributions of our the amount of reserved resources with consideration on
paper are three-fold. various, but distinct, workload burstiness.

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 3

pon adlo
kor
1-pon W )R )
OFF ON (e
e

(d
R
p

(Rb)
)R ikp aol
(Rp) (d
b
S kor
1-poff aol
rko Wk
W ae
poff la P
mro
N 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Time
Fig. 2. Two-state Markov chain. The “ON” state repre- Workload Profile ON-OFF Model

sents peak workload (Rp ) while the “OFF” state repre-

sents normal workload (Rb ). pon and pof f are the state Fig. 3. Given the predetermined Rb and Re , we conser-
switch probabilities. vatively round the solid black curve up to the dashed red
curve, based on which we can calculate pon and pof f .

3 M ODELING V IRTUAL M ACHINE W ORKLOAD

where the solid black curve represents the workload
3.1 Two-state Markov Chain over time. Given the predetermined Rb and Re , we
It has been well recognized in previous studies [4], [6], conservatively round the solid black curve up to the
[7] that VM workload is time-varying with bursty spikes, dashed red curve. Denote by W (t) the workload at time t
as shown in Fig. 1. Several works [9], [10], [22], [23], in the dashed red curve, e.g., W (4) = Rp , and W (7) = Rb .
[24], [25] modeled the workload of a VM as a random Denote by SF F the number of switches from state
variable, which follows the Bernoulli distribution in [9] “OFF” to state “OFF” in two consecutive time slots
or normal distribution in [10], [24], [25]. Different from during the time period of interest; denote by SF N the
these works, we model the workload of a VM as a two- number of switches from state “OFF” to state “ON” in
state Markov chain, which takes the additional dimen- two consecutive time slots during the time period of
sion of time into consideration, and thus describes the interest, e.g., SF F = 5 and SF N = 2 in Fig. 3. Similarly,
characteristics of spikes more precisely. we can define SN N and SN F , which are equal to 5 and
Fig. 2 shows an example. We denote the resource 2, respectively, in the figure. It is then easy to see that
requirements of peak workload, normal workload, and SF N SN F
workload spike by Rp , Rb , and Re , respectively, where pon = , and pof f = .
SF N + SF F SN F + SN N
Re = Rp − Rb as demonstrated in Fig. 1. The “ON”
state represents peak workload while the “OFF” state
represents normal workload. We use pon and pof f to 3.3 Potential Benefits
denote the state switch probabilities. More specifically, The 2-state Markov chain model allows cloud tenants to
if a VM is in the ON state, then the probability of it flexibly control the tradeoff between VM performance
switching to OFF at the next time is pof f , and remaining and deployment cost through adjusting Rb and Re .
ON is 1 − pof f . Similarly if a VM is in the OFF state, When a tenant wants to maximize VM performance, the
then the probability of it switching to ON at next time tenant should choose a large Rb and a small Re . As we will
is pon and remaining ON is 1 − pon . We emphasize that show later in this paper, there may be multiple work-
this model is able to describe the characteristics of spikes load spikes that share some common physical resources.
precisely—intuitively, Re denotes the size of a spike, and Thus, when the aggregated amount of workload spikes
pon denotes the frequency of spike occurrence. Thus, that simultaneously occur is larger than the amount of
each VM can be described by a four-tuple the shared common resources, capacity overflow hap-
pens and VM performance is probably affected.
Vi = (pion , piof f , Rbi , Rei ), ∀1 ≤ i ≤ n, (1) When a tenant wants to minimize deployment cost, the
where n is the number of VMs. tenant should choose a small Rb and a large Re . By ”deploy-
ment cost”, we mean the fee which is paid by a cloud
tenant to a cloud provider. Since physical resources
3.2 Learning Model Parameters are opportunistically shared among multiple workload
This subsection provides a simple strategy for cloud spikes, the charge for workload spike should be smaller
tenants to generate model parameters for their VM than that for normal workload [9]. Therefore, decreasing
workload. It consists of two phases. Rb helps tenants to reduce the deployment cost.
First, a cloud tenant must have the workload traces Our model is also a tradeoff between modeling com-
and guarantees that they will be consistent with the plexity and precision. We could model time-varying
realistic deployment in computing clouds. This could be workload by 3-state or even more states of Markov chain,
achieved by tentatively deploying VMs in a cloud for which should capture the workload bustiness more
a relatively short period; the cloud system collects the precisely; however, the complexity in learning model
resource usage traces and feeds them back to tenants. parameters and allocating physical resources increases
Second, given a VM workload trace, a cloud tenant as well, which may complicate the interactions between
generates a four-tuple. We use Fig. 3 as an illustration, cloud providers and tenants.

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 4

Symbol Meaning
n the number of VMs
performance of Hj . The performance of each PM should
Vi the i-th VM be probabilistically guaranteed, so we have
Rbi the normal workload size of Vi
Rei the spiky workload size of Vi Φj ≤ ρ, ∀1 ≤ j ≤ m. (3)
Rpi the peak workload size of Vi , and Rpi = Rbi + Rei
pion the probability of Vi switching from OFF to ON Here, ρ is a predetermined value serving as the threshold
piof f the probability of Vi switching from ON to OFF of COR. Main notations are summarized in Fig. 4 for
m the number of PMs quick reference. Our problem can be stated as follows.
Hj the j-th PM
Cj the physical capacity of Hj
Problem 1: (Burtiness-Aware Server Consolidation,
xij the variable indicating whether V Mi is placed on P Mj BASC) Given a set of n VMs and a set of m PMs, find a
Φj the capacity overflow ratio of PM P Mj VM-to-PM mapping X to minimize the number of PMs
ρ the threshold of capacity overflow ratio
used while making sure that (1) the initial placement sat-
Fig. 4. Main notations for quick reference. isfies capacity constraint, and (2) the capacity overflow
ratio of each PM is not larger than the threshold ρ. It can
be formally formulated as follows:
4 P ROBLEM F ORMULATION ∑
n

We consider a computing cloud with 1-dimensional re- min |{j| xij > 0, 1 ≤ j ≤ m}|
i=1
source; for scenarios with multi-dimensional resources, (4)
we provide a few remarks in Section 8. There are m s.t. COj0 = 0, ∀1 ≤ j ≤ m
physical machines in the computing cloud, and each PM Φj ≤ ρ, ∀1 ≤ j ≤ m
is described by its physical capacity Here, |S| denotes the cardinality of set S. In the
Hj = (Cj ), ∀1 ≤ j ≤ m. (2) following theorem, we can prove that, the BASC problem
is NP-complete.
We use a binary matrix X = [xij ]n×m to represent Theorem 1: The BASC problem is NP-complete.
the results of placing n VMs on m PMs: xij = 1, if Vi Proof: We prove this theorem by reduction from the
is placed on Hj , and 0 otherwise. We assume that the Bin Packing (BP) problem [31], which is NP-complete.
workloads of VMs are mutually independent. Let Wi (t) The decision version of the BP problem is as follows.
be the resource requirements of Vi at time t. According Given n items with sizes s1 , s2 , ..., sn ∈ (0, 1], can we
the Markov chain model, we have pack them in no more than k unit-sized bins?
 Given an instance of the decision version of the BP
Rbi if Vi is in the “OFF” state at time t, problem, we can construct an instance of the decision
Wi (t) = version of our problem as follows: let Rbi = Rpi = si ,
Rpi if Vi is in the “ON” state at time t.
∀1 ≤ i ≤ n; let m = k; let Cj = 1, ∀1 ≤ j ≤ m; and let
Then, the ∑
aggregated resource requirement of VMs on ρ = 0, i.e., capacity overflow is not allowed.
n
PM Hj is i=1 xij Wi (t). It is not hard to see that the construction can be
finished in polynomial time; thus, we reduce solving
Let COjt indicate whether the capacity overflow hap-
the NP-complete BP problem to solving a special case
pens on PM Hj at time t, i.e.,
of our problem, implying that our problem is NP-hard.

 ∑n It is easy to verify that the BASC problem is also in NP;
1 if xij Wi (t) > Cj ,
t
COj = i=1
the theorem follows immediately.

0 otherwise.
5 B URSTINESS -AWARE R ESOURCE R ESER -
Intuitively, the results of VM placement should guaran- VATION
tee that the capacity constraint is satisfied on each PM In this section, we first present the main idea of our
at the beginning of the time period of interest, i.e., solution to the BASC problem, then we design a resource
reservation strategy for a single PM, based on which
COj0 = 0, ∀1 ≤ j ≤ m.
we develop QUEUE, a complete server consolidation
We now can define our metric for probabilistic perfor- algorithm. In the end, we provide a concrete example
mance guarantee—capacity overflow ratio (COR), which is to help readers better understand our algorithm.
the fraction of time that the aggregated workloads of a
PM exceed its physical capacity. Denoting the capacity 5.1 Overview of QUEUE
overflow ratio of PM Hj as Φj , we have We propose reserving a certain amount of physical re-
∑ t sources on each PM to accommodate workload spikes.
1≤t≤T COj
Φj = , The main idea is to abstract the reserved spaces as
T blocks (i.e., serving windows in queueing theory). We
where T is the length of the time period of interest. give an informal illustration of the evolution process of
It is easy to see that, a smaller Φj implies a better our queueing system in Fig. 5.

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 5

Queueing Reducing
System Re Re Re R e Re Blocks Re Re Re OFF to ON (i.e., VMs that enter ( ) the queueingx!system)
Re Re Re Re Re
at time t, respectively. We use xy to denote y!(x−y)! , i.e.,
x choose y. Since the workloads of VMs are mutually
Rb Rb Rb Rb Rb Rb R b Rb Rb Rb Rb Rb Rb Rb Rb
independent, we have
( )
V1 V2 V3 V4 V5 V1 V2 V 3 V4 V5 V1 V 2 V3 V4 V5 θ(t) x
Physical Machine Physical Machine Physical Machine P r{O(t) = x} = pof f (1 − pof f )θ(t)−x ,
(a) (b) (c) x
and
Fig. 5. An illustration of the evolution process. (a) The ( )
original provisioning strategy for peak workload. (b) Gath- k − θ(t) x
P r{I(t) = x} = pon (1 − pon )k−θ(t)−x ,
ering all Re ′ s together to form a queueing system. (c) Re- x
ducing the number of blocks while still satisfying Equ. (3). which suggest that both O(t) and I(t) follow the bino-
mial distribution:
{
Initially, all VMs are provisioned by Rb + Re , and each O(t) ∼ B(θ(t), pof f ),
(5)
VM has its own block (denoted as Re in Fig. 5). A VM I(t) ∼ B(k − θ(t), pon ).
uses only its Rb part during periods of normal workload,
however, when a workload spike occurs, the extra Re Without loss of generality, we assume that the switch
part is put into use. We note that, the collected Re ′ s between two consecutive states of all VMs happens at
altogether form a queueing system—when a workload the end of each time interval. Then we have the recursive
spike occurs in a VM, the VM enters the queueing system relation of θ(t),
and occupies one of the idle blocks; when the spike θ(t + 1) = θ(t) − O(t) + I(t). (6)
disappears, the corresponding VM leaves the queue-
ing system and releases the block. It is worth noting Combining Equs. (5) and (6) together, we see that, the
that, there is no waiting space in the queueing system; next state θ(t + 1) only depends on the current state
thus, the PM capacity constraint would be violated if a θ(t) and not on the past sequence of states θ(t − 1),
workload spike occurs while all the blocks are occupied, θ(t − 2), ..., θ(0). Therefore, the stochastic process θ(0),
which never happens when the number of blocks equals θ(1), ... of discrete time ({0, 1, 2, ...}) and discrete space
the number of co-located VMs (as shown in Fig. 5(b)). ({0, 1, 2, ..., k}) is a Markov chain. The stochastic process
However, we may find that a certain number of blocks is said to be in state i (1 ≤ i ≤ k) if the number of busy
are idle for the majority of the time in Fig. 5(b), so we can blocks is i. Fig. 6 shows the transition graph of the chain.
reduce the number of blocks while only incurring very Let pij be the transition probability from state i to state
few capacity violations (as shown in Fig. 5(c)). Therefore, j. That is to say, if θ(t) = i, then the probability that
our goal becomes reserving minimal number of blocks θ(t + 1) = j is pij . For the sake( ) of convenience, when
on each PM while the performance constraint in Equ. (3) y > x or y ≤ x < 0, we let xy be 0. Then, pij can be
is still satisfied. derived as follows.
pij =P r{θ(t + 1) = j|θ(t) = i}
5.2 Resource Reservation Strategy for a Single PM ∑
i

In this subsection, We focus on resource reservation for = P r{O(t) = r, I(t) = j − i + r|θ(t) = i}

r=0
a single PM. For the sake of convenience, we set the size
of each block as the size of the maximum spike of all ∑i
= P r{O(t) = r|θ(t) = i}
co-located VMs on a PM. In Section 6, we will present
r=0 (7)
how to cope with heterogenous workload spikes in an
× P r{I(t) = j − i + r|θ(t) = i}
effort to further improve the performance of QUEUE.
∑i ( )
We also assume that all VMs have the same state switch i r
= pof f (1 − pof f )i−r
probabilities, i.e., pion = pon and piof f = pof f , for all 1 ≤ r
i ≤ n. In Section 8, we will show how to cluster VMs (
r=0
)
k−i
when they have different state switch probabilities. × pj−i+r (1 − pon )k−j−r .
Suppose there are k VMs on the PM of interest and j − i + r on
initially each VM Vi occupies Rbi resources. We initialize In the above formula, the first and second equations
the number of blocks reserved on this PM as k, and our are due to the definition of pij and the recursive relation
objective is to reduce the number of blocks to K (K < k), of θ(t), respectively. Observing that O(t) and I(t) are
while the capacity overflow ratio Φ does not exceed the independent of each other, we get the third equation.
threshold ρ. Let θ(t) be the number of busy blocks at The last equation can be obtained by replacing O(t) and
time t, implying that, there are θ(t) VMs in the ON state I(t) with their distributions.
and (k − θ(t)) VMs in the OFF state. Let O(t) and I(t) The stochastic matrix P = [pij ](k+1)×(k+1) is called
denote the number of VMs that switch state from ON to the transition matrix of the Markov chain. We see
OFF (i.e., VMs that leave the queueing system) and from that, it does not depend on the time. Let π (t) =

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 6

p0k
Algorithm 1 Calculating Minimum Blocks (CalMinBlk)
p02 p1k
p2k Input: k, the number of co-located VMs on a PM;
p00
p01 p12 pon , the switch probability from “OFF” to “ON”;
0 p11 1 p22 2 k pof f , the switch probability from “ON” to “OFF”;
p10 p21 pk2 pkk
ρ, capacity overflow ratio threshold
p20 Output: K, the minimum number of blocks that should
pk1
be reserved on a PM
pk0
1: Calculate the transition matrix P using Equ. (7)
Fig. 6. The transition graph of the stochastic process 2: Prepare the coefficient matrix of the homogeneous
{θ(0), θ(1), ..., θ(t), ...}. The stochastic process is said to system of linear equations described in Equ. (9)
be in state i (1 ≤ i ≤ k) if the number of busy blocks is i. 3: Solve the the homogeneous system via Gaussian
pij is the transition probability from state i to state j. elimination and get the stationary distribution π
4: Calculate K from π using Equ. (8)
5: return K;
(t) (t) (t)
(π0 , π1 , ..., πk ) be the distribution of the chain at time
(t)
t, i.e., πh = P r{θ(t) = h}, ∀0 ≤ h ≤ k. For our chain,
which is finite, π (t) is a vector of k+1 nonnegative entries which suggests that K is the minimum number that
∑k ∑K
(t)
such that h=0 πh = 1. In linear algebra, vectors of this guarantees h=0 πh ≥ 1 − ρ.
type are called stochastic vectors. Then, it holds that This is because, when he number of reserved blocks on
PM Hj is reduced from k to K, if the queueing system is
π (t+1) = π (t) P. in state h, which is larger than K, then capacity overflow
Suppose π is a distribution over the state space occurs, i.e., COjt = 1, and vice versa. Thus we have
{0, 1, 2, ..., k} such that, if the chain starts with an initial ∑
1≤t≤T COj
t ∑k ∑
K
distribution π (0) that is equal to π, then after a transition, Φj = = πh = 1 − πh ≤ ρ.
the distribution of the chain is still π (1) = π. Then the T
h=K+1 h=0
chain will stay in the distribution π forever:
We now show how to calculate the stationary distri-
P P P
π −→ π −→ π −→ · · · · · · bution π. According to its definition, we have π = πP,
which is equivalent to the following homogeneous sys-
Such π is called a stationary distribution. For our
tem of linear equations that can be solved by Gaussian
chain, we have the following theorem.
elimination.  k
Theorem 2: For the Markov chain defined in Fig. 6 and ∑

 πh ph0 − π0 = 0
Equ. (7), given an arbitrary initial distribution π (0) , π (t) 



will converge to the same distribution π, which satisify 

h=0
 ∑ πh ph1 − π1 = 0
 k

π = πP, and lim (π (0) Pt )h = πh , ∀0 ≤ h ≤ k. h=1 (9)

t→∞ 


 ......
Proof: According to the Markov chain convergence 

∑
 k
theorem [14], it is sufficient to prove that, our chain 
 πh phk − πk = 0
is finite, aperiodic, and irreducible. Since the number h=k
of VMs that a single PM can host is finite (in our Alg. 1 summarizes the entire process of how to cal-
case, it is k), the state space of our chain is finite. As culate the minimum number of reserved blocks, given
shown in Equ. (7), pii > 0 for any state i, so all states parameters k, pon , pof f , and ρ. Calculating the transition
are aperiodic. Finally, from any state i, we can reach matrix P requires O(k 3 ) time; solving the linear equa-
any other state j, so the transition graph is strongly tions using Gaussian elimination costs roughly O(k 3 )
connected, implying that our chain is irreducible. The time; finding K that satisfies Equ. (8) needs O(k) time.
theorem follows immediately. Therefore, the time complexity of Alg. 1 is O(k 3 ).
When the chain stays in the stationary distribution,
we see that πh is equivalent to the proportion of times
that the stochastic process is in state h. In our case, it 5.3 QUEUE
means that πh denotes the proportion of time wherein In this subsection, we present the complete server con-
the number of busy resource blocks is h. solidation algorithm, QUEUE, which is to place a set of
We then can derive the minimum number of blocks n VMs onto a set of m PMs.
that keeps the capacity overflow ratio not larger than ρ. As we mentioned before, we conservatively set the
Denote by K the minimum number of blocks; we argue size of a single block as the size of the maximum spike
that K satisfies the following constraint: of all the VMs on each PM, which may result in low
utilization if the workload spikes of the co-located VMs
∑
K−1 ∑
K
πh < 1 − ρ ≤ πh , (8) differ too much. Therefore, in QUEUE, we tend to place
h=0 h=0 VMs with similar Re ′ s on the same PM in an effort to

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 7

Algorithm 2 QUEUE pon=0.1,poff=0.5,ρ=0.10

20
pon=0.1,poff=0.5,ρ=0.05
Input: V1 , V2 , ..., Vn , specifications of n VMs; pon=0.2,poff=0.5,ρ=0.05
15
H1 , H2 , ..., Hm , specifications of m PMs;

MinN[k]
pon , the switch probability from “OFF” to “ON”; 10

pof f , the switch probability from “ON” to “OFF”; 5

ρ, capacity overflow ratio threshold;
d, the maximum number of VMs allowed on a PM; 0
0 4 8 12 16 20 24 28 32 36 40 44 48 52
Output: X, a VM-to-PM placement matrix k, the number of co-located VMs on a PM
1: // Preprocessing phase
2: M inN ← an array of size d + 1 Fig. 7. The minimum number, M inN [k], of blocks that
3: M inN [0] ← 0 should be reserved on a PM, given different settings of
4: for each k ∈ [1, d] do pon , pof f and ρ.
5: M inN [k] ← CalMinBlk(k, pon , pof f , ρ)
6: end for
minimize the average size of blocks on all PMs.
7: // Sorting phase
In the third phase (lines 12-16), we adopt the First Fit
8: Cluster VMs based on their Re ′ s
Decrease (FFD) [31] heuristic to place VMs on PMs. For
9: Sort clusters in descending order of Re
each Vi in the sorted order, we place Vi on the first PM
10: In each cluster, sort VMs in descending order of Rb
Hj that satisfies the following constraint:
11: Sort PMs in descending order of C
12: // FFD-based placement phase max{Rei , max{Res |s ∈ Tj }} × M inN [|Tj | + 1]
13: X ← [0]n×m
∑
+ Rbi + Rbs ≤ Cj , (10)
14: for each Vi in the sorted order do
s∈Tj
15: place Vi on the first Hj (in the sorted order) that
satisfies Equ. (10), and set xij ← 1; where Tj denotes the set of indices of VMs that have
16: end for already been placed on Hj , and Cj is the capacity of Hj .
17: return X; We note that, the size of the reserved resources is the
block size multiplying the number of blocks, where block
size is conservatively set to the maximum Re among all
co-located VMs. Therefore, this constraint indicates that,
reduce the average size of a single block. Alg. 2 shows
VM Vi can be placed on Hj if and only if the sum of the
the details of QUEUE, which consists of three phases.
new size of the queueing system and the new total size
In the preprocessing phase (lines 1-6), we introduce of Rb ′ s does not exceed the physical capacity of Hj . If
an array named M inN , which stores the information Equ. (10) holds, we set xij be 1. At the end of QUEUE,
about the minimum number of blocks that need to be we return the VM-to-PM mapping result, i.e., X.
reserved on a PM, given k, pon , pof f , and ρ. That is, if Finally, we present the complexity analysis of QUEUE.
there are k VMs placed on a PM, then we need to reserve In the preprocessing phase, Alg. 1 is invoked at most
M inN [k] blocks to bound the capacity overflow ratio. d+1 times. Remember that the time complexity of Alg. 1
Without loss of generality, we assume that a single PM is O(k 3 ), thus, this phase costs O(d4 ) time. The sim-
can host up to d VMs, thus we can calculate the array ple clustering phase takes O(n) time. More developed
M inN [k] for all possible k ∈ [1, d] before VM placement. clustering techniques are out of the scope of this paper.
We also let M inN [0] = 0 for compatibility. Fig. 7 shows The sorting phase takes O(n log n) time. The FFD-based
the array M inN given different settings of pon , pof f , and placement phase takes O(mn) time. Overall, the time
ρ. We notice that, for the same k, when the capacity complexity of the complete consolidation algorithm is
overflow ratio threshold ρ increases (from green circles O(d4 + n log n + mn).
to red triangles), M inN [k] decreases; when pon increases
(from green circles to blue squares), M inN [k] increases.
5.4 A Concrete Example
These observations are consistent with our intuitions.
During the sorting phase (lines 7-11), we first cluster In this subsection, we provide a concrete example to
all VMs so that VMs with similar Re ′ s are in the same better explain the details of QUEUE. In our example,
cluster1 , and then sort these clusters in the descending there are n = 8 VMs and m = 3 PMs with the following
order of Re . In each cluster, we sort VMs in the descend- parameters (see Equs. (1) and (2)):
ing order of Rb . We also sorted PMs in the descending V1 = (0.1, 0.5, 15, 13), V2 = (0.1, 0.5, 15, 13),
order of the physical capacity C. This is a cluster-level
V3 = (0.1, 0.5, 20, 15), V4 = (0.1, 0.5, 20, 10),
heuristic to let co-located VMs have similar Re ′ s, thus to
V5 = (0.1, 0.5, 25, 15), V6 = (0.1, 0.5, 10, 9),
1. We first use linear time to find the maximum, maxR, and mini- V7 = (0.1, 0.5, 15, 10), V8 = (0.1, 0.5, 10, 9),
mum, minR, of n Re ′ s, then partition all Re ′ s into c clusters, where
the i-th cluster contains those Re ′ s that satisfy minR + i−1
c
(maxR − and
minR) ≤ Re < minR + ci (maxR − minR), for i = 1, 2, ..., c. H1 = H2 = H3 = (100).

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 8

As we mentioned in Section 5.2, we assume that

all VMs have the same state switch probabilities. The PM Rb5 = 25 Rb1 = 15 Rb6 = 10
threshold ρ of capacity overflow ratio is set to 0.05. The H1 Rb3 = 20 15 15
maximum number d of VMs allowed on a single PM is
4. We now present the details of running QUEUE.
In the preprocessing phase, we are to generate the
array M inN . Taking k = 4 for example, according to
PM
Rb2 = 15 R4b = 20 R7b = 15
Equ. (7), we can obtain the transition matrix P: H2 Rb8 = 10 13 13
 
0.6561 0.2916 0.0486 0.0036 0.0001
 0.3645 0.4860 0.1350 0.0140 0.0005 
  Normal Workload Queueing Block
 
P= 
0.2025 0.4500 0.2950 0.0500 0.0025 

 0.1125 0.3500 0.3750 0.1500 0.0125  Fig. 8. The final placement of our example using QUEUE,
 
0.0625 0.2500 0.3750 0.2500 0.0625 where only 2 PMs are needed. We conservatively round
the block size up to the maximum spike size of all co-
By solving π = πP, we have the stationary distribution: located VMs on a PM, e.g., on PM H1 , the size of each
queueing block is max{Re5 , Re3 , Re1 , Re6 } = 15.
π = (0.4823, 0.3858, 0.1157, 0.0154, 0.0008).
Since π0 + π1 < 1 − ρ ≤ π0 + π1 + π2 , we have K = 2, physical resources. Let us take PM H1 in Fig. 8 for exam-
which is also the value of M inN [4]. Similarly, we can ple, by the current design of QUEUE, a total of 30 units
find that, M inN [0] = 0, M inN [1] = M inN [2] = 1, and of physical resources need to be reserved for workload
M inN [3] = 2 (see the green circles in Fig. 7). spikes; however, instead of considering the four VMs
In the clustering phase, these 8 VMs are first clustered (i.e., V5 , V3 , V1 , and V6 ) together, we can partition them
into two groups, i.e., V1 , V2 , V3 , and V5 are the first group, into two groups and consider each of them separately.
and the rest is the second group. In each cluster, we sort For instance, we choose to have V5 and V3 in the first
VMs to be in the descending order of their Rb ′ s. The group, and have V1 and V6 in the second group. For
final order of VMs is V5 , V3 , V1 , V2 , V4 , V7 , V6 , and V8 . the former group, since M inN [2] = 1, we only have to
In the third phase of QUEUE, we first try to place reserve one block with a size of max{Re5 , Re3 } = 15; for
V5 on H1 . Since Rb5 + Re5 < C1 , it succeeds and we set the latter group, we also have to reserve one block with
x51 = 1. We then try to place V3 on H1 . According to a size of max{Re1 , Re6 } = 13. In doing so, a total of 28
Equ. (10), we have units of resources are reserved, which is less than that
in the previous case.
max{Re3 , max{Re5 }} × M inN [|{3}| + 1] + Rb3 + Rb5 < C1 ,
We, therefore, have the following intuition: on each
so we set x31 = 1. We then try to place V1 on C1 , which PM, we can try to partition the co-located VMs into
also succeeds and we set x11 = 1. Similarly, we then find several groups and consider them separately, so as to
that, in the final placement, V5 , V3 , V1 , and V6 are placed improve QUEUE by reducing the amount of resources
on H1 , and the rest is placed on H2 . reserved for workload spikes.
Fig. 8 shows the final placement of our example using A key problem in achieving our goal is how to par-
QUEUE. Note that, we conservatively round the block tition a set of k VMs into non-overlapped groups, i.e.,
size up to the maximum spike size of all co-located VMs how to partition an integer k, which is an interesting and
on a PM, e.g., on PM H1 , the size of each queueing block important problem in number theory [32]. A g-partition
is max{Re5 , Re3 , Re1 , Re6 } = 15. of an integer k is a multi-set {x1 , x2 , . . . , xi , . . . , xg } with
In contrast, without opportunistic resource sharing in xi ≥ 1 for every element xi and x1 + x2 + · · · + xg = k.
the queueing blocks, if resources are provisioned for For example, {1, 2, 4} and {1, 3, 3} are two possible 3-
peak workload, then 3 PMs are needed to host these partitions of integer 7. Denote by pg (k) the number of
8 VMs, i.e., V5 , V3 , and V6 are on H1 ; V1 , V2 , and V4 are different g-partitions of an integer k, and by p(k) the
on H2 ; and the remaining two VMs are on H3 . number of all possible partitions of an integer k. For
example, p3 (7) = 4, p(7) = 15. According to [33], we
have the following recursive relations of pg (k):
6 C OPING WITH H ETEROGENOUS S PIKES
pg (k) = pg−1 (k − 1) + pg (k − g). (11)
In this section, we present how to improve the consol-
idation performance of QUEUE through more careful Alg. 3 shows the main steps to find the optimal
treatment of heterogenous workload spikes. ordered partition. Without loss of generality, we assume
Remember that, in the last section, we conservatively that, Re1 , Re2 , ..., Rek are sorted in descending order of
round the block size up to the maximum spike size of their values. The array M inN is computed using Alg. 1.
all co-located VMs on each PM, as shown in Fig. 8. It is Since the number of all possible √ partitions of an integer k
easy to see that, this kind of rounding may waste some is very large (log p(k) = O( k) [33]), enumerating them

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 9

Algorithm 3 Finding Optimal Partition (FidOptPat) Ordered

The amount of resources used
partition
Input: M inN , an array that stores the minimum num- {4} M inN [4] × 13 = 26
bers of blocks that need to be reserved on a PM, with {1, 3} M inN [1] × 13 + M inN [3] × 10 = 33
{3, 1} M inN [3] × 13 + M inN [1] × 9 = 35
respect to k, pon , pof f , and ρ; {2, 2} M inN [2] × 13 + M inN [2] × 10 = 23
Re1 , Re2 , ..., Rek , a set of workload spikes that are {1, 1, 2} M inN [1] × 13 + M inN [1] × 10 + M inN [2] × 10 = 33
sorted in the descending order of their values; {1, 2, 1} M inN [1] × 13 + M inN [2] × 10 + M inN [1] × 9 = 32
{2, 1, 1} M inN [2] × 13 + M inN [1] × 10 + M inN [2] × 9 = 32
G, the maximum number of groups
Output: S, an ordered partition Fig. 9. All possible ordered partitions on PM H2 in Fig. 8,
1: S ← {k} where the optimal ordered partition result is {2, 2}.
2: rmin ← M inN [k] × Re1
3: Let x0 ← 0 for convenience 300

4: for g = 2 to G do

Computation time (milliseconds)

250
5: Generate all g-partitions of k (using Equ. (11))
6: for each g-partition {x1 , ..., xg } do 200 d=16

for each permutation x′1 , ..., x′g of x1 , ..., xg do

d=32
7: 150
d=64
d=128

8: r←
∑ g ∑ ′
i−1 ∑i
M inN [x′i ] × max{Rej | xh < j ≤ x′h }
100

i=1 h=0 h=0 50

9: if r < rmin then
10: rmin ← r, S ← {x′1 , ..., x′g } 0
100 200 300 400 500 600 700 800 900 1000

11: end if Number of VMs

12: end for

Fig. 10. The computation cost of QUEUE with varying
13: end for
d and n. The cost of the actual placement varies with
14: end for
different physical configurations and thus is not included.
15: return S;

7 P ERFORMANCE E VALUATION
would be time-consuming. So we use G to restrict the In this section, we conduct extensive simulations and
maximum number of groups. testbed experiments to evaluate the proposed algorithms
under different settings and reveal insights of the pro-
We use S to record the best partition so far (line 1), posed design performance.
and use rmin to record the amount of resources needed
by that partition S (line 2). For each integer g (2 ≤ g ≤
G), we first generate all possible g-partitions of k using 7.1 Simulation Setup
Equ. (11); then, for each permutation x′1 , ..., x′g of a g- Two commonly-used packing strategies are considered
partition x1 , ..., xg , we compute the amount of resources here, which both use the First Fit Decrease heuristic for
needed by this ordered partition (line 8) and compare it VM placement. The first strategy is to provision VMs
with rmin : if this ordered partition uses fewer resources for peak workload (FFD by Rp ), while the second is
than S, we update S and rmin (lines 9-11). Finally, the to provision VMs for normal workload (FFD by Rb ).
optimal ordered partition S is returned. Provisioning for peak workload is usually applied for
It takes O(p(k)) time to generate all possible partitions the initial VM placement [1], where cloud tenants choose
kg−1
of an integer k [34]. Since pg (k) ∼ g!(g−1)! , generat- the peak workload as the fixed capacity of the VM to
ing all possible g-partitions (1 ≤ g ≤ G) requires guarantee application performance. On the other hand,
∑G kg−1 provisioning for normal workload is usually applied in
O( g=1 g!(g−1)! ) time. Taking G = 3 for example, since
the consolidation process, since at runtime the majority
p1 (k) = O(1), p2 (k) = O(k), and p3 (k) = O(k 2 ), of VMs are in the OFF state, i.e., most of the VMs only
generating all possible g-partitions (1 ≤ g ≤ 3) requires have normal workloads.
O(n2 ) time. For each possible permutation of a partition, We consider both the situations without and with live
evaluating the amount of resources needed requires migration, where different metrics are used to evalu-
O(k) time, thus, the total time complexity of Alg. 3 is
∑G kg−1 ate the runtime performance. For experiments without
O(k g=1 (g−1)! ). In practice, we can choose a proper G live migration, where only local resizing is allowed to
to achieve a balance between complexity and optimality. dynamically provision resources, we use the capacity
We use PM H2 in Fig. 8 as an example, where the sizes overflow ratio (COR) defined in Section 4 as the perfor-
of the four spikes are 13, 10, 10, and 9. Without loss of mance metric. Next, in our testbed experiments, we add
generality, we let Re1 = 13, Re2 = 10, Re3 = 10, and Re4 = 9. live migration to our system to simulate a more realistic
We consider all ordered 1-partition, 2-partitions, and 3- computing cluster, in which the number of migrations
partitions of k = 4, i.e., G = 3. Fig. 9 shows the results, reflects the quality of performance, and the number of
where the optimal ordered partition is {2, 2}. active PMs reflects the level of energy consumption.

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 10

Number of PMs used 150 150 150

Number of PMs used

Number of PMs used
100 100 100

50 50 50

RP RP RP
QUEUE QUEUE QUEUE
RB RB RB
0 0 0
100 200 300 400 500 600 700 800 900 1000 100 200 300 400 500 600 700 800 900 1000 100 200 300 400 500 600 700 800 900 1000
Number of VMs Number of VMs Number of VMs

(a) Rb = Re (b) Rb > Re (c) Rb < Re

Fig. 11. Packing results. The common settings are: ρ = 0.01, d = 16, pon = 0.01, pof f = 0.09, and Cj ∈ [80, 100]. (a)
Rb = Re , Rb and Re ∈ [2, 20]. (b) Rb > Re , Rb ∈ [12, 20], Re ∈ [2, 10]. (c) Rb < Re , Rb ∈ [2, 10], Re ∈ [12, 20].

100% 100% 100%

70% 70% 70%
50% 50% 50%
30% 30% 30%

10% QUEUE 10% QUEUE 10%

QUEUE
Average COR

Average COR

Average COR
RB RB RB
Capacity Overflow Ratio Threshold Capacity Overflow Ratio Threshold Capacity Overflow Ratio Threshold

1% 1% 1%

0.1% 0.1% 0.1%

100 200 400 600 800 1000 100 200 400 600 800 1000 100 200 400 600 800 1000
Number of VMs Number of VMs 1XPEHURIVMs

(a) Rb = Re (b) Rb > Re (c) Rb < Re

Fig. 12. Comparison results of QUEUE and RB with respect to capacity overflow ratio (COR).

7.2 Simulation Results indicates the frequency of spike occurrence. For a bursty
workload, the spikes usually occur with low frequency
We first evaluate the computation cost of our algorithm
and short duration, therefore, we choose pon = 0.01 and
briefly, and then quantify the reduction of the number
pof f = 0.09. Workload patterns are distinguished via
of running PMs, as well as compare the runtime perfor-
setting different ranges for Rb and Re . For Figs. 11(a)
mance with two commonly-used packing strategies.
and 12(a), Rb = Re , Rb and Re ∈ [2, 20]; for Figs. 11(b)
To investigate the performance of our algorithm in
and 12(b), Rb > Re , Rb ∈ [12, 20], Re ∈ [2, 10], and for
various settings, three kinds of workload patterns are
Figs. 11(c) and 12(c), Rb < Re , Rb ∈ [2, 10], Re ∈ [12, 20].
used for each experiment: Rb = Re , Rb > Re and
Rb < Re , which denote workloads with normal spike We see that QUEUE significantly reduces the number
size, small spike size, and large spike size, respectively. It of PMs used, as compared with FFD by Rp (denoted
will be observed later that, the workload pattern of VMs as RP). When Rb < Re , the number of PMs used in
does affect the packing result, number of active PMs, and QUEUE is reduced by 45% compared with RP, where
number of migrations. the ratios for Rb = Re and Rb > Re are 30% and
According to the results in Section 5.3, the time com- 18%, respectively. FFD by Rb (denoted as RB) uses even
plexity of QUEUE is O(d4 + nlogn + mn). In Fig. 10, we fewer PMs, but the runtime performance is disastrous
present the experimental computation cost of QUEUE according to Fig. 12. The COR of RB is unacceptably
with reasonable d and n values. We see that, our algo- high. With larger spike sizes (Rb < Re ), the packing
rithm incurs very few overheads with moderate n and d result of QUEUE is better, because more PMs are saved
values. The cost variation with respect to n is not even compared with RP, and fewer additional PMs (for live
distinguishable in the millisecond-level. migrations) are used compared with RB (see Fig. 11(c)).
To evaluate the consolidation performance of QUEUE Simultaneously, with larger spike sizes, the average COR
in different settings, we then choose Rb and Re uniform- of QUEUE is slightly higher, but is still bounded by ρ
ly and randomly from a certain range for each VM. We (see Fig. 12(c)). The case of smaller spike sizes shows the
repeat the experiments multiple times for convergence. opposite results.
The capacity overflow ratio (COR) is used here as the We mention that, there are very few PMs with CORs
metric of runtime performance. Since FFD by Rp never slightly higher than ρ in each experiment. This is be-
incurs capacity violations, it is not included in the per- cause a Markov chain needs some time to enter into its
formance assessment. stationary distribution. Though we did not theoretically
Figs. 11 and 12 show the packing results and COR evaluate whether the chain constructed in Section 5
results, respectively. The common settings for three sub- is rapid-mixing, in our experiments, we find that the
figures are as follows: ρ = 0.01, d = 16, pon = 0.01, pof f = time period before the chain enters into its stationary
0.09, and Cj ∈ [80, 100]. As we discussed in Section 3, pon distribution is very short.

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 11

4000
Schedule Module

3500
Local Resizing
Consolidation Module

Number of requests
& Live Migration
3000
Performance Monitor Logger
2500

Place and Start VM View Dynamic Performance

2000
VM Configuration
XCP API
1500
PM Configuration

1000
0 50 100 150 200 250 300 350 400 450 500
Time
Fig. 13. The architecture of our testbed.

Fig. 14. Sample of the synthetic workload used in our

7.3 Testbed Experiment testbed experiment.
We use Xen Cloud Platform (XCP) 1.3 [35] as our Number of users
Pattern Rb Re
testbed to enable live migration in our system. XCP is Normal Peak
an open-source cloud platform of its commercial coun- M S 800 1200
Rb > R e
L M 1600 2400
terpart XenServer. Our proposed scheme can be easily S S 400 800
integrated into any existing enterprise-level computing Rb = Re M M 800 1600
cloud since it simply computes the amount of reserved L L 1600 3200
S M 400 1200
resources on each PM. A total of 15 machines (Intel Rb < R e
M L 800 2400
Core i5 Processor with four 2.8 GHz cores and 4 GB
memory) are used. Ubuntu 12.04 LTS Server Edition Fig. 15. Various workload patterns in our experiments.
is installed both on the PMs and VMs. The resource
type in QUEUE can be any one-dimensional resource
is quantified by the number of requests and each VM
such as CPU, memory, disk I/O, network bandwidth,
generates its workload with its respective Rb and Re .
or any combination of them that can be mapped to
Fig. 14 shows a sample of the synthetic workload.
one dimension. For simplicity, memory is designated as
We are also interested in studying the effect of dif-
the resource type concerned in out testbed experiments.
ferent workload patterns, thus, Rb and Re are classified
Dynamic scheduling is integrated into our testbed, thus
into three types: small (S), medium (M), and large (L). A
to automatically scale up/down on-demand, as well as
certain amount of users can be accommodated for each
to conduct live migration when local resizing is not
size—400 for small, 800 for medium and 1600 for large.
capable of allocating enough resources.
Fig. 15 shows the details of various workload patterns
Fig. 13 shows the architecture of our testbed. We
in our testbed experiments.
developed three main modules: consolidation module,
performance monitor, and schedule module. QUEUE is
implemented in the consolidation module. To construct 7.4 Testbed Experiment Results
the M inN array, QUEUE gets VM specifications from The packing results in the testbed experiments are con-
cloud users and the predetermined system parameters sistent with those in Fig. 11, so we choose not to repeat
(e.g., ρ, and d) from XCP API. The M inN array can them. Allowing live migration in our testbed makes the
be reused as long as all of k, pon , pof f , and ρ remain number of PMs used vary over time, so we record the
unchanged. The FidOptPat algorithm (Alg. 3) may be number of PMs used and the number of migrations
time-consuming. To improve time-efficiency of QUEUE, in the evaluation period. Generally speaking, if a web
we can only invoke FidOptPat when a PM does not server runs for a very long time, it probably will not
have enough physical resources to accommodate another quit in the future, so we assume that the number of PMs
VM. The second module is responsible for periodical- used and the number of migrations remain unchanged
ly retrieving performance statistics from XCP API and after the evaluation period. Hence more PMs used at the
forwarding them to the schedule module, which makes end of the evaluation period mean more overall energy
decisions about local resizing and live migration, and consumption. Therefore, we use them—the total number
sends these decisions to XCP API. of migrations and the number of active PMs at the end of
We also develop programs in VMs to simulate web the evaluation period—as the performance metrics. We
servers dealing with computation-intensive user request- update the measurements of these metrics every σ=30
s. When the number of users visiting the server is more seconds, and the length of the evaluation period is 100σ
than usual, a workload spike occurs. Users send their seconds. In fact, we observe that, the system becomes
requests to the server from time to time, and the time stable within about 10σ seconds.
interval between two consecutive requests from the same For each workload pattern, we compare the runtime
user follows negative exponential distribution with the performance of three consolidation strategies—QUEUE,
mean being 1. Since in reality this interval cannot be RB, and a simple burstiness-aware algorithm (denoted
infinitely small, we set a lower limit of 0.1. The workload as RB-EX). RB-EX is simply to reserve at least δ percent

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 12

16
QUEUE
20
RB QUEUE

1XPEHURIPigrations
14 18

Number of PMs used

RB−EX RB
16 RB−EX
12

14
10
12

8
10

6 8

6
4
4
2
2

0 0
Rb=Re Rb>Re Rb<Re 10 20 30 40 50 60 70 80 90 100
Time[VHFRQGV
(a) Number of PMs used

100
Fig. 17. Comparison of time-order patterns of migration
90
QUEUE
RB
events during one of the experiments for Rb = Re . Similar
Number of migrations

80
RB−EX
results are observed for Rb > Re and Rb < Re .
70

60

50 the over-provisioned PM tends to become the source PM

40
of migration later, which causes a vicious feedback circle
30
where migrations occur constantly inside the system,
20

10
while the number of PMs used keeps at a low level.
0
Rb=Re Rb>Re Rb<Re
We call this phenomenon cycle migration. The results of
RB-EX are more subtle. As we have observed, two kinds
(b) Number of migrations
of results are possible for RB-EX depending on different
Fig. 16. Bars show the average values, and the extend- experiment settings: (1) RB-EX uses slightly more PMs
ed whiskers show the maximum and minimum values. than RB, while cycle migration still exists like in RB; and
(ρ=0.01, pon =0.01, pof f =0.09, σ = 30s, δ = 0.3 for (2) in RB-EX, cycle migration disappears, but more PMs
RB-EX, the length of evaluation period is 100σ, and VM are used than QUEUE. From this point of view, RB-EX
configurations are set based. on Fig. 15.) performs less efficiently than QUEUE.

7.5 Summary
of all resources on each PM, which is an applicable
Key observations are summarized as follows.
consolidation strategy when, in reality, nothing about the
workload pattern (except the existence of burstiness) is (1) QUEUE reduces the number of active PMs by up to
known. In our experiments, we choose δ = 30%. The 45% with large spike size (Rb < Re ) and up to 30%
result is averaged over 10 executions for convergence. with normal spike size (Rb = Re ) in comparison
Fig. 16 shows the comparison results. At the end of with provisioning for peak workload.
the evaluation period, on average, RB uses fewer PMs (2) QUEUE incurs very few migrations, while both
than QUEUE, but it incurs many more migrations than RB and RB-EX incur excessive migrations at the
QUEUE; the performance of RB-EX is between RB and beginning of each experiment due to the over-tight
QUEUE. These performance gaps attenuate in Rb > Re initial packing, and the number of PMs used in RB
and enlarge in Rb < Re . or RB-EX increases rapidly during this period.
We also investigate the time-order patterns of migra- (3) Due to falsely picking migration targets, i.e., idle
tion events. As shown in Fig. 17, in general, QUEUE deception, RB incurs an unacceptably large number
incurs very few migrations throughout the evaluation of migrations constantly throughout the experiment,
period. At the beginning of the evaluation period, RB and the overall performance is seriously degraded.
and RB-EX incurs excessive migrations due to the over- (4) RB-EX performs less efficiently than QUEUE, while
tight initial VM placement, and the number of PMs either cycle migration exists or cycle migration dis-
used increases rapidly during this period. RB incurs appears, but more PMs are used than QUEUE.
an unacceptably large number of migrations through-
out the evaluation period, while RB-EX either incurs 8 D ISCUSSIONS
considerable number of migrations constantly, and uses Overhead of learning parameters. One limitation of the
only slightly more PMs than RB, or incurs very few 2-state Markov chain model is that, learning parameters
migrations as QUEUE and uses more PMs than QUEUE requires computing clouds to provide tentative deploy-
(sometimes uses the same number of PMs as QUEUE). ments, which may incur additional overhead to clouds.
To explain this phenomenon, we introduce a term However, this overhead can be drastically reduced if
idle deception to refer to the situation where a PM is tenants have to reserve resources for the same type of
falsely reckoned idle. In a highly-consolidated cloud, idle VMs repeatedly and lastingly. For example, about 40% of
deception is very likely to happen, i.e., a busy PM is applications are recurring in Bing’s production data cen-
likely to be selected as a migration target. As a result, ters [36]. For the same type of VMs, the cloud provider

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 13

only needs to offer one tentative deployment, and the 9 C ONCLUSION

same results could be fed back to tenants who want to In a highly consolidated computing cloud, the VM per-
deploy that type of VM. Thus, the tentative deployment formance is prone to degradation without an appropriate
overhead for cloud providers would be greatly reduced. VM placement strategy, if various and distinct burstiness
Capacity overflow due to inaccurate workload model. exists. To alleviate this problem, we have to activate
This happens when the model training phase is too more PMs, leading to more energy consumption. To
short to represent the actual workload pattern of a balance the performance and energy consumption with
VM. This kind of capacity overflow is not the main respect to bursty workload, we propose to reserve a
problem we consider in the paper; instead, we assume certain amount of resources on each PM that form a
that it is the cloud user’s responsible to adjust its model queueing sytem to accommodate burstiness. To quantify
parameters to best fit its objective. When the cloud user the amount of reserved resources is not a trivial problem.
finds the requested resource is not enough to support In this paper, we propose a burstiness-aware server
his/her application, the user may enlarge its model consolidation algorithm based on the two-state Markov
parameters; otherwise, when the user finds that a part chain. We use a probabilistic performance constraint and
of the requested resource is idle most of the time, the show that the proposed algorithm is able to guarantee
user may reduce its model parameters. In this paper, this performance constraint. The simulation and testbed
we are interested in bounding the ratio of the capacity results show that, QUEUE improves the consolidation
overflow due to insufficient queueing blocks, and we ratio by up to 45% with large spike size and around
use queueing theory to derive the minimal number of 30% with normal spike size, as compared to those pro-
queueing blocks that ensure probabilistic performance visioning for peak workload, and a better balance of
guarantee on each PM. performance and energy consumption is achieved in
Different pon ′ s and pof f ′ s. In QUEUE, we assume comparison with other commonly-used schemes.
that all VMs have the same state switch probabili-
ties, because the problem becomes very challenging
when VMs have various pon ′ s and pof f ′ s. It has been ACKNOWLEDGMENTS
proved in [22] that the bin packing problem becomes We thank the reviewers for their insightful suggestions.
#P-complete even when the sizes of items follow the This work was supported in part by NSFC (61472181,
Bernoulli distribution—a simplified version of two-state 61321491, and 91218302), EU FP7 IRSES MobileCloud
Markov chain. Therefore, it is extremely hard to have Project (612212), Key Project of Jiangsu Research Pro-
approximate solutions for situations with different pon ′ s gram (BE2013116), and Collaborative Innovation Center
and pof f ′ s. In practice, we can cluster VMs based on their of Novel Software Technology and Industrialization.
pon ′ s and pof f ′ s (i.e., pion and piof f are the 2-D coordinates
of Vi in a plane) [37], and apply QUEUE to each cluster.
Online problem. We emphasize that QUEUE can easi- R EFERENCES
ly adapt to the online situation. When a new VM arrives, [1] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Kon-
we place it on the first PM that satisfies the constraint winski, G. Lee, D. Patterson, A. Rabkin, I. Stoica et al., “A view
in Equ. (10), and recalculate the size of the queueing of cloud computing,” Communications of the ACM, vol. 53, no. 4,
pp. 50–58, 2010.
system; When a VM leaves, we simply recalculate the [2] M.-H. Oh, S.-W. Kim, D.-W. Kim, and S.-W. Kim, “Method
size of the queueing system on the PM; When a batch of and architecture for virtual desktop service,” 2013, US Patent
new VMs arrives, we use the same scheme as Alg. 2 to 20,130,007,737.
[3] M. Marzolla, O. Babaoglu, and F. Panzieri, “Server consolidation
place them. Additionally, if pon and pof f varies among in clouds through gossiping,” in Proc. of IEEE WoWMoM 2011,
VMs, we need to round them to uniform values. In pp. 1–6.
this situation, VM arrival and VM leave may affect the [4] W. Vogels, “Beyond server consolidation,” ACM Queue, vol. 6,
accuracy of the rounded pon and pof f values, which no. 1, pp. 20–26, 2008.
[5] N. Bobroff, A. Kochut, and K. Beaty, “Dynamic placement of vir-
requires periodical recalculation of pon and pof f . tual machines for managing sla violations,” in Proc. of IFIP/IEEE
Multi-dimensional resource. The resource type in the IM 2007, pp. 119–128.
proposed algorithm is one-dimensional; here, we outline [6] S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken,
“The nature of data center traffic: measurements & analysis,” in
how to transform it into a multi-dimensional version. Proc. of ACM IMC 2009, pp. 202–208.
If each dimension of resources is correlated, we can [7] N. Mi, G. Casale, L. Cherkasova, and E. Smirni, “Injecting realistic
map them to one dimension and apply the proposed burstiness to a traditional client-server benchmark,” in Proc. of
IEEE ICAC 2009, pp. 149–158.
algorithm without any major modifications. Otherwise, [8] D. Xie, N. Ding, Y. C. Hu, and R. Kompella, “The only constant is
Alg. 1 could be applied to each dimension and could be change: incorporating time-varying network reservations in data
used to quantify the amount of reserved resources for centers,” in ACM SIGCOMM 2012, pp. 199–210.
[9] S. Zhang, Z. Qian, J. Wu, S. Lu, and L. Epstein, “Virtual network
each dimension independently. In the latter case, QUEUE embedding with opportunistic resource sharing,” IEEE Transac-
in Alg. 2 is not applicable, so we need to use another tions on Parallel and Distributed Systems, vol. 25, no. 3, pp. 816–827,
heuristic such as First Fit to place VMs on PMs. The March 2014.
[10] M. Wang, X. Meng, and L. Zhang, “Consolidating virtual ma-
multi-dimension issue with bursty workload is left as chines with dynamic bandwidth demand in data centers,” in Proc.
part of future work. of IEEE INFOCOM 2011, pp. 71–75.

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/TPDS.2015.2425403, IEEE Transactions on Parallel and Distributed Systems
IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. –, NO. -, MONTH YEAR 14

[11] A. Verma, G. Kumar, and R. Koller, “The cost of reconfiguration Sheng Zhang received his BS and PhD de-
in a cloud,” in Proc. of ACM Middleware 2010, pp. 11–16. grees from Nanjing University in 2008 and 2014,
[12] W. Voorsluys, J. Broberg, S. Venugopal, and R. Buyya, “Cost of respectively. Currently, He is an assistant pro-
virtual machine live migration in clouds: A performance evalua- fessor in the Department of Computer Science
tion,” in Proc. of IEEE CloudCom 2009, pp. 254–265. and Technology, Nanjing University. He is also a
[13] Z. Luo and Z. Qian, “Burstiness-aware server consolidation via member of the State Key Lab. for Novel Software
queuing theory approach in a computing cloud,” in Proc. of IEEE Technology. His research interests include cloud
IPDPS 2013, pp. 332–341. computing and mobile networks. To date, he
[14] S. M. Ross, Introduction to Probability Models, Tenth Edition. Or- has published more than 15 papers, including
lando, FL, USA: Academic Press, Inc., 2011. those appeared in IEEE Transactions on Parallel
[15] X. Meng, V. Pappas, and L. Zhang, “Improving the scalability of and Distributed Systems, IEEE Transactions on
data center networks with traffic-aware virtual machine place- Computers, ACM MobiHoc, and IEEE INFOCOM. He received the Best
ment,” in Proc. of IEEE INFOCOM 2010, pp. 1–9. Paper Runner-Up Award from IEEE MASS 2012.
[16] D. Jayasinghe, C. Pu, T. Eilam, M. Steinder, I. Whally, and E. S-
nible, “Improving performance and availability of services hosted
on iaas clouds with structural constraint-aware virtual machine Zhuzhong Qian is an associate professor at the
placement,” in Proc. of IEEE SCC 2011, pp. 72–79. Department of Computer Science and Technol-
[17] Q. Zhang, M. F. Zhani, R. Boutaba, and J. L. Hellerstein, “HAR- ogy, Nanjing University, P. R. China. He received
MONY: Dynamic heterogeneity-aware resource provisioning in his PhD. Degree in computer science in 2007.
the cloud,” in Proc. of IEEE ICDCS 2013, pp. 510–519. Currently, his research interests include cloud
[18] S. Zhang, Z. Qian, J. Wu, and S. Lu, “SEA: Stable resource computing, distributed systems, and pervasive
allocation in geographically distributed clouds,” in Proc. of IEEE computing. He is the chief member of several na-
ICC 2014, pp. 2932–2937. tional research projects on cloud computing and
[19] M. Alicherry and T. Lakshman, “Network aware resource alloca- pervasive computing. He has published more
tion in distributed clouds,” in Proc. of IEEE INFOCOM 2012, pp. than 30 research papers in related fields.
963 – 971.
[20] C. Guo, G. Lu, H. J. Wang, S. Yang, C. Kong, P. Sun, W. Wu,
and Y. Zhang, “Secondnet: a data center network virtualization
architecture with bandwidth guarantees,” in ACM CoNEXT 2010, Zhaoyi Luo received his BS degree from Nan-
pp. 15:1–15:12. jing University in 2013. He is a Master student
[21] U. Sharma, P. Shenoy, S. Sahu, and A. Shaikh, “A cost-aware (MMath) in the David Cheriton School of Com-
elasticity provisioning system for the cloud,” in Proc. of IEEE puter Science, University of Waterloo, Canada.
ICDCS 2011, pp. 559–570. His research interests include computer-aided
[22] J. Kleinberg, Y. Rabani, and É. Tardos, “Allocating bandwidth for tools and techniques for analyzing software re-
bursty connections,” SIAM Journal on Computing, vol. 30, no. 1, quirements and specifications.
pp. 191–217, 2000.
[23] A. Goel and P. Indyk, “Stochastic load balancing and related
problems,” in Proc. of IEEE FOCS 1999, pp. 579–586.
[24] M. Chen, H. Zhang, Y.-Y. Su, X. Wang, G. Jiang, and K. Yoshihira,
“Effective VM sizing in virtualized data centers,” in Proc. of
IFIP/IEEE IM 2011, pp. 594–601. Jie Wu (F’09) is the chair and a Laura H. Carnell
[25] D. Breitgand and A. Epstein, “Improving consolidation of virtual professor in the Department of Computer and
machines with risk-aware bandwidth oversubscription in com- Information Sciences at Temple University. He
pute clouds,” in Proc. of IEEE INFOCOM 2012, pp. 2861–2865. is also an Intellectual Ventures endowed visiting
[26] Z. Gong, X. Gu, and J. Wilkes, “PRESS: Predictive elastic resource chair professor at the National Laboratory for
scaling for cloud systems,” in Proc. of CNSM 2010, pp. 9–16. Information Science and Technology, Tsinghua
[27] R. Calheiros, R. Ranjan, and R. Buyya, “Virtual machine pro- University. Prior to joining Temple University, he
visioning based on analytical performance and qos in cloud was a program director at the National Science
computing environments,” in Proc. of IEEE ICPP 2011, pp. 295– Foundation and was a Distinguished Professor
304. at Florida Atlantic University. His current re-
[28] G. Casale, N. Mi, and E. Smirni, “Model-driven system capacity search interests include mobile computing and
planning under workload burstiness,” IEEE Transactions on Com- wireless networks, routing protocols, cloud and green computing, net-
puters, vol. 59, no. 1, pp. 66–80, 2010. work trust and security, and social network applications. Dr. Wu regularly
[29] G. Casale, N. Mi, L. Cherkasova, and E. Smirni, “Dealing with publishes in scholarly journals, conference proceedings, and books.
burstiness in multi-tier applications: Models and their parameter- He serves on several editorial boards, including IEEE Transactions
ization,” IEEE Transactions on Software Engineering, vol. 38, no. 5, on Service Computing and the Journal of Parallel and Distributed
pp. 1040–1053, 2012. Computing. Dr. Wu was general co-chair/chair for IEEE MASS 2006,
[30] Z. Huang and D. Tsang, “SLA guaranteed virtual machine con- IEEE IPDPS 2008, IEEE ICDCS 2013, and ACM MobiHoc 2014, as
solidation for computing clouds,” in Proc. of IEEE ICC 2012, pp. well as program co-chair for IEEE INFOCOM 2011 and CCF CNCC
1314–1319. 2013. He was an IEEE Computer Society Distinguished Visitor, ACM
[31] V. V. Vijay, Approximation Algorithms. Springer, 2003. Distinguished Speaker, and chair for the IEEE Technical Committee on
[32] G. H. Hardy, E. M. Wright, D. R. Heath-Brown, and J. H. Silver- Distributed Processing (TCDP). Dr. Wu is a CCF Distinguished Speaker
man, An introduction to the theory of numbers. Clarendon press and a Fellow of the IEEE. He is the recipient of the 2011 China Computer
Oxford, 1979, vol. 4. Federation (CCF) Overseas Outstanding Achievement Award.
[33] G. E. Andrews, The Theory of Partitions. Cambridge University
Press, 1976.
Sanglu Lu received her BS, MS, and PhD de-
[34] I. Stojmenović and A. Zoghbi, “Fast algorithms for genegrating
grees from Nanjing University in 1992, 1995,
integer partitions,” International Journal of Computer Mathematics,
and 1997, respectively, all in computer science.
vol. 70, no. 2, pp. 319–332, 1998.
She is currently a professor in the Department
[35] “Xen cloud platform,” http://www.xenproject.org/.
of Computer Science and Technology and the
[36] S. Agarwal, S. Kandula, N. Bruno, M.-C. Wu, I. Stoica, and
State Key Laboratory for Novel Software Tech-
J. Zhou, “Re-optimizing data-parallel computing,” in Proc. of
nology. Her research interests include distribut-
USENIX NSDI 2012, pp. 281–294.
ed computing, wireless networks, and pervasive
[37] R. Xu and I. Wunsch, D., “Survey of clustering algorithms,” IEEE
computing. She has published over 80 papers in
Transactions on Neural Networks, vol. 16, no. 3, pp. 645–678, May
referred journals and conferences in the above
2005.
areas. She is a member of IEEE.

1045-9219 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.