Measuring the Robustness of Resource Allocations

in a Stochastic Dynamic Environment

Jay Smith1,2 , Luis D. Briceño2 , Anthony A. Maciejewski2 , Howard Jay Siegel2,3 ,

Timothy Renner3 , Vladimir Shestak2 , Joshua Ladd4 , Andrew Sutton3 , David Janovy2 ,
Sudha Govindasamy2 , Amin Alqudah2 , Rinku Dewri3 , Puneet Prakash2 ∗
1
IBM Colorado State University
2
6300 Diagonal Highway Dept. of Electrical Engineering
3
Boulder, CO 80301 Dept. of Computer Science
4
[email protected] Mathematics Department
Fort Collins, CO 80523–1373 USA
{ldbricen,aam,hj}@colostate.edu

Abstract different capabilities to optimize system performance fea-

tures. Often, heterogeneous, distributed computing sys-
Heterogeneous distributed computing systems often must tems must operate in an environment replete with uncer-
operate in an environment where system parameters are tainty. Robustness in this context can be defined as the de-
subject to uncertainty. Robustness can be defined as the gree to which a system can function correctly in the pres-
degree to which a system can function correctly in the pres- ence of parameter values different from those assumed [1].
ence of parameter values different from those assumed. We We present the use of a stochastic robustness metric [14]
present a methodology for quantifying the robustness of re- to quantify the robustness of a resource allocation in a dy-
source allocations in a dynamic environment where task ex- namic environment. This formulation of the stochastic ro-
ecution times are stochastic. The methodology is evaluated bustness metric is used to predict the typical relative perfor-
through measuring the robustness of three different resource mance of three different resource allocation heuristics taken
allocation heuristics within the context of a stochastic dy- from the literature and adapted to the presented problem. A
namic environment. A Bayesian regression model is fit to the Bayesian regression model is fit to the combined results of
combined results of the three heuristics to demonstrate the the three heuristics to demonstrate the relationship between
correlation between the stochastic robustness metric and the stochastic robustness metric and the presented perfor-
the presented performance metric. The correlation results mance metric. The accuracy of the robustness predictions
demonstrated the significant potential of the stochastic ro- are then evaluated to determine their utility in predicting
bustness metric to predict the relative performance of the resource allocation heuristic performance in a dynamic en-
three heuristics given a common objective function. vironment.
The major contribution of this paper is a mathemat-
ical formulation for quantifying the robustness of a re-
source allocation in a stochastic dynamic environment and
1. Introduction
a methodology for applying the robustness formulation to
predict heuristic performance. We demonstrate the pro-
Heterogeneous parallel and distributed computing is de- posed methodology by successfully predicting the relative
fined as the coordinated use of compute resources that have performance of three heuristics taken from the literature and
∗ This research was supported by the NSF under Contract No: CNS- adapted to this environment. Finally, we present a detailed
0615170, by the Colorado State University Center for Robustness in Com- evaluation of the three heuristics using both the proposed
puter Systems (funded by the Colorado Commission on Higher Education robustness metric and a performance objective appropriate
Technology Advancement Group through the Colorado Institute of Tech-
to the studied environment.
nology), and by the Colorado State University George T. Abell Endow-
ment. The environment considered in this research is that of
1-4244-0910-1/07/$20.00 c 2007 IEEE. a heterogeneous, distributed computing system designed to

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
service a high volume web site of world-wide interest. The 2. Problem Statement
system being modeled was used to implement the 1998
World Cup web site [3] that processed more than 1.3 bil- 2.1. Introduction
lion HTTP requests during the summer of 1998. The web
site was provided to a world-wide audience by four het-
The system studied in this research is an instance of a
erogeneous, geographically dispersed systems—each with
more general class of dynamic, heterogeneous computing
their own processing capacity and workload distribution
(HC) system where task arrival times are not known in ad-
techniques. This class of system is very challenging to
vance and exact task execution times are uncertain prior
implement but occurs surprisingly frequently. The World
to their completion. All incoming tasks to the system are
Cup football tournament is just one example of an event
assumed to have been previously classified into one of C
of world-wide appeal that necessitates web based cover-
classes prior to their arrival. Each of the classes corresponds
age. Sites of this type are typically constructed for specific
to a gross classification of the relative complexity of the re-
events and the volume of traffic is on the order of billions
quest being processed. Each task class defines a set of pmfs,
of requests in a period of only a few months or less. Other
where each pmf describes the probability of all execution
such events might include the Summer Olympics, the Win-
times for that class on a given machine within the HC suite.
ter Olympics, or the tour de France; all are reasonably ex-
Further, all of the classes of tasks (HTTP requests) that the
pected to draw the attention of a world-wide audience in the
system may be asked to perform are known in advance, i.e.,
billions.
the web server has prior knowledge about what web pages
it is providing.
This research developed resource allocation heuristics Each arriving task has a relative deadline limiting the to-
for a single location of a distributed system capable of tal time available to process each request. The relative dead-
processing a high volume of requests. Incoming requests line for each task class is assumed to have been established
were dispersed to one of four processing centers. Thus, in advance. Because tasks in this environment are HTTP
each location was responsible for processing some fraction requests for data, made by a user of the website, it is as-
of the total traffic to the site. This work focused on a loca- sumed that if a task cannot be completed by its deadline
tion comprising eight heterogeneous servers responsible for then the request can be considered to be “timed out” and
processing 45% of the overall traffic [2]. the user that submitted the original request will make the
request again. Therefore, there is no benefit to completing
tasks that miss their deadlines and, consequently, tasks that
A task is defined to be a menu-driven HTTP request for miss their deadlines will be discarded.
data from the web site. Mapping tasks to machines in this
distributed system is rather challenging as it must be done 2.2. Performance Metric
under uncertainty because the exact execution time required
to process a task is not known a priori. However, past ob- In this environment, each incoming task has a hard dead-
servations of task execution times can be used to construct line for its completion, i.e., failure to complete a task by its
a probability mass function (pmf) [17] that models the pos- deadline will result in a penalty. To model the impact of
sible execution times for a given task. The pattern of task missing a task deadline the resource allocation heuristic will
arrivals to the site was modeled after real traffic patterns ob- be penalized by a constant factor of 1 for each task deadline
served by the 1998 World Cup web site [2]. that is missed. That is, for a given task i with deadline βimax
let comp(i) be the actual completion time of task i and de-
fine the cost to process a task i, denoted cost(i), as follows
In the next section, we present the details of the problem
to be addressed. Section 3 defines a means for determin- 
0, if comp(i) ≤ βimax ;
ing stochastic completion times for tasks in this system us- cost(i) = (1)
1, otherwise.
ing the details of the problem statement. The definition of
stochastic completion times is then used in Section 4 to de- The overall cost of a resource allocation is defined as the
rive a stochastic robustness metric that is subsequently used sum of the cost of each processed task. Define P T as the
to predict heuristic performance. In Section 5, we present set of all tasks that are processed by the system, then the
the heuristics that have been developed as part of this re- objective of a resource allocation in this environment can
search. Details of the simulation environment are presented be expressed as,
in Section 6 and the results of the heuristics are evaluated
in Section 7. Section 8 presents an overview of the relevant
X
Minimize cost(i). (2)
related work. Section 9 concludes the paper. ∀i∈P T

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
Intuitively, resource allocations in this environment are ex- Let M Q(t) be the set of all tasks that are either pend-
pected to minimize the number of tasks that miss their dead- ing execution or are currently executing on any of the M
lines. machines in the HC suite at time t. To determine the com-
pletion time for a task i on machine j at time t, identify the
2.3. Mapping Events subset of tasks in M Q(t) that were mapped to machine j
in advance of task i, denoted M Qij (t). The execution time
pmfs for the pending tasks will be convolved [9] with the
All of the resource allocation heuristics evaluated in this
completion time distribution of the currently executing task
work operate in a pseudo-batch mode [10, 11]. In a pseudo-
and the execution time distribution for task i on machine j
batch mode heuristic, all tasks that have not begun execu-
to produce the stochastic completion time pmf for task i on
tion and are not next in line to begin execution can be con-
machine j.
sidered for remapping when a mapping event occurs. Map-
The execution time pmf for the currently executing task
ping events occur within the system whenever a new task
on machine j requires some additional processing prior to
arrives or an existing task completes.
its convolution with the pmfs of the pending tasks to create
a completion time pmf. For example, if the currently exe-
3. Stochastic Task Completion Time cuting task on machine j began execution at time tj prior
to time t, some of the impulse values of the pmf describ-
ing the completion time of the currently executing task may
Although some tasks may belong to the same class, task
be in the past. Therefore, to accurately describe the com-
execution times may still vary depending on the details of
pletion time of task i at time t requires that these past im-
the data requested. For this reason, task execution times are
pulses be removed from the pmf and the remaining distri-
modeled as random variables. In addition, it is reasonable
bution renormalized. After renormalization, the resulting
to assume that each task execution time is independent be-
distribution describes the completion time of the currently
cause any single HTTP request can be satisfied without any
executing task at time t on machine j. To simplify notation,
need to process another HTTP request, i.e., each request is
define an operator GT (s, d) that accepts a scalar s and a
self-contained.
pmf d as input and returns a renormalized probability distri-
Let each task i belong to exactly one class ci in the set of
bution where all impulse values of the returned distribution
all task classifications C where membership in a class im-
are greater than s. The completion time pmf of the currently
plies a specific random variable Tci j representing the execu-
executing task on machine j is determined by applying the
tion time of that task class on machine j (one of the M ma-
GT operator to its completion time pmf, using the current
chines in the HC suite). In this research, it is assumed that
time t. The resulting distribution is then convolved with the
the probability distributions describing the random variable
pmfs of the pending tasks on machine j and the execution
Tci j were created from measurements of the response times
time distribution of task i to produce the completion time
of actual requests for data from the site. A typical method
pmf for task i on machine j at the current time t.
for creating such a distribution relies on a histogram esti-
mator [17] that produces a discrete probability distribution
known as a probability mass function. Define fcj to be a 4. Stochastic Robustness Metric (SRM) for a
unimodal pmf describing the execution time of tasks in class Dynamic Mapper
c on machine j. We consider only unimodal distributions of
execution times to simplify the application of the stochastic 4.1. Instantaneous SRM
robustness metric.
Determining the completion time for a machine j, and Recall that the individual deadline of each task has been
therefore the completion time of a particular task i, requires defined in advance; let βimax denote the deadline for the ith
a means of combining the execution times for all tasks as- task to arrive to the system. Let fc1 j be the execution time
signed to that machine. In a deterministic model of task pmf of the currently executing task on machine j. Order the
execution times, the estimated execution times for all tasks members of M Qij (t) according to their scheduled order of
assigned to machine j would be summed with the machine execution on machine j and let fc2 j be the execution time
ready time to produce a completion time. A similar pro- pmf of the first pending task on machine j, with fc|M Qij (t)| j
cedure is followed in the stochastic case as well. How- as the execution time pmf of the last pending task on ma-
ever, calculating stochastic completion times requires the chine j that is ahead of task i.
summation of random variables as opposed to determinis- Convolution of a scalar with a pmf has the effect of shift-
tic values. The summation of random variables given their ing the pmf by the value of the scalar and has no impact on
pmfs can be found as the convolution of their corresponding the distribution of probabilities in the pmf. Therefore, if
pmfs [9]. M Qij (t) = ∅, i.e., M Qij (t) is empty, the completion time

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
distribution for task i on machine j is merely its execution 4.3 Using the Dynamic SRM value
time distribution fci j shifted to the current time. The com-
pletion time distribution at time t for task i, denoted Fi (t)
can be found as follows, In a dynamic environment, the set of tasks being consid-
 ered is constantly changing due to task arrivals and com-

 t ∗ fci j , if M Qij (t) = ∅; pletions. Recall that to compute ψ(t) the start time of the
currently executing task on each machine is required (i.e.,

Fi (t) = GT {t, tj ∗ fc1 j } ∗ fc2 j ∗ · · ·

 tj ). Determing the start time for a task i requires knowledge
 ∗f ∗f , otherwise.
c|M Qij (t)| ci j of the actual execution time of the previously executed task
(3) k on that machine to calculate task k’s actual completion
Following from our prior work on robustness [14], the time. During simulations used for heuristic evaluation, our
robustness of the finishing time for task i can be found as the methodology utilizes the expectation of the execution time
probability that task i will finish before its deadline. This pmf as the actual execution time for each task. Thus, the
probability defines a local robustness
 characteristic,  denoted start time of the subsequent task i is known, enabling the
ψi (t), and can be expressed as P Fi (t) ≤ βimax . The in- calculation of ψ(t).
dividual local robustness characteristics can then be com- Taking the expectation of the class pmfs to produce the
bined to produce the stochastic robustness metric at time t, most likely actual execution times to use for the evaluation
denoted ψ(t), as follows, simulations is reasonable in this environment because the
Y    task execution time pmfs are assumed to be unimodal. Fur-
max

ψ(t) = P Fi (t) ≤ βi . (4) ther study is required to determine the most effective ap-
∀i∈M Q(t) proach when execution time pmfs are not unimodal.
This combination of local robustness characteristics de- A second factor in evaluating a resource allocation
fines an instantaneous measure of robustness (instantaneous heuristic is the set of tasks and the ordering of task arrivals.
SRM) for this resource allocation at a particular time t. In- Because of variations in the set of tasks to be executed and
tuitively, the measure defines the probability that all tasks changes in their arrival ordering, multiple simulation trials
pending or currently executing at time t will meet their should be conducted to adequately predict the typical rel-
deadlines. ative performance among the evaluated resource allocation
heuristics. In evaluating our results, we will demonstrate
4.2. Dynamic SRM Value that in a dynamic environment a small number of simulation
trials are required to sufficiently indicate the performance of
a resource allocation heuristic relative to our given perfor-
The instantaneous measure of robustness is used as a ba-
mance objective. Each trial involves a unique set of tasks
sis for defining a single dynamic SRM value. To define that
with a unique arrival order.
value, recall that a mapping event occurs whenever a task
completes execution or a new task arrives at the system. To produce a dynamic SRM value for a resource alloca-
An instantaneous SRM value is generated at each mapping tion heuristic a relatively small number of simulations are
event during the course of a simulation trial. These instan- executed where the mean of each task execution time distri-
taneous SRM values are then combined to create a sam- bution is used as the actual execution time. This produces
ple dynamic SRM value for the resource allocation heuris- a dynamic SRM value for each resource allocation (simu-
tic. Given the relationship between the dynamic SRM value lation trial) where the dynamic SRM value is determined
and the performance metric, the dynamic SRM value can as the average of the instaneous SRM values within that
be used to predict the relative performance of two resource simulation trial. The dynamic SRM values for all of the
allocation heuristics. simulation trials are then combined by taking their average
If a heuristic consistently maintains a high ψ(t) value to determine a single dynamic SRM value for the resource
over some number of mapping events then there is a con- allocation heuristic.
sistently low probability that tasks will miss their deadlines The dynamic SRM values for different resource alloca-
over that same period. Therefore, the average ψ(t) value tion heuristics can then be compared to select the approach
over a large enough number of mapping events should cor- that is more robust within the given environment. That is,
relate with a consistently low probability that tasks will miss given the presented formulation of the instantaneous SRM,
their deadlines. That is, heuristics that maintain a high aver- a simulation trial that has a higher dynamic SRM value
age ψ(t) value can reasonably be expected to produce a low should reasonably be expected to produce fewer task dead-
cost. For this reason, the dynamic SRM value is defined as line misses. In the next section, we present the heuristics
the average of the instantaneous SRM values found, at each that are to be evaluated using the dynamic SRM value in
mapping event, during a simulation trial. Section 7.

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
5. Resource Allocation Heuristics to Evaluate 5.3. Segmented Two Phase Greedy (STG)

5.1. Introduction The segmented two phase greedy heuristic relies on

“segmenting” the collection of tasks to be allocated into n
The goal of resource allocation heuristics is to select groups and then applying a two phase greedy heuristic to
a mapping of tasks to machines and scheduling of tasks each group [18]. A weighting factor is used to determine
within a machine that minimizes an objective function. The segments. The new STG heuristic introduced here for this
presented heuristics do not directly attempt to maximize the environment calculates a task’s weight based on the prob-
stochastic robustness metric, instead focusing on minimiz- ability of the task meeting its deadline. That is, the lower
ing the primary objective function. In the results section, the probability that a task will complete within its deadline
the heuristics will be compared using both the stochastic the higher its queuing priority will be. A weight πi that is
robustness metric and their average performance relative to used to assign the task queueing order is defined for a task i
minimizing Equation 2. All of the heuristics were given a given M machines as follows,
limited amount of time to complete a mapping event.
M
P[Fi (t) ≤ βimax ]
P
5.2. Two Phase Greedy j=1
πi = . (6)
M
The Two Phase Greedy heuristic is based on the princi-
ples of the Min-Min algorithm (first presented in [8], and The individual weights are used to define a weighted
shown to perform well in many environments [7], [10], expected time to compute for each task, denoted Wij for
[15]). The heuristic allocates one task at each iteration, con- a given task i on machine j.
tinuing until all task allocations have been resolved. In the
Wij = πi E[Tci j ] (7)
first phase of each iteration, the Two Phase Greedy heuris-
tic determines the best assignment (according to the perfor-
The weighted expected execution times are combined
mance goal) for each of the tasks left unmapped. In the sec-
to produce a weighted expected completion time, denoted
ond phase, it selects the task to map based on the best result ECT
Wij (t) for a given task i on machine j at time t.
found in the first phase. The completion time distribution
The weighted expected completion time is calculated using
for a given machine j at time t is denoted F j (t). Given
Equation 8.
the set of tasks assigned to machine j at time t, denoted
M Qj (t), F j (t) can be found as follows: ECT
Wij (t) = Wij + E[F j (t)] (8)
j
F (t) = GT {t, tj ∗ fc1 j } ∗ fc2 j ∗ · · · ∗ fc|M Qj (t)| . (5)
Tasks are sorted in ascending order according to the av-
The Two Phase Greedy heuristic is summarized in Figure 1. erage of their Wij values across all machines at the time of
the mapping event. The sorted task list is then divided into
while not all tasks are mapped n segments of equal length that are allocated to machines
for each unmapped task i using a two phase greedy heuristic. The two phase greedy
find machine mj such that heuristic is used to minimize the weighted expected com-

mj ← argmin E[F j (t) ∗ fci j ] ; pletion time for the last to finish task. Figure 2 presents the
1≤j≤M details of the STG heuristic discussed here.
resolve ties arbitrarily; Because a new mapping event occurs each time a task
end for loop finishes, in general, each machine needs no more than two
let A = all (i, mj ) pairs found above pending tasks. Thus, once every machine has two tasks
select pair(s) (x, my) such that  pending, the heuristic can terminate the mapping event.
(x, y) ← argmin E[F mj (t) ∗ fci mj ] ; This was utilized in the implementation of the STG heuris-
∀(i,mj )∈A
tic to improve the heuristic’s execution time.
resolve ties arbitrarily;
map x to machine y;
update F y (t) based on assignment; 5.4. Negotiation
end while loop.
Iterative approaches have been applied to static map-
Figure 1. Pseudo-code describing the Two ping problems to search the space of task permutations in
Phase Greedy heuristic. a schedule [4, 5, 16]. Local search also has been applied
to dynamic scheduling problems [12, 13]. The negotia-
tion heuristic introduced here is modeled after such iterative

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
 sort tasks in ascending order by average Wij value this modification, denoted ω 0 , is used as an activity list for
partition sorted task list into n segments a schedule builder. The schedule builder assigns each task
for each segment S in ω 0 , in order, to the machine that maximizes the local ear-
while not all tasks are mapped liness metric as defined by Equation 9. The fitness of the
for each unmapped task i in S resulting mapping is measured using the global earliness
find the machine mECTj such that metric defined in Equation 10, where a higher global ear-
mj ← argmin Wij (t) ; liness metric indicates a more fit schedule. If the fitness of
1≤j≤M
this mapping is the best encountered, the ordering ω 0 is kept
resolve ties arbitrarily;
for the next iteration. Otherwise, the ordering is discarded
end for loop
and the original ordering ω is maintained for the next iter-
from all (i, mj ) pairs found above
ation. Negotiation terminates after N iterations have been
select pair(s) (x, y)
h such thati executed. The Negotiation heuristic is summarized in Fig-
ECT
(x, y) ← argmin Wimj
(t) ; ure 3.
∀(i,mj )
resolve ties arbitrarily; initialize ω to an ordering of all tasks to be mapped
map task x to machine y; for N iterations
update F y (t) based on assignment; randomly select two tasks for swapping
end while loop swap ordering of selected tasks
end for loop. assign resultant total ordering to ω 0
execute schedule builder using ω 0
Figure 2. Pseudo-code describing the STG
if (GEM (ω 0 ) < GEM (ω))
heuristic.
ω ← ω0
end for loop.

heuristics operating in a dynamic environment. A total or- Figure 3. Pseudo-code describing the Nego-
dering of tasks serves as input to a schedule builder that as- tiation heuristic.
signs tasks to machines such that the performance objective
is maximized. The total ordering is iteratively permuted and
the schedule builder re-applied to produce a new resource
allocation. Schedules with a higher value are kept, where
value is defined by the evaluation of a fitness function. The
6. Simulation Setup
procedure is analogous to a next-descent search in schedule
space. For this research, it is assumed that 1) the time neces-
Each iteration of the negotiation heuristic relies on com- sary for a mapping event is negligible compared to the task
puting the following two metrics. The local earliness metric arrival time, and 2) task execution times are considerably
quantifies the difference between a task’s expected finish- longer than the difference between successive inter-task ar-
ing time and its deadline given the current mapping and rival times.
scheduling. The local earliness metric for a given task i To evaluate the effectiveness of the dynamic SRM value
at time t, denoted LEMij , can be quantified as, for predicting heuristic performance we considered two sets
of simulations. In the first set of simulations, a small num-
LEMij = βimax − E[Fi (t)]. (9) ber of simulation trials were conducted to produce a dy-
namic SRM value for a resource allocation heuristic, as de-
Using the local earliness metric the global earliness metric,
fined in our methodology. For the second set of simulations,
denoted GEM for a given mapping event, can be found
a larger number of trials were conducted to produce sample
as the sum of the local earliness metrics for all mappable
cost values for each heuristic, where the actual execution
tasks. That is, given a task ordering o and an operator m(i)
time for each task is determined by sampling the execution
that returns the machine that task i has been assigned
time pmf for that task. For this work 10 simulation trials
GEM (o) =
X
LEMim(i) . (10) were used to produce a dynamic SRM value for a resource
allocation heuristic as compared to 100 simulation trials to
∀i∈M Q(t)
evaluate the performance metric.
An iteration of the negotiation algorithm is defined as All simulations consisted of 1024 tasks to be processed
follows. From a current ordering ω of tasks in M Q(t), two by eight machines, where task arrival times were not known
tasks are randomly selected for swapping (tasks are initially in advance. Each arriving task belonged to one of five
ordered by arrival times). The ordering that results from classes whose execution time pmfs for each machine were

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
known in advance. The execution time for a mapping event
for all heuristics was limited to 0.1 seconds.

7. Simulation Results

The heuristics were compared using the dynamic SRM

value and their ability to minimize cost, i.e., the number
of tasks that miss their deadlines. Recall that the dynamic
SRM value for each heuristic was constructed using a small
number of independent simulation trials, where the actual
execution times for tasks were set to the expectation of the
task execution time pmfs.
Evaluating the success of the dynamic SRM value in
comparing heuristics requires the actual performance of
each heuristic over a significant number of simulation trials.
Given the formulation of the dynamic SRM value, a high
dynamic SRM value for a heuristic should indicate a low
cost for the heuristic. In other words, heuristics that pro-
duce higher dynamic SRM values should have lower cost,
i.e., have fewer task deadline misses, than those with low Figure 4. The distributions of cost values for
dynamic SRM values. all three heuristics. The cost distributions
Figure 4 presents the cost distributions for each of the were generated using a kernel density esti-
three heuristics. Each distribution was generated using a mator and the results of the 100 simulation
kernel density estimator where the kernel function was set trials.
to be Gaussian [6]. The cost results from the 100 simulation
trials were used as the sample data for the kernel density
estimator. Also plotted in the figure are the means of each
distribution and the calculated dynamic SRM value for each cles the sample points for the Two Phase Greedy heuris-
heuristic. The cost distributions for the three heuristics are tic. Using these data points a Bayesian regression model [6]
only valid in the interval [0, 1024] corresponding to the low- was generated to fit the points to a common curve. The re-
est possible cost for a heuristic and the highest possible cost sults of the regression model are plotted in Figure 5 as the
given the defined simulation setup. the dynamic SRM value versus the logarithm of the cost.
The model combines a series of radial basis functions with
As can be seen in Figure 4, the Negotiation heuristic variance 0.2 that were uniformly distributed on the inter-
produced the lowest mean cost of 87.77 and had the high- val [0,1]. The range of cost results for the simulation trials
est dynamic SRM value of 0.609. The Two Phase Greedy were re-mapped to the interval [0,1] where 0 corresponds
heuristic had the next lowest mean cost with 279.28 and the to the smallest possible cost and 1 to the highest possible
next highest dynamic SRM value of 0.392. Finally, the STG cost, i.e., 1024. The line plotted in the figure represents the
heuristic produced the highest mean cost of 593.6 and the mean of the Bayesian model and the shaded region repre-
lowest dynamic SRM value of 0.206. sents one standard deviation around the mean. The gener-
The results of Figure 4 suggest that there is a correlation ated model appears to fit the combined sample points taken
between the dynamic SRM value and the number of tasks from the three heuristics to a single simple curve suggesting
that miss their deadlines. If there is a correlation between that there may be a correlation between changes in the dy-
the dynamic SRM value and the number of deadline misses, namic SRM value and changes in cost. Next, we consider
then a plot of all of the simulation sample points taken for the relative performance of the individual heuristics.
the three heuristics should lie on a common curve. To eval- From the plot of Figure 4, it appears that the negotia-
uate this conjecture, we combined the sample points for the tion heuristic generally outperforms the others. To more
three heuristics and applied Bayesian regression to produce accurately compare the results of the three heuristics an-
Figure 5. other distribution was generated to assess the frequency
The sample costs taken from the three heuristics are plot- with which the Negotiation heuristic outperforms the STG
ted as points in the figure where triangles represent the sam- and Two Phase Greedy heuristics. Using the resource allo-
ple points taken from the Negotiation heuristic, diamonds cation cost data taken from the 100 trials, Figure 6 shows the
represent the sample points for the STG heuristic, and cir- cost distributions with mean values for both the Negotiation

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
 Figure 5. Plot of dynamic SRM value versus the logarithm of the costs for all three heuristics. A
Bayesian regression model has been used to fit a curve to the combined set of sample points for all
three heuristics. The line in the figure is the mean of the regression model and the shaded region
represents one standard deviation around the mean.

heuristic and the STG heuristic and a cost comparison for ison,” the density estimate of the comparison cost distribu-
the two. To generate the cost comparison distribution, la- tion has a non-zero frequency for a sizable number of neg-
beled “STG vs. Negotiation Cost Comparison,” the cost val- ative values. That is, for these trials the Two Phase Greedy
ues generated for the Negotiation heuristic were subtracted Heuristic had fewer task deadline misses than the Negoti-
from those generated by the STG heuristic for each simula- ation heuristic. However, for the majority of the trials the
tion trial. The samples used to define the actual execution Negotiation heuristic outperformed the Two Phase Greedy
times were drawn in advance of the simulation trials and heuristic. From the cost comparison plot this can be seen
were the same for each heuristic. A kernel density estimator because the mean of the comparison cost distribution is
was then applied to the resultant data points to generate the positive implying that more often than not the Two Phase
distributions in the figure. For each simulation trial, the Ne- Greedy heuristic had a higher cost than Negotiation.
gotiation heuristic produced a lower cost than that produced
by the STG heuristic. However, for a small number of simu-
lation trials the STG heuristic performed comparably to the
Negotiation heuristic. These cost comparison values were
very close to zero but still slightly positive causing the tail
8. Related Work
of the cost comparison curve generated by the kernel den-
sity estimator to edge into negative values. This suggests In [14], the authors define a stochastic methodology for
that there is a small probability that given the right circum- evaluating the robustness of a resource allocation in a sta-
stances STG might perform better than Negotiation. tic environment. In that work, uncertainty in system para-
meters and its impact on system performance are modeled
Figure 7 compares the cost distribution for the Two stochastically. This stochastic model was then used to de-
Phase Greedy heuristic with the cost distribution for the Ne- rive a quantifiable measure of the robustness of a resource
gotiation heuristic. As can be seen in the comparison plot, allocation in a static environment. This was done by defin-
labeled “Two Phase Greedy vs. Negotiation Cost Compar- ing stochastic completion times in a similar manner to our

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
 Figure 6. A comparison of the STG heuristic’s Figure 7. A comparison of the Two Phase
cost distribution and the Negotiation heuris- Greedy heuristic’s cost distribution and the
tic’s cost distribution. The comparison plot, Negotiation heuristic’s cost distribution. The
labeled “STG vs. Negotiation Cost Compar- comparison plot, labeled “Two Phase Greedy
ison,” shows that the Negotiation heuristic vs. Negotiation Cost Comparison,” shows
consistently performs better than the STG that the Negotiation heuristic was not uni-
heuristic for all simulation trials. formly better than the Two Phase Greedy
heuristic.

current presentation. A major distinction between the two

formulations is that our previous work only considered a
static environment where all machines are idle at the begin-
ning of a mapping and the set of all tasks to be mapped is that all tasks complete by their deadline as opposed to the
known in advance. In this work, the completion time calcu- probability that the collection of tasks will complete by a
lations are very similar but machines may not be idle when given deadline as in [14]. A further distinction of this work
a mapping event occurs and new tasks are constantly arriv- over [14] is the use of many individual measures of the re-
ing. The stochastic completion time calculations are an im- source allocation to produce a predictive measure of the ro-
portant component of the stochastic robustness metric cal- bustness.
culation in [14]. The result of the calculation is a probabil-
ity distribution for task completion times that is then used In [11], the robustness concept is used to develop re-
in the calculation of the stochastic robustness metric for a source allocations in a dynamic environment. However,
resource allocation. that work utilizes a deterministic estimate of task execu-
Intuitively, the expression of robustness presented in [14] tion times to develop the robustness of the resource allo-
provides a measure of the likelihood that the makespan of cation’s makespan when the task execution time estimates
a resource allocation will fall within the provided bounds. vary from their predicted values. In this work, the robust-
This general concept has been used in this work but has ness of a given resource allocation heuristic is instead for-
been adapted to the details of the present dynamic environ- mulated with respect to its ability to meet individual task
ment. In this work, we were given bounds on the accept- deadlines. Another distinction between this work and [11]
able completion times for each task as opposed to a bound is that task execution times in [11] are simply deterministic
on the acceptable completion time for a collection of tasks. execution time estimates. In contrast, this work models task
The derived joint probability distribution that defines the execution times as random variables where we assume the
stochastic robustness metric corresponds to the probability existence of an empirical distribution.

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.
9. Conclusions son, M. D. Theys, and B. Yao. A comparison of eleven sta-
tic heuristics for mapping a class of independent tasks onto
Our results suggest that there is an inverse relationship heterogeneous distributed computing systems. Journal of
Parallel and Distributed Computing, 61(6):810–837, June
between the dynamic SRM value and the performance ob-
2001.
jective of the problem studied. In reviewing our results, [8] O. H. Ibarra and C. E. Kim. Heuristic algorithms for
we explored the general relationship between the dynamic scheduling independent tasks on non-identical processors.
SRM value and the performance objective by analyzing the Journal of the ACM, 24(2):280–289, Apr. 1977.
fit of a Bayesian regression model to our results. The rel- [9] A. Leon-Garcia. Probability & Random Processes for Elec-
atively good fit of the regression model to the combined trical Engineering. Addison Wesley, Reading, MA, 1989.
data of the three heuristics is strong evidence of the relation- [10] M. Maheswaran, S. Ali, H. J. Siegel, D. Hensgen, and R. F.
ship between the dynamic SRM value and the stated perfor- Freund. Dynamic mapping of a class of independent tasks
mance objective. Our results appear to demonstrate that the onto heterogeneous computing systems. Journal of Parallel
and Distributed Computing, 59(2):107–121, Nov. 1999.
dynamic SRM value can simplify the evaluation of resource
[11] A. M. Mehta, J. Smith, H. J. Siegel, A. A. Maciejewski,
allocation heuristics in a dynamic environment. That is, the A. Jayaseelan, and B. Ye. Dynamic resource allocation
methodology for determining the dynamic SRM value for heuristics that manage tradeoff between makespan and ro-
a heuristic reduces the number of simulations required to bustness. Journal of Supercomputing, Special Issue on Grid
demonstrate the superiority of one heuristic over another in Technology, accepted, to appear.
a dynamic resource allocation environment. [12] A. J. Page and T. J. Naughton. Dynamic task scheduling
Using these results we compared the performance of using genetic algorithms for heterogeneous distributed com-
three different heuristics taken from the literature and ap- puting. In Proceedings of the 19th International Parallel
plied to a stochastic dynamic environment. From this com- and Distributed Processing Symposium, Apr. 2005.
parison, the Negotiation heuristic showed promising results [13] K. Ross and N. Bambos. Local search scheduling algorithms
for maximal throughput in packet switches. In Proceedings
in this environment. Given the apparent relationship be-
of the 23rd Conference of the IEEE Communications Society
tween the dynamic SRM value and the stated performance (INFOCOM 2004), 2004.
objective, a valuable extension of this work would be the de- [14] V. Shestak, J. Smith, H. J. Siegel, and A. A. Maciejewski. A
velopment of resource allocation heuristics that incorporate stochastic approach to measuring the robustness of resource
the dynamic stochastic robustness metric during a resource allocations in distributed systems. In 2006 International
allocation. Conference on Patallel Processing (ICPP 2006), pages 6–
12, Aug. 2006.
[15] V. Shestak, J. Smith, R. Umland, J. Hale, P. Moranville,
References A. A. Maciejewski, and H. J. Siegel. Greedy approaches
to static stochastic robust resource allocation for periodic
[1] S. Ali, A. A. Maciejewski, H. J. Siegel, and J.-K. Kim. Mea- sensor driven systems. In Proceedings of The 2006 Inter-
suring the robustness of a resource allocation. IEEE Trans- national Conference on Parallel and Distributed Process-
actions on Parallel and Distributed Systems, 15(7):630–641, ing Techniques and Applications, volume 1, pages 3–9, June
July 2004. 2006.
[2] M. Arlitt and T. Jin. Workload characterization of the 1998 [16] R. Vaessens, E. Aarts, and J. Lenstra. Job-shop
world cup web site. Technical Report HPL-1999-35R1, scheduling by local search. Technical Report
Hewlett Packard Corporation, CA, Sept. 1999. http://citeseer.ist.psu.edu/vaessens94job.html, Eindhoven
[3] M. Arlitt and T. Jin. A workload characterization study of University of Technology, Eindhoven, The Netherlands,
the 1998 world cup web site. IEEE Network, 14:30–37, 1994 (accessed May 2006).
May/June 2000. [17] L. Wasserman. All of Statistics: A Concise Course in Sta-
[4] L. Barbulescu, A. E. Howe, L. D. Whitley, and M. Roberts. tistical Inference. Springer Science+Business Media, New
Trading places: How to schedule more in a multi-resource York, NY, 2005.
oversubscribed scheduling problem. In Proceedings of [18] M. Wu and W. Shu. Segmented min-min: A static mapping
the International Conference on Automated Planning and algorithm for meta-tasks on heterogeneous computing sys-
Scheduling (ICAPS 2004), pages 227–234, 2004. tems. In Proceedings of the 9th IEEE Heterogeneous Com-
[5] L. Barbulescu, L. D. Whitley, and A. E. Howe. Leap be- puting Workshop, pages 375–385, Mar. 2000.
fore you look: An effective strategy in an oversubscribed
scheduling problem. In Proceedings of the 19th National
Conference on Artifial Intelligence, pages 143–148, 2004.
[6] C. M. Bishop. Pattern Recognition and Machine Learning.
Springer Science+Business Media, LLC, 233 Spring Street,
New York, NY 10013, USA, first edition, 2006.
[7] T. D. Braun, H. J. Siegel, N. Beck, L. Bölöni, R. F. Freund,
D. Hensgen, M. Maheswaran, A. I. Reuther, J. P. Robert-

Authorized licensed use limited to: UNIVERSITY OF SYDNEY. Downloaded on April 05,2010 at 23:16:01 EDT from IEEE Xplore. Restrictions apply.