2017 3rd International Conference on Information Management

Container Oriented Job Scheduling Using Linear Programming Model

Dong Zhang, Bing-Heng Yan, Zhen Feng* Chi Zhang, Yu-Xin Wang
State Key Laboratory of High-end Server & Storage School of Computer Science and Technology
Technology Dalian University of Technology
Beijing, China Dalian, China
Inspur Electronic Information Industry Co., Ltd. e-mail: [email protected], [email protected]
Jinan, China
e-mail: {zhangdong, yanbh, fengzh}@inspur.com

Abstract—With Docker gaining widespread popularity in the the container hosts, and the energy costs of the container
recent years, the container scheduler becomes a crucial role for hosts. The illustrated scenario settings and the potential cost
the exploding containerized applications and services. In this considerations lead towards our proposed methodology, that
work, the container host energy conservation, the container minimizes the total costs on the premise that computing
image pulling costs from the image registry to the container capacity requirements are matched for both the clients and
hosts and the workload network transition costs from the servers.
clients to the container hosts are evaluated in combination. By This paper reveals itself in three points, firstly, it presents
modeling the scheduling problem as an integer linear a novel algorithm for the typical container-based applications;
programming, an effective and adaptive scheduler is proposed.
secondly, the container image pulling costs, the workload
Impressive cost savings were achieved compared to Docker
Swarm scheduler. Moreover, it can be easily integrated into
network transition costs from the clients to the container
the open-source container orchestration frameworks. hosts and the host energy costs are considered before making
the image pulling and container-launching decisions; and
Keywords—container; docker; scheduling; integer linear thirdly, to our best knowledge, this is the first work to
programming; energy conservation; container image registry employ the integer linear programming model to solve the
native container scheduling problem.
I. INTRODUCTION The remainder of this paper is organized as follows.
Since the rise of Docker, container, the technology that Section 2 reviews the related works, which inspired our
has evolved for decades, has gained widespread popularity in proposal; section 3 describes the typical container oriented
recent years. The industry has seen the superior advantages application scenario and details the linear programming
inherent in Docker, which remodels the “build, ship and run” based solution; section 4 presents the experimental results
procedure in the software engineering. Remarkably, the and gives a further analysis; section 5 concludes the paper
lightweight and high-performance features pave the way to and introduces our future work.
more opportunities for Docker in novel applications and II. RELATED WORKS
cloud-native services.
Docker has established a de facto standard for the The scheduling methodologies in the cloud environment
container engine, however, the job scheduling and have been studies for years since the coming of cloud
orchestration in the containerized environments is still an computing era. In the early period, the researches on
open issue. The open-source community proposed container- scheduling primarily focused on the collaboration between
native orchestration frameworks such as Kubernetes, Mesos the virtual machines and the physical hosts, for the purpose
Marathon and Docker Swarm; the academic circles put of performance-oriented load balance or energy-oriented
forward creative scheduling algorithms to pursue load- workload consolidation. Some works investigated the on-
balance or energy-efficient optimal solutions. The joint demand resource (processor cores, main memory, I/O
efforts from both the communities make the container devices, etc.) provisions to virtual machines, while others
technology steps further to the practical and commercial moved to a higher level of virtual machine migrations among
usages. physical servers. To list some notable literatures, Srikantaiah
In this paper, the authors consider a typical scenario et al. 2008 studied the inter-relationships between energy
where the concurrent jobs are scheduled in the context of consumption, resource utilization, and performance of
containerized workload dispatch. Requests are initiated from consolidated workloads [1]. They observed that the optimal
the client points for computing capacities, the computing operating points for the energy performance trade-offs for
servers pull the related container images from the image consolidation exist, and modeled the consolidation problem
registry and launch containers to handle the received requests. as a modified bin packing problem. Zhao et al. 2009
Mainly two kinds of costs are observed in the scenario, the introduced the MEmory Balancer (MEB), which
container image transition costs from the image registry to dynamically monitors the memory usage of each virtual

978-1-5090-6306-2/17/$31.00 ©2017 IEEE 174

machine, predicts its memory needs and periodically brand-new technology, it has taken advantages of all the
reallocates host memory [2]. The memory predictor is based previous accumulations, especially the technologies of
on the building up of the LRU histogram, and monitors the chroot, cgroups and union filesystem. Just as Ben Golub, the
swap space usage of each guest OS. Padala et al. 2009 CEO of Docker.Inc said in the Docker Con 2015: “Thank
presented a feedback control system which consists of an you to the giants. We know we are standing on your
online locally linear model estimator to capture the dynamic shoulders.”
relationship between application-level performance and Although Docker is still facing challenges from
resource allocations (i.e. CPU, disk) and a MIMO resource comparable or emerging technologies such as CoreOS rkt,
controller to determine appropriate allocations of multiple unikernel or Intel clear container, Docker has established a
resources [3]. Urgaonkar et al. 2010 employed the queueing de facto standard for the container engine. Industrial and
information available in the system instead of the predictors academic communities begin to take more efforts to the
to make online control decisions, which is quite adaptive to higher-level container scheduling and orchestration systems.
unpredictable changes in the workloads and does not require The Docker Swarm scheduler features three strategies:
the estimation and prediction of its statistics [4]. Morin et al. spread, binpack and random. The spread strategy spreads
2011 also modeled the workload (VM) placement problem containers thinly over many machines, the binpack strategy
as an instance of the multi-dimensional bin-packing problem packs containers into machines as few as possible, the
and designed an ant colony optimization inspired algorism to random strategy works as its name suggests. Kubernetes
compute the placement dynamically [5]. Beloglazov et al. divides scheduling into two stages, the predicates stage and
2011 proposed an energy-aware resource allocation heuristic the priorities stage. The predicates stage finds all the
approach in two steps, firstly, VMs are allocated using available nodes for the request pods by judging the ports,
modified best fit decreasing (MBFD) algorithms; secondly, resource fits, disk conflicts and hostnames. The priorities
optimization of the current VM allocation is conducted by stage chooses the best fit node from the available set based
the minimization of migrations (MM) policy and the highest on the strategies such as the least resource utilization, service
potential growth (HPG) policy [6]. Sun et al. 2013 put spreading or complete equality. Generally speaking, the
forward a two-tiered resource (CPU, memory and disk) strategies from the industrial and the open-source
allocation mechanism to provide on-demand capacities to the communities are efficient, but too simple to handle
concurrent applications [7]. Application performance as well complicated application scenarios. Meanwhile, academic
as CPU utilization improvement was observed with a set of researchers are investigating more adaptive scheduling
proposed algorithms. Yuan et al. 2014 presents a particle strategies. Monsalve et al. 2015 introduced the concept of
swarm optimization based virtual machines resources time slicing for the dynamic CPU resource allocation in the
scheduling algorithm, which takes processing elements containerized cloud environments [11]. Only a single
computing capacity and the computational complexity into container with all its threads and processes is scheduled at
considerations [8]. Zhang et al. 2015 proposed an inverse any time on shared resources.
clustering-based job placement method to balance the In this paper, an efficient and adaptive container
workloads among the nodes and meet the resource scheduling algorism is put forward. The authors have
requirements for various jobs [9]. borrowed merits from the VM scheduling methodologies,
Virtual machine has been the dominant virtualization and the container image pulling cost, workload network
form for years, however, with the birth of Docker, container transition costs and the energy conservation are included into
promptly gains its power as a competitive cloud service form. consideration. Moreover, the proposed algorism is designed
Actually, before Docker seized imaginations and opened the for the typical deployment scenario of containers, and it can
doors to larger container usage, the container technology has be easily integrated into existing container orchestration
walked through a long evolutionary process (Osnat, 2016 frameworks.
[10]). The concept of containers first emerged in the year
1979, in the Unix V7 system, the chroot system call was III. METHODOLOGY
introduced to segregate different processes with independent A typical containerized application scenario is illustrated
root directories. Two decades later to 2000, the Jails was in Fig. 1. Here the resources (i.e., processor cores, memories)
introduced to FreeBSD, the “jails” can be seen as smaller requested by the client points, and the resources available in
systems of FreeBSD, with the ability to assign an IP address the container host, are measured by units. In the data center,
for each system and configuration. Then in 2001 a Jails-like we have n container hosts, for the k-th host, it has the
VServer patch for the Linux kernel was raised, which could computing capacities of Capacity(k) units. In the upside of
partition resources (file systems, network addresses, memory) the figure, we have m client points. The client points can be
on a computer system. From then on, the container desktops, where the requests are initiated via the web service,
technology accelerated the pace and a set of novel models the clients can also be sensors, in the context of internet of
and features were released, such as the Oracle Solaris things (IoT). For the t-th client, suppose it requests
Containers in 2004, Open VZ in 2005, Google Control workloads of Workload(t) units. There exists network
Groups (cgroups) in 2006, LXC (LinuX Containers) in 2008, transition cost (e.g. latency, bandwidth, maximum
CloudFoundry Warden in 2011 and the Google LMCTFY connections) between the clients and hosts, we denote the
(Let Me Contain That for You) in 2013. Eventually in the cost between the k-th host and t-th client is cost(k, t) per unit.
year 2013, Docker exploded in popularity. Docker is not a Besides, to launch the containers on the hosts, the hosts must

175
pull the responding containers from the image registry. The The constraint (1) ensures that all the workloads from the
image is composed of multiple layer files, if all the required client k should be processed on the hosts.
layer files are already stored in the host, the cost is zero; To meet the criterion (2), the constraint (2) is set as:
however, if only parts of the layer files are ready, then the
missing layers should be downloaded from the image σ௠
௧ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൑ ƒ’ƒ…‹–›ሺሻ (2)
registry, which leads to extra costs (illustrated in Fig. 2).
The constraint (2) ensures that all the workloads running
on the k-th host should not exceed its capacity.
The costs happened in the workload network transition is
defined as:
௡
݂ଵ ሺ‫ݔ‬ሻ ൌ σ௠
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴ܿݏ݊ܽݎݐ‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ (3)

The correlations between the server utilization and the

electric power consumption have been widely investigated,
in this work, the authorss employ Morin et al. 2011’s model
to estimate the power consumption of the container hosts [5]:

Figure 1. Containerized application scenario ܲ௞ ሺ—ሻ ൌ ൫ܲ௞ǡ௠௔௫ െ ܲ௞ǡ௜ௗ௟௘ ൯ ൈ — ൅ ܲ௞ǡ௜ௗ௟௘ (4)

With ܲ௞ǡ௜ௗ௟௘ and ܲ௞ǡ௠௔௫ being the average power values

for the k-th host, when the system is idle and fully utilized,
respectively. In our scenario, the idle state means that no
container is running in the host, while the fully-utilized state
means the containers running on the hosts have reached the
host’s capacity.
The character u in function (4) means the utilization rate
and it can be generated by:

σ೘
೟సభ ௫ሺ௞ǡ௧ሻ
—ൌ (5)
஼௔௣௔௖௜௧௬ሺ௞ሻ

Figure 2. Multiple-layer image pulling from the registry

Then the energy cost is summarized as:
In the production environment, the t-th client initiates a
task request with Workload(t). The workloads are divided σ೘
೟సభ ௫ሺ௞ǡ௧ሻ
into multiple batches and dispatched to one or more ݂ଶ ሺ‫ݔ‬ሻ ൌ σ௡௞ୀଵ ቆ൫ܲ௞ǡ௠௔௫ െ ܲ௞ǡ௜ௗ௟௘ ൯ ൈ ൅ ܲ௞ǡ௜ௗ௟௘ ቇ
஼௔௣௔௖௜௧௬ሺ௞ሻ
container hosts. The container hosts pull the container (6)
images from the image registry and launch containers to
execute the workloads. The key issue in the whole workflow Without loss of generality, we suppose the task from
is the designing of a scheduler to dispatch the workloads to client t requires the image t to run on the container host, then
container hosts, to meet the following criterions: the image pulling cost is defined as:
(1) All the workloads from clients are containerized
and processed in the hosts. ݂ଷ ሺ‫ݕ‬ሻ ൌ σ௠ ௡
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴݈݈ܿݑ݌‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻ (7)
(2) The workloads scheduled to a specific host cannot
exceed the host’s computing capacity. where y(k, t) = 1 if the node k pulls the image t from the
For the purpose of global optimization, the electric power image registry, y(k, t) = 0 if not. In another word, y(k, t) = 1
consumed by all the hosts, the image pulling costs and the if x(k, t) > 0, y(k, t) = 0 if x(k, t) = 0. To meet this constraint,
costs during the workload network transition should be we set
minimized.
To meet the criterion (1), denote x(k, t) as the workloads ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൑ ‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻ ή ƒ’ƒ…‹–›ሺ݇ሻ (8)
from the t-th client, which are assigned to the k-th host. Then
the constraint (1) is set as: In function (8), if y(k, t) = 0, then x(k, t) is strictly
confined to 0.
σ௡௞ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൌ ‘”Ž‘ƒ†ሺ‫ݐ‬ሻ (1) Take functions (3), (6) and (7) into consideration, the
resulting cost function is introduced as:
where x(k, t) is a natural number .

176
݂ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ ݂ଵ ሺ‫ݔ‬ሻ ൅ ߙ݂ଶ ሺ‫ݔ‬ሻ ൅  ߚ݂ଷ ሺ‫ݕ‬ሻ ൌ
௡ ௡
σ௠
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴ܿݏ݊ܽݎݐ‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൅ ߙ σ௞ୀଵ ቆ൫ܲ௞ǡ௠௔௫ െ

σ೘
೟సభ ௫ሺ௞ǡ௧ሻ
ܲ௞ǡ௜ௗ௟௘ ൯ ൈ ൅
஼௔௣௔௖௜௧௬ሺ௞ሻ

௡
ܲ௞ǡ௜ௗ௟௘ ቇ ൅ ߚ σ௠
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴݈݈ܿݑ݌‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻ

(9)
where the weighting factors ߙ and ߚ are introduced to
balance the three costs.
This work aims to dispatch the workloads from all the
client points to the container hosts to achieve the minimal
costs of the energy consumption and the network transition. Figure 4. Image pulling cost from the registry
This is reflected in the objective function (10):
To gain a first sight of the three scheduling methods, a
‹  ݂ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ ݂ଵ ሺ‫ݔ‬ሻ ൅ ߙ݂ଶ ሺ‫ݔ‬ሻ ൅  ߚ݂ଷ ሺ‫ݔ‬ሻ (10) small-scale test was performed initially, with two clients and
four container servers. The client workloads and host
capacities were generated in random, but were limited
Subject to the following constraints: between 10 and 20. The network transition costs between
clients and hosts were also random numbers between 1 and 4.
σ௡௞ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൌ ‘”Ž‘ƒ†ሺ‫ݐ‬ሻ (11) The image pulling costs were set between 0 and 3, based on
the layers required to be downloaded from the registry.
σ௠ (12) Without loss of generality, the weighting factors α and β in
௧ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൑ ƒ’ƒ…‹–›ሺሻ
the function (7) were set to 1. Following Morin et al. 2011’s
‫ ݔ‬൒ Ͳ and ‫ݔ‬is integer (13) practice, the idle and fully utilized power values were fixed
to 171 and 218 Watt, on a Dell PowerEdge 1950 server with
‫ ݕ‬is binary (14) 4 GB of RAM and two Intel Xeon 5148 2.33 GHz CPUs [5].
The randomly generated host capacities, client workloads
Thereby, an integer linear programming representation is and workload network transition costs are shown in Fig. 3.
introduced by the objective function (10) and the constrains The image pulling costs from the registry are shown in Fig. 4.
(11), (12) and (13). Then the optimal solutions to the Algorithm 1: Liner Programming based Scheduling
‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻǡ ݇ ൌ ͳǡʹǡ ǥ ݊ǡ ‫ ݐ‬ൌ ͳǡʹǡ ǥ ǡ ݉ can be reached. Algorithm for Containers
The complete procedure is presented in Algorithm 1.
Inputs:
IV. EXPERIMENTAL RESULTS AND ANALYSIS
Container host capacities: ƒ’ƒ…‹–›ሺሻǡ ݇ ൌ ͳǡʹǡ ǥ ǡ ݊
In this section, the authors conducted series of
experiments to compare the total costs of the proposed Client requested workloads: ‘”Ž‘ƒ†ሺ‫ݐ‬ሻǡ ‫ ݐ‬ൌ ͳǡʹǡ ǥ ݉
method and the binpack and random methods employed in
Network transition costs per workload unit:
the Docker Swarm orchestration framework. For the sake of
–”ƒ•̴ܿ‫ݐݏ݋‬ሺ݇ǡ ‫ݐ‬ሻ
simplification and to expel other factors, the three scheduling
algorithms (LP in our proposal, Binpack and Random in Container image pulling costs between image registry and
Docker Swarm) are implemented via MATLAB in a hosts: ’—ŽŽ̴ܿ‫ݐݏ݋‬ሺ݇ǡ ‫ݐ‬ሻ
controlled experimental environment.
Outputs:
Workloads placed on the host k for the client t: ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ
Procedure:
# All the workloads from the client k should be
processed on the hosts:
௡

෍ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൌ ‘”Ž‘ƒ†ሺ‫ݐ‬ሻ

௞ୀଵ
# All the workloads performed on the host k should not
Figure 3. A small-scale containerized scenario demo exceed its capacity:

177
 ௠

෍ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ ൑ ƒ’ƒ…‹–›ሺ݇ሻ

௧ୀଵ
# The network transition costs between all the clients
and hosts:
௠ ௡

݂ଵ ሺ‫ݔ‬ሻ ൌ ෍ ෍ ܿ‫ݐݏ݋‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ

௧ୀଵ ௞ୀଵ
# The energy consumption costs for all the container
hosts:
Figure 7. Random scheduler with total cost 167.9
௡
σ௠௧ୀଵ ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ
݂ଶ ሺ‫ݔ‬ሻ ൌ ෍ ൭൫ܲ௞ǡ௠௔௫ െ ܲ௞ǡ௜ௗ௟௘ ൯ ൈ To simulate the large-scale concurrent scenarios, we
‫ݕݐ݅ܿܽ݌ܽܥ‬ሺ݇ሻ scaled out the quantities of hosts and clients, from 20 hosts
௞ୀଵ
with 30 clients, to 50 hosts with 100 clients, and 100 hosts
൅ ܲ௞ǡ௜ௗ௟௘ ൱ with 200 clients. The client workloads, host capacities,
network transition costs and the image pulling costs were
# The pulling cost for image k to host t: randomly generated, between 10 and 20, 50 and 100, 1 and 4,
0 and 3 respectively. The tests were performed for five
݂ଷ ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ rounds under the same scales, to observe the stability of the
σ௠ ௡ ௠ ௡
௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴݈݈ܿݑ݌‬ሺ݇ǡ ‫ݐ‬ሻ σ௧ୀଵ σ௞ୀଵ ‫ݐݏ݋̴݈݈ܿݑ݌‬ሺ݇ǡ ‫ݐ‬ሻ ή schedulers. The results are recorded in Tables 1 to 3 and an
‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻand y is binary intuitive comparison is shown in Fig. 8.
It can be seen that the Binpack and Random schedulers
# To minimize the total costs: got comparable total costs, while the LP scheduler achieved
‹ ݂ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ ݂ଵ ሺ‫ݔ‬ሻ ൅ ߙ݂ଶ ሺ‫ݔ‬ሻ ൅  ߚ݂ଷ ሺ‫ݕ‬ሻ impressively results and the total costs were reduced by more
# Employ the integer linear programming method to than 50% in almost all the settings.
pursue the optimal solution to ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ and ‫ݕ‬ሺ݇ǡ ‫ݐ‬ሻ To observe the growth curves of the total costs with the
growing scales, we performed a group of experiments that
Fig. 5 to Fig. 7 demonstrate the outcomes with three the client number increased from 100 to 200, and the
scheduling algorithms. It can be seen that all the schedulers container host number increased from 50 to 200. The curves
behaved as expected. For the total costs, LP scheduler in our are illustrated in Fig. 9. There are heavy overlaps between
proposal achieved the lowest cost, the Random scheduler got the curves: the curves of the LP method coincide with each
the total cost of 167.8, for our surprise, the Binpack other although they used different quantities of hosts, the
scheduler resulted in the highest cost. curves of Binpack and Random wound together and it is hard
to distinguish them. However, we can still tell the clear story
that the costs by the LP method are far lower than other
methods. Moreover, with the increase of the client number,
the costs of the LP method increase by about 2000 from
2300 to 4300, while the costs of the other two increase by
about 5000 from 5000 to 10000. Therefore, the LP method
enjoys a far lower growth rates.

TABLE I. TOTAL COSTS UNDER 20 HOSTS AND 30 CLIENTS

LP Binpack Random
Figure 5. LP scheduler with total cost 137.01
Round 1 705.3 1630.2 1560.7
Round 2 698.1 1447.9 1624.3
Round 3 692.7 1543.7 1831.0
Round 4 631.7 1237.5 1387.3
Round 5 718.5 1516.5 1671.4

TABLE II. TOTAL COSTS UNDER 50 HOSTS AND 100 CLIENTS

LP Binpack Random
Round 1 2285.3 4904.5 5536.4
Round 2 2214.7 4844.7 4993.4
Round 3 2231.1 5047.5 4820.2
Round 4 2203.8 4640.2 4965.8
Figure 6. Binpack scheduler with total cost 185.3 Round 5 2268.5 4648.7 5097.0

178
 TABLE III. TOTAL COSTS UNDER 100 HOSTS AND 200 CLIENTS TABLE IV. ENERGY CONSUMPTIONS UNDER 100 HOSTS AND 200
CLIENTS
LP Binpack Random
Round 1 4398.3 9488.3 9844.8 LP Binpack Random
Round 2 4421.8 9828.4 9811.3 Round 1 1587.4255 1994.5625 2053.9633
Round 3 4592.4 9775.1 10509.9 Round 2 1544.9884 1833.5165 1991.5988
Round 4 4366.9 9743.8 9849.2 Round 3 1524.3855 1918.8469 1964.6325
Round 5 4334.9 9680.2 9796.3 Round 4 1542.1529 1884.4235 1952.5552
Round 5 1500.0833 1890.6129 1851.3717

TABLE V. TRANSITION COSTS UNDER 100 HOSTS AND 200 CLIENTS

LP Binpack Random
Round 1 3011 7671 7441
Round 2 3009 7615 7815
Round 3 2921 7374 7316
Round 4 2946 7222 7419
Round 5 2948 6899 7403

From Table 4, it can be observed that the LP scheduler

still got the best performance. The Binpack scheduler
Figure 8. Comparison on the total costs under varied scales outperformed the Random scheduler in most cases, it is
reasonable since that Binpack is targeted for the energy
In practice, the users may only consider the energy conservation. In Table 5, where only the transition costs
consumptions or network transition costs alone. Therefore, were covered, we can observe remarkably large performance
we further extended the experiments by modifying the margins between the proposed LP scheduler and other
objective function as: schedulers.

σ೘
೟సభ ௫ሺ௞ǡ௧ሻ
V. CONCLUSION
‹  ݂ሺ‫ݔ‬ሻ ൌ σ௡௞ୀଵ ቆ൫ܲ௞ǡ௠௔௫ െ ܲ௞ǡ௜ௗ௟௘ ൯ ൈ ൅
஼௔௣௔௖௜௧௬ሺ௞ሻ In this paper, we put forward an Integer Linear
Programming-based solution to the containerized application
ܲ௞ǡ௜ௗ௟௘ ቇ (15) scheduling in the cloud environment. The proposed method
has achieved impressive performance in the energy
In cases where only the energy consumption matters, or: conservation, registry image pulling costs and the network
transition costs savings. It makes a step further to an adaptive
‹  ݂ሺ‫ݔ‬ሻ ൌ σ௠ ௡
௧ୀଵ σ௞ୀଵ ܿ‫ݐݏ݋‬ሺ݇ǡ ‫ݐ‬ሻ ή ‫ݔ‬ሺ݇ǡ ‫ݐ‬ሻ (16) and effective scheduler for the native container as a service
scenarios.
In cases where only the network transition cost counts. We have planned our future work in two fields. Firstly, in
The experimental results for the objective functions (15) the current scheme, the workload is represented by units, we
and (16) are shown in Tables 4 and 5 respectively, under the intend to extend the workload into multiple dimensions, to fit
100 hosts and 200 clients setting. for the multiple resource requests, leading to that the
workload will become a vector with CPU and memory
dimensions. Secondly, the scheduler prototype is currently
written in MATLAB, we will reconstruct it into the GO or
Python language, to make the proposed scheduler as a plugin
for the open-source container orchestration frameworks. The
scheduler prototype source code will be available at GitHub.
ACKNOWLEDGEMENT
This work is supported by the National High Technology
Research and Development Program (863 Program) of China
(NO. 2015AA015301).
REFERENCES
[1] S. Srikantaiah, A. Kansal, F. Zhao, “Energy aware consolidation for
cloud computing,” Proceedings of the 2008 conference on Power
aware computing and systems, San Diego, California, 2008, pp: 10-
10
[2] W. M. Zhao, Z. L. Wang, “Dynamic memory balancing for virtual
machines,” Proceedings of the 2009 ACM SIGPLAN/SIGOPS
international conference on Virtual execution environments,
Washington, DC, USA, 2009.
Figure 9. Cost growth curves with growing scales

179
[3] P. Padala , K. Y. Hou, K. G. Shin, X. Z., M. Uysal , Z. K. Wang , S. [8] H. Yuan, C. B. Li , M. K. Du, “Virtual Machine Resources
Singhal , A. Merchant, “Automated Control of Multiple Virtualized Scheduling Based on Improved Particle Swarm Optimization in
Resources,” Proceeding of the 4th ACM European Conference on Cloud Computing,” Journal of Software, 2014, 9(3).
Computer Systems, Nuremberg, Germany, 2009, pp: 13-26. [9] D. Zhang, B. H. Yan, Z. Feng, K. Y. Qi, Z. Y. Su, “Inverse
[4] R. Urgaonkar, U. C. Kozat, K. Igarashi and M. J. Neely, "Dynamic Clustering-based Job Placement Method for Efficient Big Data
resource allocation and power management in virtualized data Analysis,” Proceedings of the 2015 IEEE 17th International
centers," 2010 IEEE Network Operations and Management Conference on High Performance Computing and Communications,
Symposium - NOMS 2010, Osaka, 2010, pp. 479-486. 2015 IEEE 7th International Symposium on Cyberspace Safety and
[5] E. Feller, L. Rilling and C. Morin, "Energy-Aware Ant Colony Based Security, and 2015 IEEE 12th International Conf on Embedded
Workload Placement in Clouds," 2011 IEEE/ACM 12th International Software and Systems, Washington, DC, USA, 2015, pp. 1796-1799.
Conference on Grid Computing, Lyon, 2011, pp. 26-33. [10] R. Osnat, “A Brief History of Containers: From 1970s chroot to
[6] A. Beloglazov, J. Abawajy and R. Buyya, “Energy-aware resource Docker 2016,” http://blog.aquasec.com/a-brief-history-of-containers-
allocation heuristics for efficient management of data centers for from-1970s-chroot-to-docker-2016
Cloud computing,” Future Generation Computer Systems, 2012, vol. [11] J. Monsalve, A. Landwehr and M. Taufer, "Dynamic CPU Resource
28(5), pp: 755-768 Allocation in Containerized Cloud Environments," 2015 IEEE
[7] Y. Song, Y. Sun and W. Shi, "A Two-Tiered On-Demand Resource International Conference on Cluster Computing, Chicago, IL, 2015,
Allocation Mechanism for VM-Based Data Centers," in IEEE pp. 535-536.
Transactions on Services Computing, vol. 6, no. 1, pp. 116-129, First
Quarter 2013.

180