ContainerLeaks: Emerging Security Threats of

Information Leakages in Container Clouds

Xing Gao1,2 , Zhongshu Gu3 , Mehmet Kayaalp3 , Dimitrios Pendarakis3 , Haining Wang1
1 University
of Delaware, 2 College of William and Mary, 3 IBM T.J. Watson Research Center
{xgao, hnw}@udel.edu, {zgu, mkayaal, dimitris}@us.ibm.com
Abstract—Container technology provides a lightweight oper- between a container and a host, and thus they might expose
ating system level virtual hosting environment. Its emergence system-wide information to containerized applications. Some
profoundly changes the development and deployment paradigms subsystems are considered to be of low priority for container
of multi-tier distributed applications. However, due to the incom- adaptations. The rest are facing implementation difficulties
plete implementation of system resource isolation mechanisms in for transforming their code base, and their maintainers are
the Linux kernel, some security concerns still exist for multiple
containers sharing an operating system kernel on a multi-tenancy
reluctant to accept drastic changes. In order to close these
container cloud service. In this paper, we first present the loopholes, current container runtime software and container
information leakage channels we discovered that are accessible cloud providers typically leverage access control policies to
within the containers. Such channels expose a spectrum of mask the user-kernel interfaces of these container-oblivious
system-wide host information to the containers without proper subsystems. However, such manual and temporary fixes could
resource partitioning. By exploiting such leaked host information, only cover a small fraction of the exposed attack surfaces.
it becomes much easier for malicious adversaries (acting as
tenants in the container clouds) to launch advanced attacks that In this paper, we systematically explore and identify the
might impact the reliability of cloud services. Additionally, we in-container leakage channels that may accidentally expose
discuss the root causes of the containers’ information leakages information of host OSes and co-resident containers. Such
and propose a two-stage defense approach. As demonstrated information leakages include host-system state information (e.g.,
in the evaluation, our solution is effective and incurs trivial power consumption, performance data, global kernel data, and
performance overhead.
asynchronous kernel events) and individual process execution
information (e.g., process scheduling, cgroups, and process
I. I NTRODUCTION running status). The distinguishing characteristic information
Cloud computing has been widely adopted to consolidate exposed at specific timings could help uniquely identify
computing resources. Multi-tenancy is the enabling feature a physical machine. Furthermore, a malicious tenant may
of cloud computing that allows computation instances from optimize attack strategies and maximize attack effects by
different tenants running on a same physical server. Among acquiring the system-wide knowledge in advance. We discover
different types of cloud services, the multi-tenancy container these leakage channels in our local testbed on Docker and
cloud has recently emerged as a lightweight alternative to LinuX Container (LXC) and verify their (partial) existence on
conventional virtual machine (VM) based cloud infrastructures. five public commercial multi-tenancy container cloud services.
Container is an operating system (OS) level virtualization
technology with multiple building blocks in the Linux kernel, In order to reveal the security risks of these leakage
including resource isolation/control techniques (e.g., namespace channels, we design an advanced attack, denoted as synergistic
and cgroup) and security mechanisms (e.g., Capabilities, power attack, to exploit the seemingly innocuous information
SELinux, AppArmor, and seccomp). By avoiding the overhead leaked through these channels. We demonstrate that such
of additional abstraction layers, containers are able to achieve information exposure could greatly amplify the attack effects,
near-native performance and outperform VM-based systems reduce the attack costs, and simplify the attack orchestration.
in almost all aspects [2], [14], [30]. In addition, the advent Power attacks have proved to be real threats to existing data
of container management and orchestration systems, such as centers [26], [43]. With no information of the running status
Docker and Kubernetes, have profoundly changed the ecosystem of underlying cloud infrastructures, existing power attacks can
of building, shipping, and deploying multi-tier distributed only launch power-intensive workloads blindly to generate
applications in the cloud. power spikes, with the hope that high spikes could trip branch
circuit breakers to cause power outages. Such attacks could
Despite the success of container services, there always exist be costly and ineffective. However, by learning the system-
security and privacy concerns for running multiple containers, wide status information, attackers can (1) pick the best timing
presumably belonging to different tenants, on the same OS to launch an attack, i.e., superimpose the power-intensive
kernel. To support multi-tenancy on container clouds, we workload on the existing power spikes triggered by benign
have observed on-going efforts in the Linux kernel to enforce workloads, and (2) synchronize multiple power attacks on the
cross-container isolation and de-privilege user-level containers. same physical machine/rack by detecting proximity-residence of
Existing container-enabling kernel features have greatly shrunk controlled containers. We conduct proof-of-concept experiments
the attack surface exposed to container tenants and could on one real-world container cloud service and quantitatively
restrain most existing malicious attacks. However, not all sub- demonstrate that our attack is able to yield higher power spikes
systems of the Linux kernel can distinguish execution contexts at a lower cost.
 We further analyze in depth the root causes of these leakage performance, enhancing developer efficiency, and facilitating
channels and find that such exposures are due to the incomplete service deployment. Here we introduce two key techniques,
coverage of container implementation in the Linux kernel. namespace and cgroup, that enable containerization on Linux.
We propose a two-stage defense mechanism to address this
problem in container clouds. In particular, to defend against 1) Namespace: The first namespace was introduced in the
the synergistic power attacks, we design and implement a Linux kernel 2.4.19. The key idea of namespace is to isolate
power-based namespace in the Linux kernel to partition power and virtualize system resources for a group of processes, which
consumption at a finer-grained (container) level. We evaluate form a container. Each process can be associated with multiple
our power-based namespace from the perspectives of accuracy, namespaces of different types. The kernel presents a customized
security, and performance overhead. Our experimental results (based on namespace types) view of system resources to each
show that our system can neutralize container-based power process. The modifications to any namespaced system resources
attacks with trivial performance overhead. are confined within the associated namespaces, thus incurring
no system-wide changes.
Overall, the major contributions of this work are summa-
rized as follows: The current kernel has seven types of namespaces: mount
(MNT) namespace, UNIX timesharing system (UTS) namespace,
• We systematically explore and identify information PID namespace, network (NET) namespace, inter-process
leakages in container cloud environments. We further communications (IPC) namespace, USER namespace, and
analyze these information leakages in depth and trace CGROUP namespace. The MNT namespace isolates a set
out their root causes. of file system mount points. In different MNT namespaces,
processes have different views of the file system hierarchy.
• We demonstrate that adversaries can exploit these The UTS namespace allows each container to have its own
identified information leakages to launch a new type of host name and domain name, and thus a container could be
advanced power attack, denoted as synergistic power treated as an independent node. The PID namespace virtualizes
attack. Attackers can optimize their attack strategies the process identifiers (pids). Each process has two pids:
and maximize their attack effects. We prove that such one pid within its PID namespace and one (globally unique)
seemingly harmless information leakages may also pid on the host. Processes in one container could only view
pose serious security threats to cloud infrastructures. processes within the same PID namespace. A NET namespace
• We design and implement a power-based namespace contains separate virtual network devices, IP addresses, ports,
in the Linux kernel to enhance resource isolation and IP routing tables. The IPC namespace isolates inter-
for containers. Our results show that the proposed process communication resources, including signals, pipes, and
system can effectively defend against container-based shared memory. The USER namespace was recently introduced
synergistic power attacks with trivial overhead. to isolate the user and group ID number spaces. It creates
a mapping between a root user inside a container to an
The rest of this paper is organized as follows. Section II in- unprivileged user on the host. Thus, a process may have full
troduces the background of container technology and describes privileges inside a user namespace, but it is de-privileged on the
power attack threats on data centers. Section III presents the host. The CGROUP namespace virtualizes the cgroup resources,
in-container leakage channels discovered by us and their leaked and each process can only have a containerized cgroup view
information. Section IV details the synergistic power attack via cgroupfs mount and the /proc/self/cgroup file.
that leverages the leaked information through these channels.
Section V presents a general two-stage defense mechanism 2) Cgroup: In the Linux kernel, cgroup (i.e., control group)
and the specific design and implementation of our power-based provides a mechanism for partitioning groups of processes
namespace in the Linux kernel. Section VI shows the evaluation (and all their children) into hierarchical groups with controlled
of our defense framework from different aspects. Section VII behaviors. Containers leverage the cgroup functionality to apply
discusses the limitations and future work. Section VIII surveys per-cgroup resource limits to each container instance, thus
related work, and we conclude in Section IX. preventing a single container from draining host resources.
Such controlled resources include CPU, memory, block IO,
network, etc. For the billing model in cloud computing, cgroup
II. BACKGROUND can also be used for assigning corresponding resources to
In this section, we briefly describe the background knowl- each container and accounting for their usage. Each cgroup
edge of three topics: internals of Linux containers, multi-tenancy subsystem provides a unified sysfs interface to simplify the
container cloud services, and existing power attacks in data cgroup operations from the user space.
centers.
B. Container Cloud
A. Linux Kernel Support for Container Technology
With these kernel features available for resource isolation
Containers depend on multiple independent Linux kernel and management, the Linux kernel can provide the lightweight
components to enforce isolation among user-space instances. virtualization functionality at the OS level. More namespace
Compared to VM-based virtualization approaches, multiple and cgroup subsystems are expected to be merged into the
containers share the same OS kernel, thus eliminating additional upstream Linux kernel in the future to enhance the container
performance overheads for starting and maintaining VMs. security. Containerization has become a popular choice for
Containers have received much attention from the industry virtual hosting in recent years with the maturity of container
and have grown rapidly in recent years for boosting application runtime software. LXC is the first complete implementation

2
of the Linux container manager built in 2008. Docker, which monitoring the power consumption, a data center can restrict the
was built upon LXC (now with libcontainer), has become power consumption of servers through a power-based feedback
the most popular container management tool in recent years. loop. At the host level, Running Average Power Limit (RAPL) is
Docker can wrap applications and their dependencies (e.g., code, a technique for monitoring and limiting the power consumption
runtime, system tools, and system libraries) into an image, thus for a single server. RAPL has been introduced by Intel since
guaranteeing that application behaviors are consistent across Sandy Bridge microarchitecture. It provides fine-grained CPU-
different platforms. level energy accounting at the microsecond level and can be
used to limit the power consumption for one package.
A large number of cloud service providers, including Ama-
zon ECS, IBM Bluemix, Microsoft Azure, and Google Compute Power capping mechanisms significantly narrow down the
Engine, have already provided container cloud services. For power attack surface, but it cannot address the problem of
multi-tenancy container cloud services, containers can either power oversubscription, which is the root cause of power
run on a bare metal physical machine or a virtual machine. outages in data centers. Although host-level power capping for
In both situations, containers from different tenants share the a single server could respond immediately to power surges,
same Linux kernel with the host OS. the power capping mechanisms at the rack or PDU level still
suffer from minute-level delays. Assuming attackers could
C. Power Attacks on Data Centers deploy power viruses into physically adjacent servers, even
if each server consumes power lower than its power capping
Power attacks have been demonstrated to be realistic threats limit, the aggregated power consumption of controlled servers
to existing cloud infrastructures [26], [43]. Considering the altogether can still exceed the power supply capacity and trip
cost of upgrading power facilities, current data centers widely the circuit breaker. We demonstrate in the following sections
adopt power oversubscription to host the maximum number that malicious container tenants can launch synergistic power
of servers within the existing power supply capabilities. The attacks by controlling the deployment of their power-intensive
safety guarantees are based on the assumption that multiple workloads and leveraging benign workloads in the background
adjacent servers have a low chance of reaching peaks of power to amplify their power attacks.
consumption simultaneously. While power oversubscription
allows deploying more servers without increasing power
capacity, the reduction of power redundancy increases the III. I NFORMATION L EAKAGES IN C ONTAINER C LOUDS
possibility of power outages, which might lead to forced As we mentioned in Section II, the Linux kernel provides a
shutdowns for servers on the same rack or on the same power multitude of supports to enforce resource isolation and control
distribution unit (PDU). Even normal workloads may generate for the container abstraction. Such kernel mechanisms are
power spikes that cause power outages. Facebook recently the enabling techniques for running containers on the multi-
reported that it prevented 18 potential power outages within tenancy cloud. Due to priority and difficulty levels, some
six months in 2016 [37]. The situation would have been worse components of the Linux kernel have not yet transformed to
if malicious adversaries intentionally drop power viruses to support containerization. We intend to systematically explore
launch power attacks [15], [16]. The consequence of a power which parts of the kernel are left uncovered, what the root
outage could be devastating, e.g., Delta Airlines encountered a causes are, and how potential adversaries can exploit them.
shutdown of a power source in its data center in August 2016,
which caused large-scale delays and cancellations of flights
A. Container Information Leakages
[8]. Recent research efforts [26], [43] have demonstrated that
it is feasible to mount power attacks on both traditional and We first conduct experiments on our local Linux machines
battery-backed data centers. with Docker and LXC containers installed. Linux provides two
types of controlled interfaces from userspace processes to the
Launching a successful power attack requires three key
kernel, system calls, and memory-based pseudo file systems.
factors: (1) gaining access to servers in the target data center by
System calls are mainly designed for user processes to request
legitimately subscribing services, (2) steadily running moderate
kernel services. The system calls have strict definitions for
workloads to increase the power consumption of servers to
their public interfaces and are typically backward compatible.
their capping limits, (3) abruptly switching to power-intensive
However, memory-based pseudo file systems are more flexible
workloads to trigger power spikes. By causing a power spike
for extending kernel functionalities (e.g., ioctl), accessing kernel
in a short time window, a circuit breaker could be tripped to
data (e.g., procfs), and adjusting kernel parameters (e.g., sysctl).
protect servers from physical damages caused by overcurrent
In addition, such pseudo file systems enable manipulating kernel
or overload.
data via normal file I/O operations. Linux has a number of
The tripping condition of a circuit breaker depends on the memory-based pseudo file systems (e.g., procfs, sysfs, devfs,
strength and duration of a power spike. In order to maximize securityfs, debugfs, etc.) that serve the different purposes of
the attack effects, adversaries need to run malicious workloads kernel operations. We are more interested in procfs and sysfs,
on a group of servers belonging to the same rack or PDU. which are by default mounted by container runtime software.
In addition, the timing of launching attacks is also critical.
If a specific set of servers (e.g., on the same rack) in a data As illustrated in the left part of Figure 1, we design a cross-
center have already run at their peak power state, the chance validation tool to automatically discover these memory-based
of launching a successful power attack will be higher [43]. pseudo files that expose host information to containers. The
key idea is to recursively explore all pseudo files under procfs
The techniques of power capping [25] have been designed and sysfs in two execution contexts, one running within an
to defend against power attacks. At the rack and PDU level, by unprivileged container and the other running on the host. We

3
 Containerized Process Host Process Local Container Multi-Tenancy
interfaces. The data format is in the form of [ifname pri-
Pseudo File System Pseudo File System Testbed Container Cloud Testbed ority]. We find that the kernel handler function hooked at
net prio.ifpriomap is not aware of the NET namespace, and
Co-residence thus it discloses all network interfaces on the physical machine
Verification
Differential to the containerized applications. To be more specific, the read
Analysis Performance
❷ ❷ Inference operation of net prio.ifpriomap is handled by the function
OS Kernel ❶ ❶
read_priomap. Tracing from this function, we find that it
invokes for_each_netdev_rcu and sets the first parameter
Kernel Data
Space Synergistic as the address of init_net. It iterates all network devices of
Non-namespaced data Power Attack the host, regardless of the NET namespace. Thus, from the view
Namespaced data
of a container, it can read the names of all network devices of
the host.
Fig. 1: The framework for information leakage detection and
cloud inspection. 2) Case study II — RAPL in containers: RAPL was recently
introduced by Intel for setting power limits for processor pack-
ages and DRAM of a single server, which can respond at the
align and reorder the files based on their file paths and then millisecond level [19]. In the container cloud, the sysfs interface
conduct pair-wise differential analysis on the contents of the of RAPL, which locates under /sys/class/powercap/intel-rapl, is
same file between these two contexts. If the system resources accessible to containers. Therefore, it is possible for container
accessed from a specific pseudo file has not been namespaced tenants to obtain the system-wide power status of the host,
in the Linux kernel, the host and container reach the same piece including the core, DRAM, and package, through this sysfs
of kernel data (as the case of · in Figure 1). Otherwise, if interface. For example, container users can read the current
properly namespaced, the container can retrieve its own private energy counter in micro joules from the pseudo file energy uj.
and customized kernel data (as the case of ¶ in Figure 1). The function handler of energy uj in the Intel RAPL Linux
Using this cross-validation tool, we can quickly identify the driver is get_energy_counter. This function retrieves the
pseudo files (and their internal kernel data structures) that may raw energy data from the RAPL MSR. As namespace has
expose system-wide host information to the container. not been implemented for the power data, the energy_raw
pointer refers to the host’s energy consumption data.
B. Leakage Channel Analysis We further investigate the information leakage problems on
We list all pseudo files that may leak host information in container cloud services that adopt the Docker/LXC container
Table I. Those leakage channels contain different aspects of host engine. We choose five commercial public multi-tenancy
information. Container users can retrieve kernel data structures container cloud services for leakage checking and present the
(e.g., /proc/modules shows the list of loaded modules), kernel results in Table I. We anonymize the names (CCi stands for
events (e.g., /proc/interrupts shows the number of interrupts ith Container Cloud) of these container cloud services before
per IRQ), and hardware information (e.g., /proc/cpuinfo and the cloud providers patch the channels. The indicates that the
/proc/meminfo show the specification of CPU and memory, channel exists in the cloud, while the # indicates the opposite.
respectively). In addition, container users are able to retrieve We find that most of the leakage channels on local machines
performance statistics data through some channels. For example, are also available in the container cloud services. Some of
containers can obtain hardware sensor data (if these sensors are them are unavailable due to the lack of support for specific
available in the physical machine), such as power consumption hardware (e.g., Intel processor before Sandy Bridge or AMD
for each package, cores, and DRAM through the RAPL sysfs processors that do not support RAPL). For cloud CC5 , we find
interface, and the temperature for each core through the that the information of some channels is different from our local
Digital Temperature Sensor (DTS) sysfs interface. Moreover, testbed, which means that the cloud vendor has customized
the usage of processors, memory, and disk I/O is also exposed some additional restrictions. For example, only the information
to containers. While leaking such information seems harmless about the cores and memory belonging to a tenant is available.
at first glance, it could be exploited by malicious adversaries to However, those channels partially leak the host information
launch attacks. More detailed discussion is given in Section IV. and could still be exploited by advanced attackers. We mark
them as # G.
We further investigate the root causes of these information
leakages by inspecting the kernel code (in the Linux kernel C. Inference of Co-resident Container
version 4.7). Generally, the leakage problems are caused by the
incomplete implementation of namespaces in the kernel. To be We further look in depth into specific cases to see whether
more specific, we summarize the two main causes as follows: they could be exploited to detect co-resident containers.
(1) Context checks are missing for existing namespaces, and (2)
some Linux subsystems are not (fully) namespaced. We give 1) Co-residence problems in cloud settings: Co-residence
two case studies on net prio.ifpriomap and RAPL in containers is a well-known research problem in cloud security. In order
to reveal the origins of leakages. to extract a victim’s information, adversaries tend to move
malicious instances to the same physical host with the victim.
1) Case study I — net prio.ifpriomap: The pseudo file Zhang et al. have shown that it is possible for an attacker
net prio.ifpriomap (under /sys/fs/cgroup/net prio) contains to hijack user accounts [47] and extract private keys [46]
a map of the priorities assigned to traffic starting from with co-resident instances. In addition, the cost of achieving
processes in a cgroup and leaving the system on various co-residence is rather low [36]. Co-residence still remains a

4
 TABLE I: LEAKAGE CHANNELS IN COMMERCIAL CONTAINER CLOUD SERVICES.
Potential Vulnerability Container Cloud Services1
Leakage Channels Leakage Information
Co-re DoS Info leak CC1 CC2 CC3 CC4 CC5
/proc/locks Files locked by the kernel # G
#
/proc/zoneinfo Physical RAM information # #
G
/proc/modules Loaded kernel modules information # #
/proc/timer_list Configured clocks and timers # #
/proc/sched_debug Task scheduler behavior # # # #
/proc/softirqs Number of invoked softirq handler
/proc/uptime Up and idle time # #
G
/proc/version Kernel, gcc, distribution version # #
/proc/stat Kernel activities G
#
/proc/meminfo Memory information #
/proc/loadavg CPU and IO utilization over time # #
G
/proc/interrupts Number of interrupts per IRQ #
/proc/cpuinfo CPU information # #
/proc/schedstat Schedule statistics # #
G
/proc/sys/fs/* File system information # #
/proc/sys/kernel/random/* Random number generation info #
/proc/sys/kernel/sched_domain/* Schedule domain info #
/proc/fs/ext4/* Ext4 file system info #
/sys/fs/cgroup/net_prio/* Priorities assigned to traffic # # # # #
/sys/devices/* System device information # #
/sys/class/* System device information # # #

problem in existing clouds, due to the intention of consolidating values and are unique for every host machine. Similarly,
server resources and reducing cost. Traditional methods to verify energy uj in the RAPL sysfs interface is the accumulated energy
co-residence are based on cache [44] or memory-based leakage counter in micro joules. The data read from channels in this
channels [38]. The accuracy of those methods may downgrade group change at real time, but are still unique to represent
due to the high noise in cloud settings. a host. We rank the channels in this group based on their
growth rates. A faster growth rate indicates a lower chance of
2) Approaches and results of checking co-resident contain-
duplication.
ers: Since containers can read the host information through the
leakage channels we discovered, we tend to measure whether The metric V demonstrates whether the data change with
some channels can be used for checking container co-residence. time. With this feature available, two containers can make
We define three metrics, namely uniqueness (U), variation (V), snapshots of this pseudo file periodically at the same time.
and manipulation (M) to quantitatively assess each channel’s Then they can determine co-residence by checking whether
capability of inferring co-residence. two data snapshot traces match with each other. For example,
The metric U indicates whether this channel bestows starting from the same time, we can record MemFree in
characteristic data that can uniquely identify a host machine. /proc/meminfo from two containers every second for one minute.
It is the most important and accurate factor for determining If these two 60-point data traces match with each other, we
whether two containers locate on the same host. We have found are confident that these two containers run on the same host.
17 leakage channels (ranked top 17 in Table II) that satisfy Each channel contains a different capacity of information for
this metric. Generally we can classify these channels into three inferring co-residence, which can be naturally measured via the
groups: joint Shannon entropy. We define the entropy H in Formula (1).
Each channel C contains multiple independent data fields Xi ,
1) Channels containing unique static identifiers. For exam- and n represents the number of independent data fields. Each Xi
ple, boot id under /proc/sys/kernel/random is a random string has possible values {xi1 , · · · , xim }. We rank the capability of
generated at boot time and is unique for each running kernel. revealing co-residence for the nine channels (for which U=False
If two containers can read the same boot id, this is a clear and V=True) based on their entropy results in Table II.
sign that they are running on the same host kernel. The data
for channels in this group are both static and unique. n m
X X
2) Channels into which container tenants can dy- H[C(X1 , · · · , Xn )] = [− p(xij ) log p(xij )]. (1)
i=1 j=1
namically implant unique signatures. For example, from
/proc/sched debug, container users can retrieve all active
process information of the host through this interface. A tenant The metric M indicates whether the container tenants can
can launch a process with a uniquely crafted task name inside manipulate the data. We mark a channel if tenants can
the container. From the other containers, they can verify co- directly implant specially-crafted data into it. For example, we
residence by searching this task name in their own sched debug. can create a timer in a program with a special task name inside
Similar situations apply to timer list and locks. a container. This task name and its associated timer will appear
3) Channels containing unique dynamic identifiers. For in /proc/timer list. Another container can search for this special
example, /proc/uptime has two data fields: system up time and task name in the timer list to verify co-residence. We mark a
system idle time in seconds since booting. They are accumulated channel # G if tenants can only indirectly influence the data in
this channel. For example, an attacker can use taskset command
1 The most recent check on the leakage channels was made on November to bond a computing-intensive workload to a specific core, and
28, 2016. check the CPU utilization, power consumption, or temperature

5
 1200
Sampling interval:1s
TABLE II: LEAKAGE CHANNELS FOR CO-RESIDENCE VERIFICATION.
1150

Power (W)
Leakage Channels U V M Rank
1100

/proc/sys/kernel/random/boot_id # # 1150 1050

Sampling interval:30s
/sys/fs/cgroup/net_prio/net_prio.ifpriomap # # 1000
1100 0 50 100 150 200
Time (s)
/proc/sched_debug

Power (W)
1050
/proc/timer_list
/proc/locks 1000

/proc/uptime #
G 950
0 1 2 3 4 5 6 7
Time (day)
/proc/stat #
G
/proc/schedstat #
G Fig. 2: The power consumption for 8 servers in one week.
/proc/softirqs #
G
/proc/interrupts #
G reliability of data centers. We demonstrate that adversaries can
/sys/devices/system/node/node#/numastat #
G exploit these information leakages discovered by us to amplify
/sys/class/powercap/.../energy_uj2 #
G the attack effects, reduce the attack costs, and facilitate attack
/sys/devices/system/.../usage3 #
G orchestration. All experiments are conducted in a real-world
/sys/devices/system/.../time4 #
G container cloud.
/proc/sys/fs/dentry-state #
G
/proc/sys/fs/inode-nr #
G
A. Attack Amplification
/proc/sys/fs/file-nr #
G
/proc/zoneinfo # #
G The key to launching a successful power attack is to
/proc/meminfo # #
G generate a short-time high power spike that can surpass the
/proc/fs/ext4/sda#/mb_groups # #
G power facility’s supply capacity. As we mentioned in II-C,
/sys/devices/system/node/node#/vmstat # #
G the root cause of power attacks is the wide adoption of
/sys/devices/system/node/node#/meminfo # #
G
power oversubscription, which makes it possible for power
/sys/devices/platform/.../temp#_input5 # #
G
spikes to surpass the safe threshold. In addition, a rack-level
power capping mechanism can only react in minute-level time
/proc/loadavg # #
G
granularity, leaving space for the occurrence of a short-time
/proc/sys/kernel/random/entropy_avail # #
G
high power spike. In the most critical situation, the overcharging
/proc/sys/kernel/.../max_newidle_lb_cost6 # #
of power may trip the branch circuit breaker, cause a power
/proc/modules # # #
outage, and finally bring down the servers. The heights of power
/proc/cpuinfo # # #
spikes are predominantly determined by the resources that are
/proc/version # # # controlled by attackers. Existing power attacks maximize the
Low High power consumption by customizing power-intensive workloads,
denoted as power viruses. For example, Ganesan et al. [15],
[16] leveraged genetic algorithms to automatically generate
from another container. Those entries could be exploited by power viruses that consume more power than normal stress
advanced attackers as covert channels to transmit signals. benchmarks. However, launching a power attack from scratch
or being agnostic about the surrounding environment wastes
For those channels that do not have these U V M properties,
unnecessary attacking resources.
we consider them hard to be exploited. For example, most
servers in a cloud data center probably install the same OS In a real-world data center, the average utilization is around
distribution with the same module list. Although /proc/modules 20% to 30%, as reported by Barroso et al. [9]. With such
leaks the information of loaded modules on the host, it is low utilization, the chance of tripping the circuit breaker by
difficult to use this channel to infer co-resident containers. indiscriminately launching power attacks is extremely low.
However, although the average utilization is low, data centers
IV. S YNERGISTIC P OWER ATTACK still encounter power outage threats under peak demands [37].
This indicates that the power consumption of physical servers
At first glance, the leaked information discovered in fluctuates enormously with the changing workloads. To confirm
Section III seems difficult to exploit. Because both procfs and this assumption, we conduct an experiment to monitor the
sysfs are mounted read-only inside the containers, malicious whole-system power consumption (via the RAPL leakage
tenants can only read such information, but modification is not channel in case study II of Section III) of eight physical servers
allowed. We argue that attackers can make better decisions by in a container cloud for one week. We present the result in
learning the runtime status of the host machine. Figure 2. We first average the power data with a 30-second
interval and observe drastic power changes on both Day 2
In this section, we present a potential synergistic power
and Day 5. Furthermore, we pick a high power consumption
attack in the scope of power outage threats that may impact the
region in Day 2 and average the data at the interval of one
2 /sys/class/powercap/intel-rapl:#/intel-rapl:#/energy uj
second (which is a typical time window for generating a power
3 /sys/devices/system/cpu/cpu#/cpuidle/state#/usage spike). The peak power consumption could reach 1,199 Watts
4 /sys/devices/system/cpu/cpu#/cpuidle/state#/time (W). In total, there was a 34.72% (899W ∼ 1,199W) power
5 /sys/devices/platform/coretemp.#/hwmon/hwmon#/temp# input difference in this one-week range. We anticipate that the power
6 /proc/sys/kernel/sched domain/cpu#/domain#/max newidle lb cost consumption difference would be even larger if we could

6
 250

Synergistic
Periodical 200

Power (W)
150

100

No attack
50 1 Container
2 Containers
3 Containers
0
0 200 400 600 800 1000
Time (s)

Fig. 3: The power consumption of 8 servers under attack. Fig. 4: The power consumption of a server under attack.

monitor it for a longer time period, such as on a holiday continuous and periodic attacks. In Figure 3, we compare the
like Black Friday, when online shopping websites hosted on a attack effects of a synergistic power attack with a periodic attack
cloud may incur a huge power surge. (launching power attacks every 300 seconds). Synergistic power
attacks can achieve a 1,359W power spike with only two trials
For a synergistic power attack in a container cloud, instead
in 3,000 seconds, whereas periodic attacks were launched nine
of indiscriminately starting a power-intensive workload, the
times and could only reach 1,280W at most.
adversaries can monitor the whole-system power consumption
through the RAPL channel and learn the crests and troughs of
the power consumption pattern at real time. Therefore, they C. Attack Orchestration
can leverage the background power consumption (generated Different from traditional power attacks, another unique
by benign workloads from other tenants on the same host) characteristic of synergistic power attack is its attack orches-
and superimpose their power attacks when the servers are at tration. Assume an attacker is already controlling a number
their peak running time. This is similar to the phenomenon of container instances. If these containers scatter in different
of insider trading in the financial market—the one with more locations within a data center, their power additions on multiple
insider information can always trade at the right time. The physical servers put no pressure on power facilities. Existing
adversaries can boost their power spikes, by adding on already- power-capping mechanisms can tolerate multiple small power
high power consumption, to a higher level with the “insider” surges from different locations with no difficulty. The only
power consumption information leaked through the RAPL way to launch a practical power attack is to aggregate all
channel. “ammunition” into adjacent locations and attack a single power
supply simultaneously. Here we discuss in depth on the
B. Reduction of Attack Costs orchestration of attacking container instances.
From the attackers’ perspective, they always intend to As we mentioned in Section III, by exploiting multiple
maximize attack outcomes with the lowest costs. Running leakage channels7 , attackers can aggregate multiple container
power-intensive workloads continuously could definitely catch instances into one physical server. Specifically in our experiment
all the crests of benign power consumption. However, it may on CC1 , we choose to use timer list to verify the co-residence
not be practical for real-world attacks for several reasons. of multiple containers. The detailed verification method is
First, it is not stealthy. To launch a power attack, the attacker explained in Section III-C. We repeatedly create container
needs to run power-intensive workloads. Such behavior has instances and terminate instances that are not on the same
obvious patterns and could be easily detected by cloud providers. physical server. By doing this, we succeed in deploying three
Second, utilization-based billing models are now becoming containers on the same server with trivial effort. We run four
more popular. More cloud services provide finer-grained prices copies of Prime [7] benchmark within each container to fully
based on CPU/memory utilization and the volume of network utilize the four allocated cores. The results are illustrated
traffic. For instance, Elastic Container provides containers with in Figure 4. As we can see, each container can contribute
CPU metering-based billing for customers [3]. IBM Cloud approximately 40W power. With three containers, an attacker
provides billing metrics for computing resources in the cloud can easily raise the power consumption to almost 230W, which
[4]. Amazon EC2 [1] offers Burstable Performance Instances is about 100W more than the average power consumption for
that could occasionally burst but do not fully run most of the a single server.
time. The VMware OnDemand Pricing Calculator [5] even
gives an estimate for different utilization levels. For example, We also find /proc/uptime to be another interesting leakage
it charges $2.87 per month for an instance with 16 VCPUs channel. The uptime includes two data entries, the booting time
with an average of 1% utilization, and $167.25 for the same of the physical server and the idle time of all cores. In our
server with full utilization. Under these cloud billing models, experiment, we find that some servers have similar booting
continuous power attacks may finally lead to an expensive bill. times but different idle times. Typically servers in data centers
do not reboot once being installed and turned on. A different
For synergistic power attacks, monitoring power consump- idle time indicates that they are not the same physical server,
tion through RAPL has almost zero CPU utilization. To achieve
the same effects (the height of power spikes), synergistic power 7 Typically, if a channel is a strong co-residence indicator, leveraging this
attacks can significantly reduce the attack costs compared to one channel only should be enough.

7
 cpuacct
cgroup CPU Cycles RAPL Host
Core Modeling Value Power
cgroup
Initialization Container
Collected Data
perf_event
cgroup Retired Host DRAM Modeling
Instructions Collected Data
Container
Cache Misses Calibration Power
perf_event Package Modeling
Branch Misses

(a) Data Collection (b) Power Modeling (c) On-the-fly Calibration

Fig. 5: The workflow of power-based namespace.

while a similar booting time demonstrates that they have a easy to be precisely partitioned to each container. However, we
high probability of being installed and turned on at the same consider this to be a fundamental solution to the problem.
time period. This is strong evidence that they might also be in In particular, to defend against synergistic power attacks,
close proximity and share the same circuit breaker. Attackers we design and implement a proof-of-concept power-based
can exploit this channel to aggregate their attack container namespace in the Linux kernel to present the partitioned power
instances into adjacent physical servers. This greatly increases usage to each container.
their chances of tripping circuit breakers to cause power outages.
B. Power-based Namespace
V. D EFENSE A PPROACH
We propose a power-based namespace to present per-
A. A Two-Stage Defense Mechanism container power usage through the unchanged RAPL interface
to each container. Without leaking the system-wide power
Intuitively, the solution should eliminate all the leakages consumption information, attackers cannot infer the power state
so that no leaked information could be retrieved through those of the host, thus eliminating their chance of superimposing
channels. We divide the defense mechanism into two stages to power-intensive workloads on benign power peaks. Moreover,
close the loopholes: (1) masking the channels and (2) enhancing with per-container power usage statistics at hand, we can
the container’s resource isolation model. dynamically throttle the computing power (or increase the
In the first stage, the system administrators can explicitly usage fee) of containers that exceed their predefined power
deny the read access to the channels within the container, e.g., thresholds. It is possible for container cloud administrators to
through security policies in AppArmor or mounting the pseudo design a finer-grained billing model based on this power-based
file “unreadable”. This does not require any change to the kernel namespace.
code (merging into the upstream Linux kernel might take some There are three goals for our design. (1) Accuracy: as there
time) and can immediately eliminate information leakages. This is no hardware support for per-container power partitioning, our
solution depends on whether legitimate applications running software-based power modeling needs to reflect the accurate
inside the container use these channels. If such information power usage for each container. (2) Transparency: applications
is orthogonal to the containerized applications, masking it inside a container should be unaware of the power variations
will not have a negative impact on the container tenants. We outside this namespace, and the interface of power subsystem
have reported our results to Docker and all the cloud vendors should remain unchanged. (3) Efficiency: power partitioning
listed in Table I, and we have received active responses. We should not incur non-trivial performance overhead in or out of
are working together with container cloud vendors to fix this containers.
information leakage problem and minimize the impact upon
applications hosted in containers. This masking approach is a We illustrate the workflow of our system in Figure 5. Our
quick fix, but it may add restrictions for the functionality of power-based namespace consists of three major components:
containerized applications, which contradicts the container’s data collection, power modeling, and on-the-fly calibration.
concept of providing a generalized computation platform. We maintain the same Intel RAPL interface within containers,
but change the implementation of handling read operations
In the second stage, the defense approach involves fixing on energy usages. Once a read operation of energy usage is
missing namespace context checks and virtualizing more system detected, the modified RAPL driver retrieves the per-container
resources (i.e., the implementation of new namespaces) to performance data (data collection), uses the retrieved data to
enhance the container’s isolation model. We first reported model the energy usage (power modeling), and finally calibrates
information disclosure bugs related to existing namespaces the modeled energy usage (on-the-fly calibration). We discuss
to Linux kernel maintainers, and they quickly released a new each component in detail below.
patch for one of the problems ([CVE-2017-5967]). For the other
channels with no namespace isolation protection, we need to 1) Data collection: In order to model per-container power
change the kernel code to enforce a finer-grained partition consumption, we need to obtain the fine-grained performance
of system resources. Such an approach could involve non- data for each container. Each container is associated with
trivial efforts since each channel needs to be fixed separately. a cpuacct cgroup. A cpuacct cgroup accounts for the CPU
Virtualizing a specific kernel component might affect multiple cycles on a processor core for a container. The CPU cycles are
kernel subsystems. In addition, some system resources are not accumulated. We only use CPU cycles to compute the rate of

8
 15 4 with application types. To make our model more accurate, we
3.5
further include the cache miss rate [24] and branch miss rate
3
to build a multi-degree polynomial model to fit the slope.
10
For the DRAM, we use the number of cache misses to
Energy (J)

Energy (J)
2.5

2
profile the energy. Figure 7 presents the energy consumption
5
prime
stress−4M 1.5
prime
stress−4M
for the same benchmarks with the same configurations in the
loop
stress−32M
loop
stress−32M
core experiment. It clearly indicates that the number of cache
1
stress−128M
libquantum
Almost no
cache misses
stress−128M
libquantum
misses is approximately linear to the DRAM energy. Based on
0
1 2 3 4 5
Number of instructions
6 7
9
8
0.5
0 1 2 3
Number of Cache Misses
4
9
5 this, we use the linear regression of cache misses to model the
x 10 x 10
DRAM energy.
Fig. 6: Power modeling: the Fig. 7: Power modeling: the
relation between core energy relation between DRAM en- For the power consumption of package, we sum the values
and the number of retired in- ergy and the number of cache of core, DRAM, and an extra constant. The specific models
structions. misses. are illustrated in Formula (2), where M represents the modeled
energy; CM, BM, C indicate the number of cache misses,
branch misses, and CPU cycles, respectively; and F is the
the cache miss rate and branch miss rate later. The Linux kernel function derived through multiple linear regressions to fit the
also has a perf event subsystem, which supports accounting slope. I is the number of retired instructions. α, β, γ, and λ
for different types of performance events. The granularity of are the constants derived from the experiment data in Figures 6
performance accounting could be a single process or a group and 7.
of processes (considered as a perf event cgroup). By now, we
only retrieve the data for retired instructions, cache misses, CM BM
and branch misses (which are needed in the following power Mcore = F( , ) · I + α,
C C
modeling component) for each perf event cgroup. Our current Mdram = β · CM + γ, (2)
implementation is extensible to collect more performance event
types corresponding to the changes of power modeling in the Mpackage = Mcore + Mdram + λ.
future.
Here we discuss the influence of floating point instructions
We monitor the performance events from the initialization for power modeling. While an individual floating point instruc-
of a power-based namespace and create multiple perf events, tion might consume more energy than an integer operation,
each associated with a specific performance event type and workloads with high ratios of floating point instructions might
a specific CPU core. Then we connect the perf cgroup of actually result in lower power consumption overall, since the
this container with these perf events to start accounting. In functional units might be forced to be idle in different stages of
addition, we need to set the owner of all created perf events the pipeline. It is necessary to take the micro-architecture into
as TASK TOMBSTONE, indicating that such performance consideration to build a more refined model. We plan to pursue
accounting is decoupled from any user process. this direction in our future work. Furthermore, the choices of
parameters α, β, γ are also affected by the architecture. Such
2) Power modeling: To implement a power-based names- a problem could be mitigated in the following calibration step.
pace, we need to attribute the power consumption to each
container. Instead of providing transient power consumption, 3) On-the-fly calibration: Our system also models the
RAPL offers accumulated energy usages for package, core, and energy data for the host and cross-validates it with the actual
DRAM, respectively. The power consumption can be calculated energy data acquired through RAPL. To minimize the error of
by measuring the energy consumption over a time unit window. modeling data, we use the following Formula (3) to calibrate
Our power-based namespace also provides accumulative per- the modeled energy data for each container. The Econtainer
container energy data, in the same format as in the original represents the energy value returned to each container. This
RAPL interface. on-the-fly calibration is conducted for each read operation to
the RAPL interface and can effectively reduce the number of
We first attribute the power consumption for the core. errors in the previous step.
Traditional power modeling leverages CPU utilization [29] to
attribute the power consumption for cores. However, Xu et al. Mcontainer
Econtainer = · ERAPL . (3)
[43] demonstrated that the power consumption could vary signif- Mhost
icantly with the same CPU utilization. The underlying pipeline
and data dependence could lead to CPU stalls and idling of VI. D EFENSE E VALUATION
function units. The actual numbers of retired instructions [24], In this section, we evaluate our power-based namespace
[33] under the same CPU utilization are different. Figure 6 on a local machine in three aspects: accuracy, security, and
reveals the relation between retired instructions and energy. performance. Our testbed is equipped with Intel i7-6700
We test on four different benchmarks: the idle loop written in 3.40GHz CPU with 8 cores, 16GB of RAM, and running
C, prime, 462.libquantum in SPECCPU2006, and stress with Ubuntu Linux 16.04 with kernel version 4.7.0.
different memory configurations. We run the benchmarks on
a host and use Perf [6] to collect performance statistics data.
A. Accuracy
We can see that for each benchmark, the energy consumption
is almost strictly linear to the number of retired instructions. We use the SPECCPU2006 benchmark to measure the
However, the gradients of fitted lines change correspondingly accuracy of the power modeling. We compare the modeled

9
 TABLE III: PERFORMANCE RESULTS OF UNIX BENCHMARKS.
1 Parallel Copy 8 Parallel Copies
Benchmarks Original Modified Overhead Original Modified Overhead
Dhrystone 2 using register variables 3,788.9 3,759.2 0.78% 19,132.9 19,149.2 0.08%
Double-Precision Whetstone 926.8 918.0 0.94% 6,630.7 6,620.6 0.15%
Execl Throughput 290.9 271.9 6.53% 7,975.2 7,298.1 8.49%
File Copy 1024 bufsize 2000 maxblocks 3,495.1 3,469.3 0.73% 3,104.9 2,659.7 14.33%
File Copy 256 bufsize 500 maxblocks 2,208.5 2,175.1 0.04% 1,982.9 1,622.2 18.19%
File Copy 4096 bufsize 8000 maxblocks 5,695.1 5,829.9 -2.34% 6,641.3 5,822.7 12.32%
Pipe Throughput 1,899.4 1,878.4 1.1% 9,507.2 9,491.1 0.16%
Pipe-based Context Switching 653.0 251.2 61.53% 5,266.7 5,180.7 1.63%
Process Creation 1416.5 1289.7 8.95% 6618.5 6063.8 8.38%
Shell Scripts (1 concurrent) 3,660.4 3,548.0 3.07% 16,909.7 16,404.2 2.98%
Shell Scripts (8 concurrent) 11,621.0 11,249.1 3.2% 15,721.1 15,589.2 0.83%
System Call Overhead 1,226.6 1,212.2 1.17% 5,689.4 5,648.1 0.72%
Fig. 8: The accuracy of our Fig. 9: Transparency: a ma- System Benchmarks Index Score 2,000.8 1,807.4 9.66% 7,239.8 6,813.5 7.03%

energy modeling approach to licious container (Container

estimate the active power for 2) is unaware of the power
the container from aggregate condition for the host. C. Performance
event usage and RAPL. We use UnixBench to compare the performance overhead
before and after enabling our system. Table III lists all results.
power usage with the ground truth obtained through RAPL. As the results show, CPU benchmarks such as Dhrystone
The power consumption is equal to the energy consumption (testing integer and string operations) and Whetstone (testing
per second. Due to the restriction of the security policy of float point arithmetic performance) incur negligible overhead.
the Docker container, we select a subset of SPECCPU2006 Other benchmarks like shell scripts, pipe throughput, and
benchmarks that are feasible to run inside the container and system call also trigger little overhead. The pipe-based context
have no overlap with the benchmarks used for power modeling. switching does incur a 61.53% overhead in the case of one
The error ξ is defined as follows: parallel copy, but it decreases to 1.63% for 8 parallel copies.
We anticipate that inter-cgroup context switching involves
|(ERAPL − ∆diff ) − Mcontainer | enabling/disabling the performance event monitor, whereas
ξ= , (4)
ERAPL − ∆diff intra-cgroup context switching does not involve any such
overhead. This could explain why 8 parallel copies can maintain
where ERAPL is the power usage read from RAPL on the a similar performance level with the power-based namespace
host, and Mcontainer is the modeled power usage for the same disabled. In addition, context switching only contributes to a
workload read within the container. Note that both the host very small portion of the whole-system performance overhead,
and container consume power at an idle state with trivial so there is trivial impact for the normal use. As demonstrated
differences. We use a constant ∆diff as the modifier reflecting in the last row of Table III, the overall performance overheads
the difference in power consumption at an idle state for a host for the UnixBench are 9.66% for one parallel copy and 7.03%
and a container. The results, illustrated in Figure 8, show that for 8 parallel copies, respectively. Our system’s performance
our power modeling is accurate as the error values of all the depends heavily on the implementation of perf event cgroup
tested benchmarks are lower than 0.05. and could improve with the advancement of a performance
monitoring subsystem.

B. Security VII. D ISCUSSION

We also evaluate our system from the security perspective. A. Synergistic Power Attacks without the RAPL Channel
With the power-based namespace enabled, the container should
only retrieve the power consumed within the container and be We also notice that servers in some container clouds are not
unaware of the host’s power status. We launch two containers equipped with RAPL or other similar embedded power meters.
in our testbed for comparison. We run the SPECCPU2006 Those servers might still face power attacks. Without power-
benchmark in one container and leave the other one idle. We capping tools like RAPL, those servers might be vulnerable to
record the power usage per second of both containers and the host-level power attacks on a single machine. In addition, if
host. We show the results of 401.bzip2 in Figure 9. All other power data is not directly available, advanced attackers will try
benchmarks exhibit similar patterns. to approximate the power status based on the resource utilization
information, such as the CPU and memory utilization, which is
When both containers have no workload, their power still available in the identified information leakages. It would be
consumption is at the same level as that of the host, e.g., from better to make system-wide performance statistics unavailable
0s to 10s. Once container 1 starts a workload at 10s, we can find to container tenants.
that the power consumption of container 1 and the host surges
simultaneously. From 10s to 60s, container 1 and the host have B. Complete Container Implementation
a similar power usage pattern, whereas container 2 is still at a
low power consumption level. Container 2 is unaware of the The root cause for information leakage and synergistic
power fluctuation on the whole system because of the isolation power attack is the incomplete implementation of the isolation
enforced by the power-based namespace. This indicates that mechanisms in the Linux kernel. It would be better to introduce
our system is effective for isolating and partitioning power more security features, such as implementing more namespaces
consumption for multiple containers, and thus it can neutralize and control groups. However, some system resources are
the synergistic power attacks. still difficult to be partitioned, e.g., interrupts, scheduling

10
information, and temperature. People also argue that the leverage side channels or covert channels, e.g., one widely
complete container implementation is no different from a virtual adopted approach is to use cache-based covert channels [22],
machine, and loses all the container’s advantages. It is a trade- [35], [41]. Multiple instances locating on the same package
off for containers to deal with. The question of how to balance share the last-level caches. By using some dedicated opera-
security, performance, and usability in container clouds needs tions, such as cflush [44], attackers can detect co-residence
further investigation. by measuring the time of cache accessing. Liu et al. [27]
demonstrated that l3 cache side-channel attacks are practical
VIII. R ELATED W ORK for cross-core and cross-VM attacks. Zhang et al. conducted
real side-channel attacks on the cloud [46], [47] and proposed
In this section, we list some research efforts that inspire several defense mechanisms to mitigate those attacks [40], [45],
our work and highlight the differences between our work and [48]. In particular, they demonstrated that cross-tenant side-
previous research. We mainly discuss research works in the channel attacks can be successfully conducted in PaaS with co-
following three areas: resident servers [47]. Besides the cache-based channel, memory
bus [38] and memory deduplication [39] have also proved to
A. Performance and Security Research on Containers be effective for covert-channel construction. Different from
Since containers have recently become popular, researchers existing research efforts on side/covert channels, we discover a
are curious about the performance comparison between con- system-wide information leakage in the container cloud settings
tainers and hardware virtualization. Felter et al. compared and design a new methodology to quantitatively assess the
the performance of Docker and KVM by using a set of capacity of leakage channels for co-residence detection. In
benchmarks covering CPU, memory, storage, and networking addition, compared to the research on minimizing the kernel
resources [14]. Their results show that Docker can achieve attack surface for VMs [18], we proposed a two-stage defense
equal or better performance than KVM in all cases. Spoiala mechanism to minimize the space for information leakages and
et al. [34] used the Kurento Media Server to compare the power attacks on container clouds.
performance of WebRTC servers on both Docker and KVM. System status information, such as core temperature and
They also demonstrated that Docker outperforms KVM and system power consumption, have also been used to build
could support real-time applications. Morabito et al. [30] side/covert channels. Thiele et al. [10], [28] proposed a thermal
compared the performance between traditional hypervisor and covert channel based on the temperature of each core and tested
OS-level virtualization with respect to computing, storage, the capacity in a local testbed. Power consumption could also be
memory, and networks. They conducted experiments on Docker, abused to break AES [23]. In our work, we do not use the power
LXC, and KVM and observed that Disk I/O is still the bottleneck consumption data as a covert channel to transfer information.
of the KVM hypervisor. All of these works demonstrate that Instead, we demonstrate that adversaries may leverage the host
container-based OS-level virtualization can achieve a higher power consumption leakage to launch more advanced power
performance than hardware virtualization. Besides performance, attacks.
the security of a container cloud is always an important research
area. Gupta [20] gave a brief overview of Docker security. C. Power Modeling
Bui [11] also performed an analysis on Docker containers,
including the isolation problem and corresponding kernel When hardware-based power meter is absent, power mod-
security mechanisms. They claimed that Docker containers eling is the approach to approximating power consumption.
are fairly secure with default configurations. Grattafiori et Russell et al. [32] and Chakrabarti et al. [12] proposed
al. [17] summarized a variety of potential vulnerabilities instruction-level power modeling. Their works indicate that
of containers. They also mentioned several channels in the the number of branches affects power consumption. There
memory-based pseudo file systems. Previous research efforts are several works of approximating power consumption for
on the performance and security of containers encourage us VMs. Both works [21], [24] demonstrate that VM-level power
to investigate more on how containers can achieve the same consumption can be estimated by CPU utilization and last-level
security guarantees as hardware virtualization, but with trivial cache miss. Mobius et al. [29] broke the power consumption
performance trade-offs. We are among the first to systematically of VM into CPU, cache, memory, and disk. BITWATTS [13]
identify the information leakage problem in containers and modeled the power consumption at a finer-grained process
investigate potential container-based power attack threats built level. Shen et al. [33] proposed a power container to account
upon these leakage channels. for energy consumption of requests in multi-core systems.
Our defense against the synergistic power attack is mainly
B. Cloud Security and Side/Covert Channel Attacks inspired by the power modeling approach for VMs. We propose
a new power partitioning technique to approximate the per-
Cloud security has received much attention from both container power consumption and reuse the RAPL interface,
academia and industry. Co-residence detection in the cloud thus addressing the RAPL data leakage problem in the container
settings is the most closely related research topic to our settings.
work. Co-residence detection was first proposed by Ristenpart
et al. [31]. They demonstrated that an attacker can place a IX. C ONCLUSION
malicious VM co-resident with a target VM on the same sever
and then launch side-channel and covert-channel attacks. Two Container cloud services have become popular for provid-
previous works [36], [42] show that it is still practical to achieve ing lightweight OS-level virtual hosting environments. The
co-residence in existing mainstream cloud services. To verify emergence of container technology profoundly changes the eco-
co-residence on the same physical server, attackers typically system of developing and deploying distributed applications in

11
the cloud. However, due to the incomplete implementation [23] P. Kocher, J. Jaffe, and B. Jun. Differential Power Analysis. In Annual
of system resource partitioning mechanisms in the Linux International Cryptology Conference, 1999.
kernel, there still exist some security concerns for multiple [24] B. Krishnan, H. Amur, A. Gavrilovska, and K. Schwan. VM Power
container tenants sharing the same kernel. In this paper, we Metering: Feasibility and Challenges. ACM SIGMETRICS Performance
Evaluation Review, 2011.
first present a systematic approach to discovering information
[25] C. Lefurgy, X. Wang, and M. Ware. Power Capping: A Prelude to
leakage channels that may expose host information to containers. Power Shifting. Cluster Computing, 2008.
By exploiting such leaked host information, malicious container [26] C. Li, Z. Wang, X. Hou, H. Chen, X. Liang, and M. Guo. Power Attack
tenants can launch a new type of power attack that may Defense: Securing Battery-Backed Data Centers. In IEEE ISCA, 2016.
potentially jeopardize the dependability of power systems in [27] F. Liu, Y. Yarom, Q. Ge, G. Heiser, and R. B. Lee. Last-Level Cache
the data centers. Additionally, we discuss the root causes of Side-Channel Attacks are Practical. In IEEE S&P, 2015.
these information leakages and propose a two-stage defense [28] R. J. Masti, D. Rai, A. Ranganathan, C. Müller, L. Thiele, and S. Capkun.
mechanism. Our evaluation demonstrates that the proposed Thermal Covert Channels on Multi-core Platforms. In USENIX Security,
solution is effective and incurs trivial performance overhead. 2015.
[29] C. Mobius, W. Dargie, and A. Schill. Power Consumption Estimation
Models for Processors, Virtual Machines, and Servers. IEEE Transactions
R EFERENCES on Parallel and Distributed Systems, 2014.
[1] Burstable Performance Instances. https://aws.amazon.com/ec2/instance- [30] R. Morabito, J. Kjällman, and M. Komu. Hypervisors vs. Lightweight
types/#burst. Virtualization: A Performance Comparison. In IEEE IC2E, 2015.
[2] Containers Not Virtual Machines Are the Future Cloud. [31] T. Ristenpart, E. Tromer, H. Shacham, and S. Savage. Hey, You, Get Off
http://www.linuxjournal.com/content/containers. of My Cloud: Exploring Information Leakage in Third-Party Compute
[3] ElasticHosts: Linux container virtualisation allows us to beat AWS on Clouds. In ACM CCS, 2009.
price. http://www.computerworlduk.com/news/it-leadership/elastichosts- [32] J. T. Russell and M. F. Jacome. Software Power Estimation and
linux-container-virtualisation-allows-us-beat-aws-on-price-3510973/. Optimization for High Performance, 32-bit Embedded Processors. In
[4] IBM Cloud metering and billing. https://www.ibm.com/developerworks/c- IEEE ICCD, 1998.
loud/library/cl-cloudmetering/. [33] K. Shen, A. Shriraman, S. Dwarkadas, X. Zhang, and Z. Chen.
[5] OnDemand Pricing Calculator. http://vcloud.vmware.com/service- Power Containers: An OS Facility for Fine-Grained Power and Energy
offering/pricing-calculator/on-demand. Management on Multicore Servers. ACM ASPLOS, 2013.
[6] Perf Wiki. https://perf.wiki.kernel.org/index.php/Main Page. [34] C. C. Spoiala, A. Calinciuc, C. O. Turcu, and C. Filote. Performance
[7] Prime95 Version 28.9. http://www.mersenne.org/download/ . comparison of a WebRTC server on Docker versus Virtual Machine. In
IEEE DAS, 2016.
[8] Travel nightmare for fliers after power outage grounds Delta.
http://money.cnn.com/2016/08/08/news/companies/delta-system-outage- [35] E. Tromer, D. A. Osvik, and A. Shamir. Efficient Cache Attacks on
flights/. AES, and Countermeasures. Journal of Cryptology, 2010.
[9] L. A. Barroso and U. Hölzle. The Case for Energy-Proportional [36] V. Varadarajan, Y. Zhang, T. Ristenpart, and M. Swift. A Placement
Computing. Computer, 2007. Vulnerability Study in Multi-Tenant Public Clouds. In USENIX Security,
2015.
[10] D. B. Bartolini, P. Miedl, and L. Thiele. On the Capacity of Thermal
Covert Channels in Multicores. In ACM EuroSys, 2016. [37] Q. Wu, Q. Deng, L. Ganesh, C.-H. Hsu, Y. Jin, S. Kumar, B. Li,
J. Meza, and Y. J. Song. Dynamo: Facebooks Data Center-Wide Power
[11] T. Bui. Analysis of Docker Security. arXiv preprint arXiv:1501.02967, Management System. IEEE ISCA, 2016.
2015.
[38] Z. Wu, Z. Xu, and H. Wang. Whispers in the Hyper-space: High-speed
[12] C. Chakrabarti and D. Gaitonde. Instruction Level Power Model of Covert Channel Attacks in the Cloud. In USENIX Security, 2012.
Microcontrollers. In IEEE ISCAS, 1999.
[39] J. Xiao, Z. Xu, H. Huang, and H. Wang. Security Implications of
[13] M. Colmant, M. Kurpicz, P. Felber, L. Huertas, R. Rouvoy, and A. Sobe. Memory Deduplication in a Virtualized Environment. In IEEE/IFIP
Process-level Power Estimation in VM-based Systems. In ACM EuroSys, DSN, 2013.
2015.
[40] Q. Xiao, M. K. Reiter, and Y. Zhang. Mitigating Storage Side Channels
[14] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An Updated Using Statistical Privacy Mechanisms. In ACM CCS, 2015.
Performance Comparison of Virtual Machines and Linux Containers. In
IEEE ISPASS, 2015. [41] Y. Xu, M. Bailey, F. Jahanian, K. Joshi, M. Hiltunen, and R. Schlichting.
An Exploration of L2 Cache Covert Channels in Virtualized Environ-
[15] K. Ganesan, J. Jo, W. L. Bircher, D. Kaseridis, Z. Yu, and L. K. John. ments. In ACM CCSW, 2011.
System-level Max Power (SYMPO) - A Systematic Approach for Esca-
lating System-level Power Consumption using Synthetic Benchmarks. [42] Z. Xu, H. Wang, and Z. Wu. A Measurement Study on Co-residence
In ACM PACT, 2010. Threat inside the Cloud. In USENIX Security, 2015.
[16] K. Ganesan and L. K. John. MAximum Multicore POwer (MAMPO) [43] Z. Xu, H. Wang, Z. Xu, and X. Wang. Power Attack: An Increasing
- An Automatic Multithreaded Synthetic Power Virus Generation Threat to Data Centers. In NDSS, 2014.
Framework for Multicore Systems. In ACM SC, 2011. [44] Y. Yarom and K. Falkner. FLUSH+ RELOAD: A High Resolution, Low
[17] A. Grattafiori. NCC Group Whitepaper: Understanding and Hardening Noise, L3 Cache Side-Channel Attack. In USENIX Security, 2014.
Linux Containers, 2016. [45] Y. Zhang, A. Juels, A. Oprea, and M. K. Reiter. HomeAlone: Co-
[18] Z. Gu, B. Saltaformaggio, X. Zhang, and D. Xu. FACE-CHANGE: residency Detection in the Cloud via Side-Channel Analysis. In IEEE
Application-Driven Dynamic Kernel View Switching in a Virtual S&P, 2011.
Machine. In IEEE/IFIP DSN, 2014. [46] Y. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. Cross-VM Side
[19] P. Guide. Intel R
64 and IA-32 Architectures Software Developers Channels and Their Use to Extract Private Keys. In ACM CCS, 2012.
Manual, 2011. [47] Y. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. Cross-Tenant
[20] U. Gupta. Comparison between security majors in virtual machine and Side-Channel Attacks in PaaS Clouds. In ACM CCS, 2014.
linux containers. arXiv preprint arXiv:1507.07816, 2015. [48] Y. Zhang and M. K. Reiter. Düppel: Retrofitting Commodity Operating
[21] Z. Jiang, C. Lu, Y. Cai, Z. Jiang, and C. Ma. VPower: Metering Power Systems to Mitigate Cache Side Channels in the Cloud. In ACM CCS,
Consumption of VM. In IEEE ICSESS, 2013. 2013.
[22] M. Kayaalp, N. Abu-Ghazaleh, D. Ponomarev, and A. Jaleel. A High-
Resolution Side-Channel Attack on Last-Level Cache. In IEEE DAC,
2016.

12