0% found this document useful (0 votes)

764 views

OpenStack Security Guide

Uploaded by

neroliang

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

764 views

OpenStack Security Guide

Uploaded by

neroliang

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 236

OpenStack Security Guide

May 1, 2015

Secret key .................................................................................... 92

Session back end .......................................................................... 92
Allowed hosts .............................................................................. 92
Cross Site Request Forgery (CSRF) ................................................ 93
Cookies ........................................................................................ 93
Cross Site Scripting (XSS) .............................................................. 93
Cross Origin Resource Sharing (CORS) .......................................... 93
Horizon image upload ................................................................. 94
Upgrading ................................................................................... 94
Debug .......................................................................................... 94
8. Compute .......................................................................................... 95
How to select virtual consoles ...................................................... 95
9. Object Storage ................................................................................. 99
First thing to secure: the network .............................................. 100
Securing services: general ........................................................... 102
Securing storage services ............................................................ 103
Securing proxy services ............................................................... 104
Object Storage authentication .................................................... 105
Other notable items ................................................................... 106
10. Case studies: Identity management .............................................. 107
Alice's private cloud ................................................................... 107
Bob's public cloud ...................................................................... 107
11. Networking .................................................................................. 109
Networking architecture ............................................................ 109
Networking services ................................................................... 113
Securing OpenStack Networking services .................................... 117
Networking services security best practices ................................. 119
Case studies ............................................................................... 121
12. Message queuing ......................................................................... 123
Messaging security ..................................................................... 123
Case studies ............................................................................... 128
13. Data processing ............................................................................ 129
Introduction to Data processing ................................................. 129
Deployment ............................................................................... 132
Configuration and hardening ..................................................... 134
Case studies ............................................................................... 138
14. Databases .................................................................................... 141
Database back end considerations .............................................. 141
Database access control ............................................................. 142
Database transport security ........................................................ 147
Case studies ............................................................................... 149
15. Tenant data privacy ..................................................................... 151
Data privacy concerns ................................................................ 151

OpenStack Security Guide

May 1, 2015

current

Data encryption .........................................................................

Key management .......................................................................
Case studies ...............................................................................
16. Hypervisor and virtualization layer ................................................
Hypervisor selection ...................................................................
Hardening the virtualization layers .............................................
Case studies ...............................................................................
17. Instance security management .....................................................
Security services for instances .....................................................
Case studies ...............................................................................
18. Monitoring and logging ...............................................................
Forensics and incident response ..................................................
Case studies ...............................................................................
19. Compliance ..................................................................................
Compliance overview .................................................................
Understanding the audit process ................................................
Compliance activities ..................................................................
Certification and compliance statements ....................................
Privacy .......................................................................................
Case studies ...............................................................................
A. Community support .......................................................................
Documentation ..........................................................................
ask.openstack.org ......................................................................
OpenStack mailing lists ...............................................................
The OpenStack wiki ...................................................................
The Launchpad Bugs area ..........................................................
The OpenStack IRC channel ........................................................
Documentation feedback ...........................................................
OpenStack distribution packages ................................................
Glossary .............................................................................................

155
158
159
161
161
171
178
181
181
190
193
193
195
197
197
201
203
206
211
211
215
215
216
217
217
217
218
219
219
221

OpenStack Security Guide

May 1, 2015

current

List of Figures
1.1. Attack types .................................................................................. 20
9.1. An example diagram from the OpenStack Object Storage Administration Guide (2013) ........................................................................ 100
9.2. Object Storage network architecture with a management node
(OSAM) .............................................................................................. 102

vii

OpenStack Security Guide

May 1, 2015

current

Preface
Conventions .......................................................................................... ix
Document change history ...................................................................... ix

Conventions
The OpenStack documentation uses several typesetting conventions.

Notices
Notices take these forms:

Note
A handy tip or reminder.

Important
Something you must be aware of before proceeding.

Warning
Critical information about the risk of data loss or security issues.

Command prompts
$ prompt

Any user, including the root user, can run commands that
are prefixed with the $ prompt.

# prompt

The root user must run commands that are prefixed with
the # prompt. You can also prefix these commands with the
sudo command, if available, to run them.

Document change history

This version of the guide replaces and obsoletes all earlier versions.
The following table describes the most recent changes:
ix

OpenStack Security Guide

May 1, 2015

Revision Date
April 29, 2015

current

Summary of Changes
Final prep for Kilo release.

February 11, 2015 Chapter on Data processing added.

October 16, 2014

This book has been extensively reviewed and updated. Chapters have been
rearranged and a glossary has been added.

December 2, 2013 Chapter on Object Storage added.

October 17, 2013

Havana release.

July 2, 2013

Initial creation...

OpenStack Security Guide

May 1, 2015

current

1. Introduction
Acknowledgments ................................................................................. 1
Why and how we wrote this book ......................................................... 2
Introduction to OpenStack ..................................................................... 7
Security boundaries and threats ........................................................... 12
Introduction to case studies ................................................................. 21
The OpenStack Security Guide is the result of a five day sprint of collaborative work of many individuals. The purpose of this document is to provide the best practice guidelines for deploying a secure OpenStack cloud.
It is a living document that is updated as new changes are merged into the
repository, and is meant to reflect the current state of security within the
OpenStack community and provide frameworks for decision making where
listing specific security controls are not feasible due to complexity or other
environment specific details.

Acknowledgments
The OpenStack Security Group would like to acknowledge contributions
from the following organizations that were instrumental in making this
book possible. The organizations are:

OpenStack Security Guide

May 1, 2015

current

Why and how we wrote this book

As OpenStack adoption continues to grow and the product matures, security has become a priority. The OpenStack Security Group has recognized
the need for a comprehensive and authoritative security guide. The Open2

OpenStack Security Guide

May 1, 2015

current

Stack Security Guide has been written to provide an overview of security

best practices, guidelines, and recommendations for increasing the security of an OpenStack deployment. The authors bring their expertise from deploying and securing OpenStack in a variety of environments.
This guide augments the OpenStack Operations Guide and can be referenced to harden existing OpenStack deployments or to evaluate the security controls of OpenStack cloud providers.

Objectives
Identify the security domains in OpenStack
Provide guidance to secure your OpenStack deployment
Highlight security concerns and potential mitigations in present day
OpenStack
Discuss upcoming security features
To provide a community driven facility for knowledge capture and dissemination

How
As with the OpenStack Operations Guide, we followed the book sprint
methodology. The book sprint process allows for rapid development and
production of large bodies of written work. Coordinators from the OpenStack Security Group re-enlisted the services of Adam Hyde as facilitator.
Corporate support was obtained and the project was formally announced
during the OpenStack summit in Portland, Oregon.
The team converged in Annapolis, MD due to the close proximity of some
key members of the group. This was a remarkable collaboration between
public sector intelligence community members, silicon valley startups and
some large, well-known technology companies. The book sprint ran during
the last week in June 2013 and the first edition was created in five days.

OpenStack Security Guide

May 1, 2015

The initial work on this book was conducted in an overly air-conditioned
room that served as our group office for the entirety of the documentation sprint.
Learn more about how to contribute to the OpenStack docs: http://
wiki.openstack.org/Documentation/HowTo.

Introduction to OpenStack
This guide provides security insight into OpenStack deployments. The intended audience is cloud architects, deployers, and administrators. In
addition, cloud users will find the guide both educational and helpful in
provider selection, while auditors will find it useful as a reference document to support their compliance certification efforts. This guide is also
recommended for anyone interested in cloud security.
Each OpenStack deployment embraces a wide variety of technologies,
spanning Linux distributions, database systems, messaging queues, OpenStack components themselves, access control policies, logging services, security monitoring tools, and much more. It should come as no surprise that
the security issues involved are equally diverse, and their in-depth analysis would require several guides. We strive to find a balance, providing
enough context to understand OpenStack security issues and their handling, and provide external references for further information. The guide
could be read from start to finish or sampled as necessary like a reference.
7

OpenStack Security Guide

May 1, 2015

current

We briefly introduce the kinds of clouds: private, public, and hybrid before
presenting an overview of the OpenStack components and their related
security concerns in the remainder of the chapter.

Cloud types
OpenStack is a key enabler in adoption of cloud technology and has several common deployment use cases. These are commonly known as Public,
Private, and Hybrid models. The following sections use the National Institute of Standards and Technology (NIST) definition of cloud to introduce
these different types of cloud as they apply to OpenStack.

Public cloud
According to NIST, a public cloud is one in which the infrastructure is open
to the general public for consumption. OpenStack public clouds are typically run by a service provider and can be consumed by individuals, corporations, or any paying customer. A public cloud provider may expose a full
set of features such as software-defined networking, block storage, in addition to multiple instance types. Due to the nature of public clouds, they
are exposed to a higher degree of risk. As a consumer of a public cloud
you should validate that your selected provider has the necessary certifications, attestations, and other regulatory considerations. As a public cloud
provider, depending on your target customers, you may be subject to one
or more regulations. Additionally, even if not required to meet regulatory
requirements, a provider should ensure tenant isolation as well as protecting management infrastructure from external attacks.

Private cloud
At the opposite end of the spectrum is the private cloud. As NIST defines
it, a private cloud is provisioned for exclusive use by a single organization
comprising multiple consumers, such as business units. It may be owned,
managed, and operated by the organization, a third-party, or some combination of them, and it may exist on or off premises. Private cloud use cases
are diverse, as such, their individual security concerns vary.

Community cloud
NIST defines a community cloud as one whose infrastructure is provisioned
for the exclusive use by a specific community of consumers from organizations that have shared concerns. For example, mission, security requirements, policy, and compliance considerations. It may be owned, managed, and operated by one or more of the organizations in the communi8

OpenStack Security Guide

May 1, 2015

current

ty, a third-party, or some combination of them, and it may exist on or off

premises.

Hybrid cloud
A hybrid cloud is defined by NIST as a composition of two or more distinct
cloud infrastructures, such as private, community, or public, that remain
unique entities, but are bound together by standardized or proprietary
technology that enables data and application portability, such as cloud
bursting for load balancing between clouds. For example an online retailer
may have their advertising and catalogue presented on a public cloud that
allows for elastic provisioning. This would enable them to handle seasonal loads in a flexible, cost-effective fashion. Once a customer begins to process their order, they are transferred to the more secure private cloud back
end that is PCI compliant.
For the purposes of this document, we treat Community and Hybrid similarly, dealing explicitly only with the extremes of Public and Private clouds
from a security perspective. Your security measures depend where your deployment falls upon the private public continuum.

OpenStack service overview

OpenStack embraces a modular architecture to provide a set of core services that facilitates scalability and elasticity as core design tenets. This
chapter briefly reviews OpenStack components, their use cases and security considerations.

Compute
OpenStack Compute service (nova) provides services to support the management of virtual machine instances at scale, instances that host multi-tiered applications, dev/test environments, "Big Data" crunching Hadoop
clusters, and/or high performance computing.
9

OpenStack Security Guide

May 1, 2015

A cloud can be abstracted as a collection of logical components by virtue
of their function, users, and shared security concerns, which we call security domains. Threat actors and vectors are classified based on their motivation and access to resources. Our goal is to provide you a sense of the security concerns with respect to each domain depending on your risk/vulnerability protection objectives.

Security domains
A security domain comprises users, applications, servers or networks that
share common trust requirements and expectations within a system. Typically they have the same authentication and authorization (AuthN/Z) requirements and users.
Although you may desire to break these domains down further (we later
discuss where this may be appropriate), we generally refer to four distinct
security domains which form the bare minimum that is required to deploy
any OpenStack cloud securely. These security domains are:
1. Public
2. Guest
3. Management
12

OpenStack Security Guide

May 1, 2015

current

4. Data
We selected these security domains because they can be mapped independently or combined to represent the majority of the possible areas of trust
within a given OpenStack deployment. For example, some deployment
topologies may consist of a combination of guest and data domains onto
one physical network while other topologies have these domains separated. In each case, the cloud operator should be aware of the appropriate
security concerns. Security domains should be mapped out against your
specific OpenStack deployment topology. The domains and their trust requirements depend upon whether the cloud instance is public, private, or
hybrid.

Public
The public security domain is an entirely untrusted area of the cloud infrastructure. It can refer to the Internet as a whole or simply to networks over
which you have no authority. Any data that transits this domain with confi-

OpenStack Security Guide

May 1, 2015

current

dentiality or integrity requirements should be protected using compensating controls.

This domain should always be considered untrusted.

Guest
Typically used for compute instance-to-instance traffic, the guest security
domain handles compute data generated by instances on the cloud but
not services that support the operation of the cloud, such as API calls.
Public and private cloud providers that do not have stringent controls on
instance use or allow unrestricted internet access to VMs should consider
this domain to be untrusted. Private cloud providers may want to consider this network as internal and trusted, only if the proper controls are implemented to assert that the instances and all associated tenants are to be
trusted.

Management
The management security domain is where services interact. Sometimes
referred to as the "control plane", the networks in this domain transport
confidential data such as configuration parameters, user names, and passwords. Command and Control traffic typically resides in this domain, which
necessitates strong integrity requirements. Access to this domain should be
highly restricted and monitored. At the same time, this domain should still
employ all of the security best practices described in this guide.
In most deployments this domain is considered trusted. However, when
considering an OpenStack deployment, there are many systems that
bridge this domain with others, potentially reducing the level of trust you
can place on this domain. See the section called Bridging security domains [15] for more information.

Data
The data security domain is concerned primarily with information pertaining to the storage services within OpenStack. Most of the data transmitted across this network requires high levels of integrity and confidentiality. In some cases, depending on the type of deployment there may also be
strong availability requirements.
The trust level of this network is heavily dependent on deployment decisions and as such we do not assign this any default level of trust.
14

OpenStack Security Guide

May 1, 2015

current

Bridging security domains

A bridge is a component that exists inside more than one security domain.
Any component that bridges security domains with different trust levels or
authentication requirements must be carefully configured. These bridges
are often the weak points in network architecture. A bridge should always
be configured to meet the security requirements of the highest trust level
of any of the domains it is bridging. In many cases the security controls for
bridges should be a primary concern due to the likelihood of attack.

The diagram above shows a compute node bridging the data and management domains, as such the compute node should be configured to meet
the security requirements of the management domain. Similarly, the API
Endpoint in this diagram is bridging the untrusted public domain and the
management domain, which should be configured to protect against attacks from the public domain propagating through to the management
domain.

OpenStack Security Guide

May 1, 2015

current

In some cases deployers may want to consider securing a bridge to a higher standard than any of the domains in which it resides. Given the above
example of an API endpoint, an adversary could potentially target the API
endpoint from the public domain, leveraging it in the hopes of compromising or gaining access to the management domain.
The design of OpenStack is such that separation of security domains is difficult - as core services will usually bridge at least two domains, special consideration must be given when applying security controls to them.

Threat classification, actors and attack vectors

Most types of cloud deployment, public or private, are exposed to some
form of attack. In this chapter we categorize attackers and summarize potential types of attacks in each security domain.

Outbound attacks and reputational risk

Careful consideration should be given to potential outbound abuse from
a cloud deployment. Whether public or private, clouds tend to have lots
of resource available. An attacker who has established a point of presence
within the cloud, either through hacking or entitled access, such as rogue
employee, can bring these resources to bear against the internet at large.
Clouds with compute services make for ideal DDoS and brute force engines. The issue is more pressing for public clouds as their users are largely
unaccountable, and can quickly spin up numerous disposable instances for
outbound attacks. Major damage can be inflicted upon a company's reputation if it becomes known for hosting malicious software or launching
attacks on other networks. Methods of prevention include egress security
groups, outbound traffic inspection, customer education and awareness,
and fraud and abuse mitigation strategies.

Attack types
The diagram shows the types of attacks that may be expected from the actors described in the previous section. Note that there will always be exceptions to this diagram but in general, this describes the sorts of attack
that could be typical for each actor.

OpenStack Security Guide

May 1, 2015

current

Figure1.1.Attack types

The prescriptive defense for each form of attack is beyond the scope of
this document. The above diagram can assist you in making an informed
decision about which types of threats, and threat actors, should be protected against. For commercial public cloud deployments this might include prevention against serious crime. For those deploying private clouds
for government use, more stringent protective mechanisms should be in
place, including carefully protected facilities and supply chains. In contrast
those standing up basic development or test environments will likely require less restrictive controls (middle of the spectrum).

OpenStack Security Guide

May 1, 2015

current

Introduction to case studies

The level of detail contained in this type of table can be beneficial as the
information can immediately inform, guide, and assist with validating security requirements. Standard security components such as firewall configuration, service port conflicts, security remediation areas, and compliance
become easier to maintain when concise information is available. An example of this type of table is provided below:

Referencing a table of services, protocols and ports can help in understanding the relationship between OpenStack components. It is highly recommended that OpenStack deployments have information similar to this
on record.

Case studies
Earlier in the section called Introduction to case studies [21] we introduced the Alice and Bob case studies where Alice is deploying a government cloud and Bob is deploying a public cloud each with different security
requirements. Here we discuss how Alice and Bob would address their system documentation requirements. The documentation suggested above
includes hardware and software records, network diagrams, and system
configuration details.

Alice's private cloud

Alice needs detailed documentation to satisfy FedRAMP requirements. She
sets up a configuration management database (CMDB) to store information regarding all of the hardware, firmware, and software versions used
throughout the cloud. She also creates a network diagram detailing the
cloud architecture, paying careful attention to the security domains and
the services that span multiple security domains.
Alice also needs to record each network service running in the cloud, what
interfaces and ports it binds to, the security domains for each service, and
why the service is needed. Alice decides to build automated tools to log into each system in the cloud over secure shell (SSH) using the Python Fabric
library. The tools collect and store the information in the CMDB, which simplifies the audit process.
25

OpenStack Security Guide

May 1, 2015

Bob's public cloud

In this case, Bob will approach these steps the same as Alice.

current

OpenStack Security Guide

May 1, 2015

current

3. Management
Continuous systems management ........................................................
Integrity life-cycle .................................................................................
Management interfaces .......................................................................
Case studies .........................................................................................

27
32
39
43

A cloud deployment is a living system. Machines age and fail, software becomes outdated, vulnerabilities are discovered. When errors or omissions
are made in configuration, or when software fixes must be applied, these
changes must be made in a secure, but convenient, fashion. These changes
are typically solved through configuration management.
Likewise, it is important to protect the cloud deployment from being configured or manipulated by malicious entities. With many systems in a cloud
employing compute and networking virtualization, there are distinct challenges applicable to OpenStack which must be addressed through integrity
lifecycle management.
Finally, administrators must perform command and control over the cloud
for various operational functions. It is important these command and control facilities are understood and secured.

Continuous systems management

A cloud will always have bugs. Some of these will be security problems.
For this reason, it is critically important to be prepared to apply security
updates and general software updates. This involves smart use of configuration management tools, which are discussed below. This also involves
knowing when an upgrade is necessary.

Vulnerability management
For announcements regarding security relevant changes, subscribe to the
OpenStack Announce mailing list. The security notifications are also posted through the downstream packages, for example, through Linux distributions that you may be subscribed to as part of the package updates.
The OpenStack components are only a small fraction of the software in
a cloud. It is important to keep up to date with all of these other components, too. While certain data sources will be deployment specific, it is important that a cloud administrator subscribe to the necessary mailing lists
in order to receive notification of any security updates applicable to the
27

OpenStack Security Guide

May 1, 2015

current

organization's environment. Often this is as simple as tracking an upstream

Linux distribution.

Note
OpenStack releases security information through two channels.
OpenStack Security Advisories (OSSA) are created by
the OpenStack Vulnerability Management Team (VMT).
They pertain to security holes in core OpenStack services.
More information on the VMT can be found here: https://
wiki.openstack.org/wiki/Vulnerability_Management
OpenStack Security Notes (OSSN) are created by the OpenStack Security Group (OSSG) to support the work of the
VMT. OSSN address issues in supporting software and
common deployment configurations. They are referenced
throughout this guide. Security Notes are archived at
https://launchpad.net/ossn/

Triage
After you are notified of a security update, the next step is to determine
how critical this update is to a given cloud deployment. In this case, it is
useful to have a pre-defined policy. Existing vulnerability rating systems
such as the common vulnerability scoring system (CVSS) v2 do not properly
account for cloud deployments.
In this example we introduce a scoring matrix that places vulnerabilities
in three categories: Privilege Escalation, Denial of Service and Information
Disclosure. Understanding the type of vulnerability and where it occurs in
your infrastructure will enable you to make reasoned response decisions.
Privilege Escalation describes the ability of a user to act with the privileges of some other user in a system, bypassing appropriate authorization
checks. A guest user performing an operation that allows them to conduct
unauthorized operations with the privileges of an administrator is an example of this type of vulnerability.
Denial of Service refers to an exploited vulnerability that may cause service or system disruption. This includes both distributed attacks to overwhelm network resources, and single-user attacks that are typically caused
through resource allocation bugs or input induced system failure flaws.
Information Disclosure vulnerabilities reveal information about your system or operations. These vulnerabilities range from debugging informa-

OpenStack Security Guide

May 1, 2015

current

tion disclosure, to exposure of critical security data, such as authentication

credentials and passwords.
Attacker position / Privilege level
External

Cloud user

Secure bootstrapping
Nodes in the cloudincluding compute, storage, network, service, and hybrid nodesshould have an automated provisioning process. This ensures
that nodes are provisioned consistently and correctly. This also facilitates
security patching, upgrading, bug fixing, and other critical changes. Since
this process installs new software that runs at the highest privilege levels
in the cloud, it is important to verify that the correct software is installed.
This includes the earliest stages of the boot process.
There are a variety of technologies that enable verification of these early
boot stages. These typically require hardware support such as the trusted
platform module (TPM), Intel Trusted Execution Technology (TXT), dynamic root of trust measurement (DRTM), and Unified Extensible Firmware In32

OpenStack Security Guide

May 1, 2015

current

terface (UEFI) secure boot. In this book, we will refer to all of these collectively as secure boot technologies. We recommend using secure boot,
while acknowledging that many of the pieces necessary to deploy this require advanced technical skills in order to customize the tools for each environment. Utilizing secure boot will require deeper integration and customization than many of the other recommendations in this guide. TPM
technology, while common in most business class laptops and desktops for
several years, and is now becoming available in servers together with supporting BIOS. Proper planning is essential to a successful secure boot deployment.
A complete tutorial on secure boot deployment is beyond the scope of this
book. Instead, here we provide a framework for how to integrate secure
boot technologies with the typical node provisioning process. For additional details, cloud architects should refer to the related specifications and
software configuration manuals.

Node provisioning
Nodes should use Preboot eXecution Environment (PXE) for provisioning.
This significantly reduces the effort required for redeploying nodes. The
typical process involves the node receiving various boot stagesthat is progressively more complex software to execute from a server.

We recommend using a separate, isolated network within the management security domain for provisioning. This network will handle all PXE
traffic, along with the subsequent boot stage downloads depicted above.
33

OpenStack Security Guide

May 1, 2015

current

Note that the node boot process begins with two insecure operations:
DHCP and TFTP. Then the boot process uses TLS to download the remaining information required to deploy the node. This may be an operating system installer, a basic install managed by Chef or Puppet, or even a complete file system image that is written directly to disk.
While utilizing TLS during the PXE boot process is somewhat more challenging, common PXE firmware projects, such as iPXE, provide this support. Typically this involves building the PXE firmware with knowledge of
the allowed TLS certificate chain(s) so that it can properly validate the server certificate. This raises the bar for an attacker by limiting the number of
insecure, plain text network operations.

Verified boot
In general, there are two different strategies for verifying the boot process. Traditional secure boot will validate the code run at each step in the
process, and stop the boot if code is incorrect. Boot attestation will record
which code is run at each step, and provide this information to another
machine as proof that the boot process completed as expected. In both
cases, the first step is to measure each piece of code before it is run. In this
context, a measurement is effectively a SHA-1 hash of the code, taken before it is executed. The hash is stored in a platform configuration register
(PCR) in the TPM.
Note: SHA-1 is used here because this is what the TPM chips support.
Each TPM has at least 24 PCRs. The TCG Generic Server Specification, v1.0,
March 2005, defines the PCR assignments for boot-time integrity measurements. The table below shows a typical PCR configuration. The context indicates if the values are determined based on the node hardware
(firmware) or the software provisioned onto the node. Some values are influenced by firmware versions, disk sizes, and other low-level information.
Therefore, it is important to have good practices in place around configuration management to ensure that each system deployed is configured exactly as desired.
Register

What is measured

PCR-00

Core Root of Trust Mea- Hardware

surement (CRTM), BIOS
code, Host platform extensions

PCR-01

Host platform configura- Hardware

tion

PCR-02

OpenStack Security Guide

May 1, 2015

current

stalled provides a solid foundation on which to build the rest of the node
software stack.
Depending on the strategy selected, in the event of a failure the node will
either fail to boot or it can report the failure back to another entity in the
cloud. For secure boot, the node will fail to boot and a provisioning service within the management security domain must recognize this and log
the event. For boot attestation, the node will already be running when the
failure is detected. In this case the node should be immediately quarantined by disabling its network access. Then the event should be analyzed
for the root cause. In either case, policy should dictate how to proceed after a failure. A cloud may automatically attempt to re-provision a node a
certain number of times. Or it may immediately notify a cloud administrator to investigate the problem. The right policy here will be deployment
and failure mode specific.

Node hardening
At this point we know that the node has booted with the correct kernel
and underlying components. There are many paths for hardening a given operating system deployment. The specifics on these steps are outside
of the scope of this book. We recommend following the guidance from a
hardening guide specific to your operating system. For example, the security technical implementation guides (STIG) and the NSA guides are useful
starting places.
The nature of the nodes makes additional hardening possible. We recommend the following additional steps for production nodes:
Use a read-only file system where possible. Ensure that writeable file systems do not permit execution. This can be handled through the mount
options provided in /etc/fstab.
Use a mandatory access control policy to contain the instances, the node
services, and any other critical processes and data on the node. See the
discussions on sVirt / SELinux and AppArmor below.
Remove any unnecessary software packages. This should result in a very
stripped down installation because a compute node has a relatively
small number of dependencies.
Finally, the node kernel should have a mechanism to validate that the rest
of the node starts in a known good state. This provides the necessary link
from the boot validation process to validating the entire system. The steps
for doing this will be deployment specific. As an example, a kernel mod36

OpenStack Security Guide

May 1, 2015

current

ule could verify a hash over the blocks comprising the file system before
mounting it using dm-verity.

Runtime verification
Once the node is running, we need to ensure that it remains in a good
state over time. Broadly speaking, this includes both configuration management and security monitoring. The goals for each of these areas are
different. By checking both, we achieve higher assurance that the system
is operating as desired. We discuss configuration management in the management section, and security monitoring below.

Intrusion detection system

Host-based intrusion detection tools are also useful for automated validation of the cloud internals. There are a wide variety of host-based intrusion
detection tools available. Some are open source projects that are freely
available, while others are commercial. Typically these tools analyze data from a variety of sources and produce security alerts based on rule sets
and/or training. Typical capabilities include log analysis, file integrity checking, policy monitoring, and rootkit detection. More advanced -- often custom -- tools can validate that in-memory process images match the on-disk
executable and validate the execution state of a running process.
One critical policy decision for a cloud architect is what to do with the output from a security monitoring tool. There are effectively two options. The
first is to alert a human to investigate and/or take corrective action. This
could be done by including the security alert in a log or events feed for
cloud administrators. The second option is to have the cloud take some
form of remedial action automatically, in addition to logging the event.
Remedial actions could include anything from re-installing a node to performing a minor service configuration. However, automated remedial action can be challenging due to the possibility of false positives.
False positives occur when the security monitoring tool produces a security alert for a benign event. Due to the nature of security monitoring tools,
false positives will most certainly occur from time to time. Typically a cloud
administrator can tune security monitoring tools to reduce the false positives, but this may also reduce the overall detection rate at the same time.
These classic trade-offs must be understood and accounted for when setting up a security monitoring system in the cloud.
The selection and configuration of a host-based intrusion detection tool is
highly deployment specific. We recommend starting by exploring the fol37

OpenStack Security Guide

May 1, 2015

current

lowing open source projects which implement a variety of host-based intrusion detection and file monitoring features.
OSSEC
Samhain
Tripwire
AIDE
Network intrusion detection tools complement the host-based tools. OpenStack doesn't have a specific network IDS built-in, but OpenStack Networking provides a plug-in mechanism to enable different technologies through
the Networking API. This plug-in architecture will allow tenants to develop
API extensions to insert and configure their own advanced networking services like a firewall, an intrusion detection system, or a VPN between the
VMs.
Similar to host-based tools, the selection and configuration of a network-based intrusion detection tool is deployment specific. Snort is the
leading open source networking intrusion detection tool, and a good starting place to learn more.
There are a few important security considerations for network and hostbased intrusion detection systems.
It is important to consider the placement of the Network IDS on the
cloud (for example, adding it to the network boundary and/or around
sensitive networks). The placement depends on your network environment but make sure to monitor the impact the IDS may have on your
services depending on where you choose to add it. Encrypted traffic,
such as TLS, cannot generally be inspected for content by a Network
IDS. However, the Network IDS may still provide some benefit in identifying anomalous unencrypted traffic on the network.
In some deployments it may be required to add host-based IDS on sensitive components on security domain bridges. A host-based IDS may detect anomalous activity by compromised or unauthorized processes on
the component. The IDS should transmit alert and log information on
the Management network.

Server hardening
Servers in the cloud, including undercloud and overcloud infrastructure,
should implement hardening best practices. As OS and server hardening is
38

OpenStack Security Guide

May 1, 2015

It has become industry practice to use secure shell (SSH) access for the
management of Linux and Unix systems. SSH uses secure cryptographic
primitives for communication. With the scope and importance of SSH in
typical OpenStack deployments, it is important to understand best practices for deploying SSH.

Host key fingerprints

Often overlooked is the need for key management for SSH hosts. As most
or all hosts in an OpenStack deployment will provide an SSH service, it
is important to have confidence in connections to these hosts. It cannot
be understated that failing to provide a reasonably secure and accessible
method to verify SSH host key fingerprints is ripe for abuse and exploitation.
41

OpenStack Security Guide

May 1, 2015

current

All SSH daemons have private host keys and, upon connection, offer a host
key fingerprint. This host key fingerprint is the hash of an unsigned public key. It is important these host key fingerprints are known in advance
of making SSH connections to those hosts. Verification of host key fingerprints is instrumental in detecting man-in-the-middle attacks.
Typically, when an SSH daemon is installed, host keys will be generated. It
is necessary that the hosts have sufficient entropy during host key generation. Insufficient entropy during host key generation can result in the possibility to eavesdrop on SSH sessions.
Once the SSH host key is generated, the host key fingerprint should be
stored in a secure and queryable location. One particularly convenient solution is DNS using SSHFP resource records as defined in RFC-4255. For this
to be secure, it is necessary that DNSSEC be deployed.

Management utilities
The OpenStack Management Utilities are open-source Python command-line clients that make API calls. There is a client for each OpenStack
service (for example, nova, glance). In addition to the standard CLI
client, most of the services have a management command-line utility which
makes direct calls to the database. These dedicated management utilities
are slowly being deprecated.

Security considerations
The dedicated management utilities (*-manage) in some cases use the
direct database connection.
Ensure that the .rc file which has your credential information is secured.

References
OpenStack End User Guide section command-line clients overview
OpenStack End User Guide section Download and source the OpenStack
RC file

Out-of-band management interface

OpenStack management relies on out-of-band management interfaces
such as the IPMI protocol to access into nodes running OpenStack components. IPMI is a very popular specification to remotely manage, diagnose,
42

OpenStack Security Guide

May 1, 2015

current

and reboot servers whether the operating system is running or the system
has crashed.

Security considerations
Use strong passwords and safeguard them, or use client-side TLS authentication.
Ensure that the network interfaces are on their own
private(management or a separate) network. Segregate management
domains with firewalls or other network gear.
If you use a web interface to interact with the BMC/IPMI, always use the
TLS interface, such as HTTPS or port 443. This TLS interface should NOT
use self-signed certificates, as is often default, but should have trusted certificates using the correctly defined fully qualified domain names
(FQDNs).
Monitor the traffic on the management network. The anomalies might
be easier to track than on the busier compute nodes.
Out of band management interfaces also often include graphical machine
console access. It is often possible, although not necessarily default, that
these interfaces are encrypted. Consult with your system software documentation for encrypting these interfaces.

References
Hacking servers that are turned off

Case studies
Previously we discussed typical OpenStack management interfaces and associated backplane issues. We will now approach these issues by returning
to the Alice and Bob case studies (See the section called Introduction to
case studies [21] ) where Alice is deploying a government cloud and Bob
is deploying a public cloud each with different security requirements. In
this section, we will look into how both Alice and Bob will address:
Cloud administration
Self service
Data replication and recovery
43

OpenStack Security Guide

May 1, 2015

current

SLA and security monitoring

Alice's private cloud

When building her private cloud, while air-gapped, Alice still needs to consider her service management interfaces. Before deploying her private
cloud, Alice has completed her system documentation. Specifically she
has identified which OpenStack services will exist in each security domain.
From there Alice has further restricted access to management interfaces by
deploying a combination of IDS, TLS encryption, and physical network isolation. Additionally, Alice requires high availability and redundant services.
Thus, Alice sets up redundant infrastructure for various OpenStack API services.
Alice also needs to provide assurances that the physical servers and hypervisors have been built from a known secure state into a well-defined
configuration. To enable this, Alice uses a combination of a Configuration
Management platform to configure each machine according to the standards and regulations she must comply with. It will also enable Alice to report periodically on the state of her cloud and perform remediation to a
known state should anything be out of the ordinary. Additionally, Alice
provides hardware assurances by using a PXE system to build her nodes
from a known set of base images. During the boot process, Alice provides
further assurances by enabling Intel TXT and related trusted boot technologies provided by the hardware.

Bob's public cloud

As a public cloud provider, Bob is concerned with both the continuous
availability of management interfaces and the security of transactions to
the management interfaces. To that end Bob implements multiple redundant OpenStack API endpoints for the services his cloud will run. Additionally on the public network Bob uses TLS to encrypt all transactions between his customers and his cloud interfaces. To isolate his cloud operations Bob has physically isolated his management, instance migration, and
storage networks.
To ease scaling and reduce management overhead Bob implements a configuration management system. For customer data assurances, Bob offers a backup as a service product as requirements will vary between customers. Finally, Bob does not provide a "baremetal" or the ability to schedule an entire node, so to reduce management overhead and increase operational efficiency Bob does not implement any node boot time security.

OpenStack Security Guide

May 1, 2015

current

4. Secure communication
Introduction to TLS and SSL .................................................................
TLS proxies and HTTP services ..............................................................
Secure reference architectures ..............................................................
Case studies .........................................................................................

45
48
55
59

Inter-device communication is an issue still plaguing security researchers.

Between large project errors such as Heartbleed or more advanced attacks
such as BEAST and CRIME, secure methods of communication over a network are becoming more important. It should be remembered, however
that encryption should be applied as one part of a larger security strategy.
The compromise of an endpoint means that an attacker no longer needs
to break the encryption used, but is able to view and manipulate messages
as they are processed by the system.
This chapter will review several features around configuring TLS to secure
both internal and external resources, and will call out specific categories of
systems that should be given specific attention.

Introduction to TLS and SSL

There are a number of situations where there is a security requirement to
assure the confidentiality or integrity of network traffic in an OpenStack
deployment. This is generally achieved using cryptographic measures, such
as the Transport Layer Security (TLS) protocol.
In a typical deployment all traffic transmitted over public networks is secured, but security best practice dictates that internal traffic must also be
secured. It is insufficient to rely on security domain separation for protection. If an attacker gains access to the hypervisor or host resources, compromises an API endpoint, or any other service, they must not be able to
easily inject or capture messages, commands, or otherwise affect the management capabilities of the cloud.
All domains should be secured with TLS, including the management domain services and intra-service communications. TLS provides the mechanisms to ensure authentication, non-repudiation, confidentiality, and integrity of user communications to the OpenStack services and between the
OpenStack services themselves.
Due to the published vulnerabilities in the Secure Sockets Layer (SSL) protocols, we strongly recommend that TLS is used in preference to SSL, and
45

OpenStack Security Guide

May 1, 2015

current

that SSL is disabled in all cases, unless compatibility with obsolete browsers
or libraries is required.
Public Key Infrastructure (PKI) is the framework for securing communication in a network. It consists of a set of systems and processes to ensure
traffic can be sent securely while validating the identities of the parties.
The core components of PKI are:
End entity

User, process, or system that is the subject of a certificate.

Certification Authority (CA)

Defines certificate policies, management, and issuance of certificates.

Registration Authority (RA)

An optional system to which a CA delegates certain management functions.

Repository

Where the end entity certificates and

certificate revocation lists are stored
and looked up - sometimes referred to
as the certificate bundle.

Relying party

The endpoint that is trusting that the

CA is valid.

PKI builds the framework on which to provide encryption algorithms,

cipher modes, and protocols for securing data and authentication. We
strongly recommend securing all services with Public Key Infrastructure
(PKI), including the use of TLS for API endpoints. It is impossible for the encryption or signing of transports or messages alone to solve all these problems. Hosts themselves must be secure and implement policy, namespaces,
and other controls to protect their private credentials and keys. However,
the challenges of key management and protection do not reduce the necessity of these controls, or lessen their importance.

Certification authorities
Many organizations have an established Public Key Infrastructure with
their own certification authority (CA), certificate policies, and management for which they should use to issue certificates for internal OpenStack
users or services. Organizations in which the public security domain is Internet facing will additionally need certificates signed by a widely recognized
public CA. For cryptographic communications over the management network, it is recommended one not use a public CA. Instead, we expect and
recommend most deployments deploy their own internal CA.
46

OpenStack Security Guide

May 1, 2015

current

It is recommended that the OpenStack cloud architect consider using separate PKI deployments for internal systems and customer facing services.
This allows the cloud deployer to maintain control of their PKI infrastructure and among other things makes requesting, signing and deploying certificates for internal systems easier. Advanced configurations may use separate PKI deployments for different security domains. This allows deployers
to maintain cryptographic separation of environments, ensuring that certificates issued to one are not recognized by another.
Certificates used to support TLS on internet facing cloud endpoints (or customer interfaces where the customer is not expected to have installed anything other than standard operating system provided certificate bundles)
should be provisioned using Certificate Authorities that are installed in the
operating system certificate bundle. Typical well known vendors include
Verisign and Thawte but many others exist.
There are many management, policy, and technical challenges around creating and signing certificates. This is an area where cloud architects or operators may wish to seek the advice of industry leaders and vendors in addition to the guidance recommended here.

TLS libraries
Various components, services, and applications within the OpenStack
ecosystem or dependencies of OpenStack are implemented and can be
configured to use TLS libraries. The TLS and HTTP services within OpenStack are typically implemented using OpenSSL which has a module that
has been validated for FIPS 140-2. However, keep in mind that each application or service can still introduce weaknesses in how they use the
OpenSSL libraries.

Cryptographic algorithms, cipher modes, and protocols

We recommend only using TLSv1.2. TLSv1.0 and TLSv1.1 may be used for
broad client compatibility but we recommend using caution and only enabling these protocols if you have a strong requirement to do so. Other
TLS versions, explicitly older versions, should not be used. These older versions include SSLv2 which is deprecated, and SSLv3 which suffers from the
attack known as POODLE. When using TLS v1.2 and in control of both the
clients and the server, the cipher suite should be limited to ECDHE-ECDSA-AES256-GCM-SHA384. Where you don't control both ends and are
47

OpenStack Security Guide

May 1, 2015

current

using TLS v1+, the more general HIGH:!aNULL:!eNULL:!DES:!3DES is

a reasonable cipher selection.
However, as this book does not intend to be a thorough reference on cryptography we do not wish to be prescriptive about what specific algorithms
or cipher modes you should enable or disable in your OpenStack services.
However, there are some authoritative references we would like to recommend for further information:
National Security Agency, Suite B Cryptography
OWASP Guide to Cryptography
OWASP Transport Layer Protection Cheat Sheet
SoK: SSL and HTTPS: Revisiting past challenges and evaluating certificate
trust model enhancements
The Most Dangerous Code in the World: Validating SSL Certificates in
Non-Browser Software
OpenSSL and FIPS 140-2

Summary
Given the complexity of the OpenStack components and the number of
deployment possibilities, you must take care to ensure that each component gets the appropriate configuration of TLS certificates, keys, and CAs.
Subsequent sections discuss the following services:
Compute API endpoints
Identity API endpoints
Networking API endpoints
Storage API endpoints
Messaging server
Database server
Dashboard

TLS proxies and HTTP services

OpenStack endpoints are HTTP services providing APIs to both end-users
on public networks and to other OpenStack services on the management
48

OpenStack Security Guide

May 1, 2015

current

network. It is highly recommended that all of these requests, both internal and external, operate over TLS. To achieve this goal, API services must
be deployed behind a TLS proxy that can establish and terminate TLS sessions. The following table offers a non-exhaustive list of open source software that can be used for this purpose:
Pound
Stud
nginx
Apache httpd
In cases where software termination offers insufficient performance, hardware accelerators may be worth exploring as an alternative option. It is important to be mindful of the size of requests that will be processed by any
chosen TLS proxy.

Examples
Below we provide sample recommended configuration settings for enabling TLS in some of the more popular web servers/TLS terminators.
Before we delve into the configurations, we briefly discuss the ciphers' configuration element and its format. A more exhaustive treatment on available ciphers and the OpenSSL cipher list format can be found at: ciphers.
ciphers = "HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM"

or
ciphers = "kEECDH:kEDH:kRSA:HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!
LOW:!MEDIUM"

Cipher string options are separated by ":", while "!" provides negation of
the immediately following element. Element order indicates preference
unless overridden by qualifiers such as HIGH. Let us take a closer look at
the elements in the above sample strings.
kEECDH:kEDH

Ephemeral Elliptic Curve Diffie-Hellman (abbreviated

as EECDH and ECDHE).
Ephemeral Diffie-Hellman (abbreviated either as EDH
or DHE) uses prime field groups.
49

OpenStack Security Guide

May 1, 2015

current

Both approaches provide Perfect Forward Secrecy

(PFS). See the section called Perfect forward secrecy [54] for additional discussion on properly configuring PFS.
Ephemeral Elliptic Curves require the server to be configured with a named curve, and provide better security than prime field groups and at lower computational
cost. However, prime field groups are more widely implemented, and thus typically both are included in list.
kRSA

Cipher suites using the RSA exchange, authentication

or either respectively.

HIGH

Selects highest possible security cipher in the negotiation phase. These typically have keys of length 128 bits
or longer.

!RC4

No RC4. RC4 has flaws in the context of TLS V3. See

On the Security of RC4 in TLS and WPA.

!MD5

No MD5. MD5 is not collision resistant, and thus not

acceptable for Message Authentication Codes (MAC)
or signatures.

!aNULL:!eNULL

Disallows clear text.

!EXP

Disallows export encryption algorithms, which by design tend to be weak, typically using 40 and 56 bit
keys.
US Export restrictions on cryptography systems have
been lifted and no longer need to be supported.

!LOW:!MEDIUM

Disallows low (56 or 64 bit long keys) and medium

(128 bit long keys) ciphers because of their vulnerability to brute force attacks (example 2-DES). This rule
still permits Triple Data Encryption Standard (Triple
DES) also known as Triple Data Encryption Algorithm
(TDEA) and the Advanced Encryption Standard (AES),
each of which has keys greater than equal to 128 bits
and thus more secure.

Protocols

Protocols are enabled/disabled through

SSL_CTX_set_options. We recommend disabling SSLv2/v3 and enabling TLS.

May 1, 2015

current

Compute API SSL endpoint in Apache, which you must pair with a short
WSGI script.
<VirtualHost <ip address>:8447>
ServerName <site FQDN>
SSLEngine On
SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM
SSLCertificateFile
/path/<site FQDN>.crt
SSLCACertificateFile /path/<site FQDN>.crt
SSLCertificateKeyFile /path/<site FQDN>.key
SSLSessionTickets Off
WSGIScriptAlias / <WSGI script location>
WSGIDaemonProcess osapi user=<user> group=<group> processes=3
threads=10
<Directory <WSGI dir>>
# For http server 2.2 and earlier:
Order allow,deny
Allow from all
# Or, in Apache http server 2.4 and later:
# Require all granted
</Directory>
</VirtualHost>

HTTP strict transport security

We recommend that all production deployments use HTTP strict transport
security (HSTS). This header prevents browsers from making insecure connections after they have made a single secure one. If you have deployed
your HTTP services on a public or an untrusted domain, HSTS is especially
important. To enable HSTS, configure your web server to send a header
like this with all requests:
Strict-Transport-Security: max-age=31536000; includeSubDomains

Start with a short timeout of 1 day during testing, and raise it to one year
after testing has shown that you have not introduced problems for users.
Note that once this header is set to a large timeout, it is (by design) very
difficult to disable.

Perfect forward secrecy

Configuring TLS servers for perfect forward secrecy requires careful planning around key size, session IDs, and session tickets. In addition, for multi-server deployments, shared state is also an important consideration. The
example configurations for Apache and Nginx above disable the session
54

OpenStack Security Guide

What about high availability or load balanced deployments that need to
inspect traffic? The previous deployment model (SSL/TLS on same physical hosts as API endpoints) would not allow for deep packet inspection
since the traffic is encrypted. If the traffic only needs to be inspected for
basic routing purposes, it might not be necessary for the load balancer to
have access to the unencrypted traffic. HAProxy has the ability to extract
the SSL/TLS session ID during the handshake, which can then be used to
achieve session affinity (configuration details here). HAProxy can also use
the TLS Server Name Indication (SNI) extension to determine where traffic
should be routed to (configuration details here). These features likely cover some of the most common load balancer needs. HAProxy would be able
57

OpenStack Security Guide

May 1, 2015

current

to just pass the HTTPS traffic straight through to the API endpoint systems
in this case:

Cryptographic seperation of external and internal

environments
What if you want cryptographic separation of your external and internal environments? A public cloud provider would likely want their public
facing services (or proxies) to use certificates that are issued by a CA that
chains up to a trusted Root CA that is distributed in popular web browser
software for SSL/TLS. For the internal services, one might want to instead
use their own PKI to issue certificates for SSL/TLS. This cryptographic separation can be accomplished by terminating SSL at the network boundary,
then re-encrypting using the internally issued certificates. The traffic will
be unencrypted for a brief period on the public facing SSL/TLS proxy, but
it will never be transmitted over the network in the clear. The same re-encryption approach that is used to achieve cryptographic separation can also be used if deep packet inspection is really needed on a load balancer.
Here is what this deployment model would look like:

As with most things, there are trade-offs. The main trade-off is going to be
between security and performance. Encryption has a cost, but so does being hacked. The security and performance requirements are going to be
58

OpenStack Security Guide

May 1, 2015

current

different for every deployment, so how SSL/TLS is used will ultimately be

an individual decision.

Case studies
Earlier in the section called Introduction to case studies [21] we introduced the Alice and Bob case study where Alice is deploying a government
cloud and Bob is deploying a public cloud each with different security requirements. Here we discuss how Alice and Bob would address deployment of PKI certification authorities (CA) and certificate management.

Alice's private cloud

Alice as a cloud architect within a government agency knows that her
agency operates its own certification authority. Alice contacts the PKI office in her agency that manages her PKI and certificate issuance. Alice obtains certificates issued by this CA and configures the services within both
the public and management security domains to use these certificates.
Since Alice's OpenStack deployment exists entirely on a disconnected from
the Internet network, she makes sure to remove all default CA bundles
that contain external public CA providers to ensure the OpenStack services
only accept client certificates issued by her agency's CA.

Bob's public cloud

Bob is architecting a public cloud and needs to ensure that the publicly facing OpenStack services are using certificates issued by a major public CA.
Bob acquires certificates for his public OpenStack services and configures
the services to use PKI and TLS and includes the public CAs in his trust bundle for the services. Additionally, Bob also wants to further isolate the internal communications amongst the services within the management security domain. Bob contacts the team within his organization that is responsible for managing his organization's PKI and issuance of certificates using
their own internal CA. Bob obtains certificates issued by this internal CA
and configures the services that communicate within the management security domain to use these certificates and configures the services to only
accept client certificates issued by his internal CA.

OpenStack Security Guide

May 1, 2015

current

5. API endpoints
API endpoint configuration recommendations ...................................... 61
Case studies ......................................................................................... 63
The process of engaging an OpenStack cloud is started through the querying of an API endpoint. While there are different challenges for public and
private endpoints, these are high value assets that can pose a significant
risk if compromised.
This chapter recommends security enhancements for both public and private-facing API endpoints.

API endpoint configuration recommendations

Internal API communications
OpenStack provides both public facing and private API endpoints. By default, OpenStack components use the publicly defined endpoints. The recommendation is to configure these components to use the API endpoint
within the proper security domain.
Services select their respective API endpoints based on the OpenStack service catalog. These services might not obey the listed public or internal API
end point values. This can lead to internal management traffic being routed to external API endpoints.

Configure internal URLs in the Identity service catalog

The Identity service catalog should be aware of your internal URLs. While
this feature is not utilized by default, it may be leveraged through configuration. Additionally, it should be forward-compatible with expectant
changes once this behavior becomes the default.
To register an internal URL for an endpoint:
$ keystone endpoint-create \
--region RegionOne \
--service-id=1ff4ece13c3e48d8a6461faebd9cd38f \
--publicurl='https://public-ip:8776/v1/%(tenant_id)s' \
--internalurl='https://management-ip:8776/v1/%(tenant_id)s' \
--adminurl='https://management-ip:8776/v1/%(tenant_id)s'

OpenStack Security Guide

May 1, 2015

current

Configure applications for internal URLs

You can force some services to use specific API endpoints. Therefore, it is
recommended that each OpenStack service communicating to the API of
another service must be explicitly configured to access the proper internal
API endpoint.
Each project may present an inconsistent way of defining target API endpoints. Future releases of OpenStack seek to resolve these inconsistencies
through consistent use of the Identity service catalog.

Configuration example #1: nova

[DEFAULT]
cinder_catalog_info='volume:cinder:internalURL'
glance_protocol='https'
neutron_url='https://neutron-host:9696'
neutron_admin_auth_url='https://neutron-host:9696'
s3_host='s3-host'
s3_use_ssl=True

Configuration example #2: cinder

glance_host='https://glance-server'

Paste and middleware

Most API endpoints and other HTTP services in OpenStack use the Python
Paste Deploy library. From a security perspective, this library enables manipulation of the request filter pipeline through the application's configuration. Each element in this chain is referred to as middleware. Changing
the order of filters in the pipeline or adding additional middleware might
have unpredictable security impact.
Commonly, implementers add middleware to extend OpenStack's base
functionality. We recommend implementers make careful consideration of
the potential exposure introduced by the addition of non-standard software components to their HTTP request pipeline.
For more information about Paste Deploy, see http://pythonpaste.org/deploy/.

API endpoint process isolation and policy

You should isolate API endpoint processes, especially those that reside
within the public security domain should be isolated as much as possible.
62

OpenStack Security Guide

May 1, 2015

current

Where deployments allow, API endpoints should be deployed on separate

hosts for increased isolation.

Namespaces
Many operating systems now provide compartmentalization support. Linux supports namespaces to assign processes into independent domains.
Other parts of this guide cover system compartmentalization in more detail.

Network policy
Because API endpoints typically bridge multiple security domains, you must
pay particular attention to the compartmentalization of the API processes.
See the section called Bridging security domains [15] for additional information in this area.
With careful modeling, you can use network ACLs and IDS technologies to
enforce explicit point to point communication between network services.
As a critical cross domain service, this type of explicit enforcement works
well for OpenStack's message queue service.
To enforce policies, you can configure services, host-based firewalls (such
as iptables), local policy (SELinux or AppArmor), and optionally global network policy.

Mandatory access controls

You should isolate API endpoint processes from each other and other processes on a machine. The configuration for those processes should be restricted to those processes not only by Discretionary Access Controls, but
through Mandatory Access Controls. The goal of these enhanced access
controls is to aid in the containment and escalation of API endpoint security breaches. With mandatory access controls, such breaches severely limit
access to resources and provide earlier alerting on such events.

OpenStack Security Guide

May 1, 2015

current

cloud is not publicly accessible, but she is still concerned about securing the
endpoints against improper use. Bob's cloud, being public, must take measures to reduce the risk of attacks by external adversaries.

Alice's private cloud

Alice's organization requires that the security architecture protect the access to the public and private endpoints, so she elects to use the Apache
TLS proxy on both public and internal services. Alice's organization has implemented its own certificate authority. Alice contacts the PKI office in her
agency that manages her PKI and certificate issuance. Alice obtains certificates issued by this CA and configures the services within both the public
and management security domains to use these certificates. Since Alice's
OpenStack deployment exists entirely on a network disconnected from the
Internet, she makes sure to remove all default CA bundles that contain external public CA providers to ensure the OpenStack services only accept
client certificates issued by her agency's CA. Alice has registered all of the
services in the Identity service's catalog, using the internal URLs for access
by internal services. She has installed host-based intrusion detection on all
of the API endpoints.

Bob's public cloud

Bob must also protect the access to the public and private endpoints, so
he elects to use the Apache TLS proxy on both public and internal services. On the public services, he has configured the certificate key files with
certificates signed by a well-known Certificate Authority. He has used his
organization's self-signed CA to sign certificates in the internal services on
the Management network. Bob has registered his services in the Identity service's catalog, using the internal URLs for access by internal services.
Bob's public cloud runs services on SELinux, which he has configured with a
mandatory access control policy to reduce the impact of any publicly accessible services that may be compromised. He has also configured the endpoints with a host-based IDS.

OpenStack Security Guide

May 1, 2015

current

6. Identity
Authentication .....................................................................................
Authentication methods ......................................................................
Authorization ......................................................................................
Policies .................................................................................................
Tokens .................................................................................................
Future ..................................................................................................
Federated Identity ...............................................................................
Checklist ..............................................................................................

65
66
68
70
72
73
74
85

Identity service (keystone) provides identity, token, catalog, and policy services for use specifically by services in the OpenStack family. Identity service
is organized as a group of internal services exposed on one or many endpoints. Many of these services are used in a combined fashion by the frontend, for example an authenticate call will validate user/project credentials
with the identity service and, upon success, create and return a token with
the token service. Further information can be found by reading the Keystone Developer Documentation.

Authentication
The OpenStack Identity service (keystone) supports multiple methods of
authentication, including user name & password, LDAP, and external authentication methods. Upon successful authentication, The Identity service
provides the user with an authorization token used for subsequent service
requests.
Transport Layer Security (TLS) provides authentication between services
and persons using X.509 certificates. Although the default mode for TLS is
server-side only authentication, certificates may also be used for client authentication.

Invalid login attempts

The Identity service does not provide a method to limit access to accounts
after repeated unsuccessful login attempts. A pattern of repetitive failed
login attempts is generally an indicator of brute-force attacks (refer to Figure1.1, Attack types [20]). This type of attack is more prevalent in public
cloud deployments.
Prevention is possible by using an external authentication system that
blocks out an account after some configured number of failed login at65

OpenStack Security Guide

May 1, 2015

current

tempts. The account then may only be unlocked with further side-channel
intervention.
If prevention is not an option, detection can be used to mitigate damage.
Detection involves frequent review of access control logs to identify unauthorized attempts to access accounts. Possible remediation would include
reviewing the strength of the user password, or blocking the network
source of the attack through firewall rules. Firewall rules on the keystone
server that restrict the number of connections could be used to reduce the
attack effectiveness, and thus dissuade the attacker.
In addition, it is useful to examine account activity for unusual login times
and suspicious actions, and take corrective actions such as disabling the account. Oftentimes this approach is taken by credit card providers for fraud
detection and alert.

Multi-factor authentication
Employ multi-factor authentication for network access to privileged user
accounts. The Identity service supports external authentication services
through the Apache web server that can provide this functionality. Servers
may also enforce client-side authentication using certificates.
This recommendation provides insulation from brute force, social engineering, and both spear and mass phishing attacks that may compromise administrator passwords.

Authentication methods
Internally implemented authentication methods
The Identity service can store user credentials in an SQL Database, or may
use an LDAP-compliant directory server. The Identity database may be separate from databases used by other OpenStack services to reduce the risk
of a compromise of the stored credentials.
When you use a user name and password to authenticate, Identity does
not enforce policies on password strength, expiration, or failed authentication attempts as recommended by NIST Special Publication 800-118 (draft).
Organizations that desire to enforce stronger password policies should
consider using Identity extensions or external authentication services.

OpenStack Security Guide

May 1, 2015

current

LDAP simplifies integration of Identity authentication into an

organization's existing directory service and user account management
processes.
Authentication and authorization policy in OpenStack may be delegated
to another service. A typical use case is an organization that seeks to deploy a private cloud and already has a database of employees and users in
an LDAP system. Using this as the authentication authority, requests to the
Identity service are delegated to the LDAP system, which will then authorize or deny based on its policies. Upon successful authentication, the Identity service then generates a token that is used for access to authorized services.
Note that if the LDAP system has attributes defined for the user such as
admin, finance, HR etc, these must be mapped into roles and groups within Identity for use by the various OpenStack services. The /etc/keystone/keystone.conf file maps LDAP attributes to Identity attributes.
The Identity service MUST NOT be allowed to write to LDAP services used
for authentication outside of the OpenStack deployment as this would allow a sufficiently privileged keystone user to make changes to the LDAP
directory. This would allow privilege escalation within the wider organization or facilitate unauthorized access to other information and resources.
In such a deployment, user provisioning would be out of the realm of the
OpenStack deployment.

Note
There is an OpenStack Security Note (OSSN) regarding
keystone.conf permissions.
There is an OpenStack Security Note (OSSN) regarding potential DoS attacks.

External authentication methods

Organizations may desire to implement external authentication for compatibility with existing authentication services or to enforce stronger authentication policy requirements. Although passwords are the most common form of authentication, they can be compromised through numerous
methods, including keystroke logging and password compromise. External authentication services can provide alternative forms of authentication
that minimize the risk from weak passwords.
These include:
67

OpenStack Security Guide

May 1, 2015

current

Password policy enforcement: Requires user passwords to conform to

minimum standards for length, diversity of characters, expiration, or
failed login attempts.
Multi-factor authentication: The authentication service requires the user to provide information based on something they have, such as a onetime password token or X.509 certificate, and something they know,
such as a password.
Kerberos

Authorization
The Identity service supports the notion of groups and roles. Users belong
to groups while a group has a list of roles. OpenStack services reference
the roles of the user attempting to access the service. The OpenStack policy enforcer middleware takes into consideration the policy rule associated with each resource then the user's group/roles and association to determine if access is allowed to the requested resource.
The policy enforcement middleware enables fine-grained access control to
OpenStack resources. Only admin users can provision new users and have
access to various management functionality. The cloud users would only be able to spin up instances, attach volumes, and perform other operational tasks.

Establish formal access control policies

Prior to configuring roles, groups, and users, document your required access control policies for the OpenStack installation. The policies should be
consistent with any regulatory or legal requirements for the organization.
Future modifications to the access control configuration should be done
consistently with the formal policies. The policies should include the conditions and processes for creating, deleting, disabling, and enabling accounts, and for assigning privileges to the accounts. Periodically review the
policies and ensure that the configuration is in compliance with approved
policies.

Service authorization
Cloud administrators must define a user with the role of admin for each
service, as described in the OpenStack Cloud Administrator Guide. This ser68

OpenStack Security Guide

May 1, 2015

current

vice account provides the service with the authorization to authenticate

users.
The Compute and Object Storage services can be configured to use the
Identity service to store authentication information. Other options to store
authentication information include the use of the "tempAuth" file, however this should not be deployed in a production environment as the password is displayed in plain text.
The Identity service supports client authentication for TLS which may be
enabled. TLS client authentication provides an additional authentication
factor, in addition to the user name and password, that provides greater
reliability on user identification. It reduces the risk of unauthorized access
when user names and passwords may be compromised. However, there is
additional administrative overhead and cost to issue certificates to users
that may not be feasible in every deployment.

Note
We recommend that you use client authentication with TLS for
the authentication of services to the Identity service.
The cloud administrator should protect sensitive configuration files
from unauthorized modification. This can be achieved with mandatory access control frameworks such as SELinux, including /etc/keystone/keystone.conf and X.509 certificates.
Client authentication with TLS requires certificates be issued to services.
These certificates can be signed by an external or internal certificate authority. OpenStack services check the validity of certificate signatures
against trusted CAs by default and connections will fail if the signature is
not valid or the CA is not trusted. Cloud deployers may use self-signed certificates. In this case, the validity check must be disabled or the certificate
should be marked as trusted. To disable validation of self-signed certificates, set insecure=False in the [filter:authtoken] section in the
/etc/nova/api.paste.ini file. This setting also disables certificates
for other components.

Administrative users
We recommend that admin users authenticate using Identity service and
an external authentication service that supports 2-factor authentication,
such as a certificate. This reduces the risk from passwords that may be
compromised. This recommendation is in compliance with NIST 800-53
69

OpenStack Security Guide

May 1, 2015

current

IA-2(1) guidance in the use of multi-factor authentication for network access to privileged accounts.

End users
The Identity service can directly provide end-user authentication, or can
be configured to use external authentication methods to conform to an
organization's security policies and requirements.

Policies
Each OpenStack service has a policy file in JSON format, called
policy.json. The policy file specifies rules, and the rule that governs
each resource. A resource could be API access, the ability to attach to a volume, or to fire up instances.
The policies can be updated by the cloud administrator to further control
access to the various resources. The middleware could also be further customized. Note that your users must be assigned to groups/roles that you
refer to in your policies.
Below is a snippet of the Block Storage service policy.json file.
{
"context_is_admin": "role:admin",
"admin_or_owner": "is_admin:True or project_id:
%(project_id)s",
"default": "rule:admin_or_owner",
"admin_api": "is_admin:True",
"volume:create": "",
"volume:get_all": "",
"volume:get_volume_metadata": "",
"volume:get_volume_admin_metadata": "rule:admin_api",
"volume:delete_volume_admin_metadata": "rule:admin_api",
"volume:update_volume_admin_metadata": "rule:admin_api",
"volume:get_snapshot": "",
"volume:get_all_snapshots": "",
"volume:extend": "",
"volume:update_readonly_flag": "",
"volume:retype": "",
"volume_extension:types_manage": "rule:admin_api",
"volume_extension:types_extra_specs": "rule:admin_api",
"volume_extension:volume_type_encryption": "rule:admin_api",
"volume_extension:volume_encryption_metadata":
"rule:admin_or_owner",

OpenStack Security Guide

May 1, 2015

current

"volume_extension:extended_snapshot_attributes": "",
"volume_extension:volume_image_metadata": "",
"volume_extension:quotas:show": "",
"volume_extension:quotas:update": "rule:admin_api",
"volume_extension:quota_classes": "",
"volume_extension:volume_admin_actions:reset_status":
"rule:admin_api",
"volume_extension:snapshot_admin_actions:reset_status":
"rule:admin_api",
"volume_extension:backup_admin_actions:reset_status":
"rule:admin_api",
"volume_extension:volume_admin_actions:force_delete":
"rule:admin_api",
"volume_extension:volume_admin_actions:force_detach":
"rule:admin_api",
"volume_extension:snapshot_admin_actions:force_delete":
"rule:admin_api",
"volume_extension:volume_admin_actions:migrate_volume":
"rule:admin_api",
"volume_extension:volume_admin_actions:migrate_volume_completion":
"rule:admin_api",
"volume_extension:volume_host_attribute": "rule:admin_api",
"volume_extension:volume_tenant_attribute":
"rule:admin_or_owner",
"volume_extension:volume_mig_status_attribute":
"rule:admin_api",
"volume_extension:hosts": "rule:admin_api",
"volume_extension:services": "rule:admin_api",
"volume_extension:volume_manage": "rule:admin_api",
"volume_extension:volume_unmanage": "rule:admin_api",
"volume:services": "rule:admin_api",
"volume:create_transfer": "",
"volume:accept_transfer": "",
"volume:delete_transfer": "",
"volume:get_all_transfers": "",
"volume_extension:replication:promote": "rule:admin_api",
"volume_extension:replication:reenable": "rule:admin_api",
"backup:create" : "",
"backup:delete": "",
"backup:get": "",
"backup:get_all": "",
"backup:restore": "",

OpenStack Security Guide

May 1, 2015

current

"backup:backup-import": "rule:admin_api",
"backup:backup-export": "rule:admin_api",

"snapshot_extension:snapshot_actions:update_snapshot_status":
"",
"consistencygroup:create" : "group:nobody",
"consistencygroup:delete": "group:nobody",
"consistencygroup:get": "group:nobody",
"consistencygroup:get_all": "group:nobody",
"consistencygroup:create_cgsnapshot" : "",
"consistencygroup:delete_cgsnapshot": "",
"consistencygroup:get_cgsnapshot": "",
"consistencygroup:get_all_cgsnapshots": "",
"scheduler_extension:scheduler_stats:get_pools" :
"rule:admin_api"
}

Note the default rule specifies that the user must be either an admin or
the owner of the volume. It essentially says only the owner of a volume or
the admin may create/delete/update volumes. Certain other operations
such as managing volume types are accessible only to admin users.

Tokens
Once a user is authenticated a token is generated for authorization and access to an OpenStack environment. A token can have a variable life span;
however since the release of OpenStack Icehouse, the default value for expiry has been reduced to one hour. The recommended expiry value should
be set to a lower value that allows enough time for internal services to
complete tasks. In the event that the token expires before tasks complete,
the cloud may become unresponsive or stop providing services. An example of expended time during use would be the time needed by the Compute service to transfer a disk image onto the hypervisor for local caching.
The following example shows a PKI token. Note that token id values are
typically 3500 bytes. In this example, the value has been truncated.

OpenStack Security Guide

May 1, 2015

current

{
"token": {
"expires": "2013-06-26T16:52:50Z",
"id": "MIIKXAY...",
"issued_at": "2013-06-25T16:52:50.622502",
"tenant": {
"description": null,
"enabled": true,
"id": "912426c8f4c04fb0a07d2547b0704185",
"name": "demo"
}
}
}

The token is often passed within the structure of a larger context of an

Identity service response. These responses also provide a catalog of the
various OpenStack services. Each service is listed with its name, access endpoints for internal, admin, and public access.
The Identity service supports token revocation. This manifests as an API to
revoke a token, to list revoked tokens and individual OpenStack services
that cache tokens to query for the revoked tokens and remove them from
their cache and append the same to their list of cached revoked tokens.

Future
Domains are high-level containers for projects, users and groups. As such,
they can be used to centrally manage all keystone-based identity components. With the introduction of account domains, server, storage and other resources can now be logically grouped into multiple projects (previously called tenants) which can themselves be grouped under a master account-like container. In addition, multiple users can be managed within an
account domain and assigned roles that vary for each project.
The Identity V3 API supports multiple domains. Users of different domains
may be represented in different authentication back ends and even have
different attributes that must be mapped to a single set of roles and privileges, that are used in the policy definitions to access the various service resources.
Where a rule may specify access to only admin users and users belonging
to the tenant, the mapping may be trivial. In other scenarios the cloud administrator may need to approve the mapping routines per tenant.

OpenStack Security Guide

May 1, 2015

current

Federated Identity
Federated Identity is a mechanism to establish trusts between Identity Providers and Service Providers (SP), in this case, between Identity
Providers and the services provided by an OpenStack Cloud.
Federated Identity provides a way to securely use existing credentials to access cloud resources such as servers, volumes, and databases, across multiple endpoints provided in multiple authorized clouds using a single set of
credentials, without having to provision additional identities or log in multiple times. The credential is maintained by the user's Identity Provider.
Some important definitions:
Service Provider (SP)

A system entity that provides services to

principals or other system entities, in this
case, OpenStack Identity is the Service
Provider.

Identity Provider (IdP)

A directory service, such as LDAP, RADIUS and Active Directory, which allows
users to login with a user name and password, is a typical source of authentication tokens (e.g. passwords) at an identity provider.

SAML assertion

Contains information about a user as

provided by an IdP. It is an indication
that a user has been authenticated.

Mapping

Adds a set of rules to map Federation

protocol attributes to Identity API objects. An Identity Provider has exactly
one mapping specified per protocol.

Protocol

Contains information that dictates which

Mapping rules to use for an incoming request made by an IdP. An IdP may support multiple protocols. There are three
major protocols for federated identity:
OpenID, SAML, and OAuth.

Unscoped token

Allows a user to authenticate with the

Identity service to exchange the un74

OpenStack Security Guide

May 1, 2015

current

scoped token for a scoped token, by providing a project ID or a domain ID.

Scoped token

Allows a user to use all OpenStack services apart from the Identity service.

Why use Federated Identity?

Provisioning new identities often incurs some security risk. It is difficult
to secure credential storage and to deploy it with proper policies. A common identity store is useful as it can be set up properly once and used
in multiple places. With Federated Identity, there is no longer a need to
provision user entries in Identity service, since the user entries already exist in the IdP's databases.
This does introduce new challenges around protecting that identity.
However, this is a worthwhile tradeoff given the greater control, and
fewer credential databases that come with a centralized common identity store.
It is a burden on the clients to deal with multiple tokens across multiple
cloud service providers. Federated Identity provides single sign on to the
user, who can use the credentials provided and maintained by the user's
IdP to access many different services on the Internet.
Users spend too much time logging in or going through 'Forget Password' workflows. Federated identity allows for single sign on, which is
easier and faster for users and requires fewer password resets. The IdPs
manage user identities and passwords so OpenStack does not have to.
Too much time is spent administering identities in various service
providers.
The best test of interoperability in the cloud is the ability to enable a user with one set of credentials in an IdP to access multiple cloud services.
Organizations, each using its own IdP can easily allow their users to collaborate and quickly share the same cloud services.
Removes a blocker to cloud brokering and multi-cloud workload management. There is no need to build additional authentication mechanisms ito authenticate users, since the IdPs take care of authenticating
their own users using whichever technologies they deem to be appropriate. In most organizations, multiple authentication technologies are already in use.
75

OpenStack Security Guide

May 1, 2015

current

Configuring Identity service for Federation

Federated users are not mirrored in the Identity service back end (for example, using the SQL driver). The external IdP is responsible for authenticating users, and communicates the result of the authentication to Identity
service using SAML assertions. Identity service maps the SAML assertions to
keystone user groups and assignments created in Identity service.

Enabling Federation
To enable Federation, perform the following steps:
1.

Run the Identity service under Apache, instead of using keystone-all.

Enable TLS support. Install mod_nss according to your distribution, then apply the following patch and restart HTTPD:
--- /etc/httpd/conf.d/nss.conf.orig 2012-03-29 12:59:06.
319470425 -0400
+++ /etc/httpd/conf.d/nss.conf
2012-03-29 12:19:38.
862721465 -0400
@@ -17,7 +17,7 @@
# Note: Configurations that use IPv6 but not IPv4-mapped
addresses need two
#
Listen directives: "Listen [::]:8443" and
"Listen 0.0.0.0:443"
#
-Listen 8443
+Listen 443
##
## SSL Global Context
@@ -81,7 +81,7 @@
## SSL Virtual Host Context
##
-<virtualhost _default_:8443="">
+<virtualhost _default_:443="">
#

General setup for the virtual host

#DocumentRoot "/etc/httpd/htdocs"
</virtualhost></virtualhost>

If you have a firewall in place, configure it to allow TLS traffic. For

example:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443
-j ACCEPT

OpenStack Security Guide

May 1, 2015

current

Note this needs to be added before your reject all rule which
might be:
-A INPUT -j REJECT --reject-with icmp-host-prohibited

Copy the httpd/wsgi-keystone.conf file to the appropriate location for your Apache server, for example, /etc/httpd/
conf.d/wsgi-keystone.conf file.

Create the directory /var/www/cgi-bin/keystone/. Then

link the files main and admin to the keystone.py file in this directory.
For a distribution appropriate place, it should probably be
copied to /usr/share/openstack/keystone/httpd/
keystone.py.

Note
This path is Ubuntu-specific. For other distributions,
replace with appropriate path.
e.

If you are running with SELinux enabled ensure that the file has
the appropriate SELinux context to access the linked file. For example, if you have the file in /var/www/cgi-bin location, you
can do this by running:
# restorecon /var/www/cgi-bin

Adding it in a different location requires you set up your SELinux

policy accordingly.
f.

Make sure you use either the SQL or the memcached driver for
tokens, otherwise the tokens will not be shared between the processes of the Apache HTTPD server.
For SQL, in /etc/keystone/keystone.conf , set:
[token]
driver = keystone.token.backends.sql.Token

For memcached, in /etc/keystone/keystone.conf, set:

[token]
driver = keystone.token.backends.memcache.Token

OpenStack Security Guide

May 1, 2015

current

In both cases, all servers that are storing tokens need a shared
back end. This means either that both point to the same
database server, or both point to a common memcached instance.
g.

Install Shibboleth:
# apt-get install libapache2-mod-shib2

Note
The apt-get command is Ubuntu specific. For other
distributions, replace with appropriate command.
h.

Configure the Identity service virtual host and adjust the config to
properly handle SAML2 workflow.
Add WSGIScriptAlias directive to your vhost configuration:
WSGIScriptAliasMatch ^(/v3/OS-FEDERATION/
identity_providers/.*?/protocols/.*?/auth)$ /var/www/
keystone/main/$1

Add two <Location> directives to the wsgi-keystone.conf

file:
<Location /Shibboleth.sso>
SetHandler shib
</Location>
<LocationMatch /v3/OS-FEDERATION/identity_providers/.*?/
protocols/saml2/auth>
ShibRequestSetting requireSession 1
AuthType shibboleth
ShibRequireAll On
ShibRequireSession On
ShibExportAssertion Off
Require valid-user
</LocationMatch>

Note
The option saml2 may be different in your deployment, but do not use a wildcard value. Otherwise every Federated protocol will be handled by Shibboleth.

OpenStack Security Guide

May 1, 2015

current

The ShibRequireSession rule is invalid in Apache

2.4 or newer and should be dropped in that specific
setup.
j.

Enable the Identity service virtual host:

# a2ensite wsgi-keystone.conf

Enable the ssl and shib2 modules:

# a2enmod ssl
# a2enmod shib2

Restart Apache:
# service apache2 restart

Note
The service apache2 restart command is
Ubuntu-specific. For other distributions, replace with
appropriate command.
2.

Configure Apache to use a Federation capable authentication

method.
a.

Once you have your Identity service virtual host ready, configure
Shibboleth and upload your metadata to the Identity Provider.
If new certificates are required, they can be easily created by executing:
$ shib-keygen -y NUMBER_OF_YEARS

The newly created file will be stored under /etc/shibboleth/sp-key.pem

Upload your Service Providers metadata file to your Identity

Provider.

Configure your Service Provider by editing /etc/shibboleth/shibboleth2.xml.

For more information, see Shibboleth Service Provider Configuration.
79

OpenStack Security Guide

May 1, 2015

current

Identity service enforces external authentication when environment variable REMOTE_USER is present so make sure Shibboleth does not set the REMOTE_USER environment variable. To do
so, scan through the /etc/shibboleth/shibboleth2.xml
configuration file and remove the REMOTE_USER directives.

Examine your attributes map in the /etc/shibboleth/attributes-map.xml file and adjust your requirements if needed. For more information see Shibboleth Attributes.

Restart the Shibboleth daemon:

# service shibd restart
# service apache2 restart

Enable OS-FEDERATION extension:

Add the Federation extension driver to the [federation] section in the keystone.conf file. For example:
[federation]
driver = keystone.contrib.federation.backends.sql.
Federation

Add the saml2 authentication method to the [auth] section in

keystone.conf file:
[auth]
methods = external,password,token,saml2
saml2 = keystone.auth.plugins.saml2.Saml2

Note
The external method should be dropped to avoid
any interference with some Apache and Shibboleth
SP setups, where a REMOTE_USER environment variable is always set, even as an empty value.
c.

Add the federation_extension middleware to the api_v3

pipeline in the keystone-paste.ini file. For example:
[pipeline:api_v3]
pipeline = access_log sizelimit url_normalize token_auth
admin_token_auth
xml_body json_body ec2_extension s3_extension
federation_extension
service_v3

OpenStack Security Guide

May 1, 2015

current

Create the Federation extension tables if using the provided SQL

back end. For example:
$ keystone-manage db_sync --extension federation

Ideally, to test that the Identity Provider and the Identity service are communicating, navigate to the protected URL and attempt to sign in. If you
get a response back from keystone, even if it is a wrong response, indicates the communication.

Configuring Federation
Now that the Identity Provider and Identity service are communicating,
you can start to configure the OS-FEDERATION extension.
1.

Create Identity groups and assign roles.

No new users will be added to the Identity back end, but the Identity service requires group-based role assignments to authorize federated users. The Federation mapping function will map the user into local
Identity service groups objects, and hence to local role assignments.
Thus, it is required to create the necessary Identity service groups
that correspond to the Identity Providers groups; additionally, these
groups should be assigned roles on one or more projects or domains.
For example, groups here refers to the Identity service groups that
should be created so that when mapping from the SAML attribute
Employees, you can map it to a Identity service group devs.
The Identity service administrator can create as many groups as there
are SAML attributes, whatever the mapping calls for.

Add Identity Providers, Mappings and Protocols.

To utilize Federation, create the following in the Identity service: Identity Provider, Mapping, Protocol.

Performing Federation authentication

Authenticate externally and generate an unscoped token in Identity

service.
To start Federated authentication a user must access the dedicated
URL with Identity Providers and Protocols identifiers stored with-

OpenStack Security Guide

May 1, 2015

current

in a protected URL. The URL has a format of: /v3/OS-FEDERATION/identity_providers/{identity_provider}/protocols/{protocol}/auth.

This instance follows a standard SAML2 authentication procedure,
that is, the user will be redirected to the Identity Providers authentication webpage and be prompted for credentials. After successfully authenticating the user will be redirected to the Service Providers
endpoint. If using a web browser, a token will be returned in XML format. As an alternative to using a web browser, you can use Enhanced
Client or Proxy (ECP), which is available in the keystoneclient in
the Identity service API.
In the returned unscoped token, a list of Identity service groups the
user belongs to will be included.
For example, the following URL would be considered protected by
mod_shib and Apache, as such a request made to the URL would be
redirected to the Identity Provider, to start the SAML authentication
procedure.
# curl -X GET \
-D - http://localhost:5000/v3/OS-FEDERATION/
identity_providers/{identity_provider}/protocols/{protocol}/
auth

Note
It is assumed that the keystone service is running on
port 5000.
2.

Determine accessible resources.

By using the previously returned token, the user can issue requests to
the list projects and domains that are accessible.
List projects a federated user can access: GET /OS-FEDERATION/projects
List domains a federated user can access: GET /OS-FEDERATION/domains
For example,
# curl -X GET \
-H "X-Auth-Token: <unscoped token>" http://localhost:5000/
v3/OS-FEDERATION/projects

OpenStack Security Guide

May 1, 2015

current

or
# curl -X GET \
-H "X-Auth-Token: <unscoped token>" http://localhost:5000/
v3/OS-FEDERATION/domains

Get a scoped token.

A federated user may request a scoped token, by using the unscoped
token. A project or domain may be specified by either ID or name. An
ID is sufficient to uniquely identify a project or domain. For example,
# curl -X POST \
-H "Content-Type: application/json" \
-d '{"auth":{"identity":{"methods":["saml2"],"saml2":
{"id":"<unscoped_token_id>"}},"scope":{"project":{"domain":
{"name": "Default"},"name":"service"}}}}' \
-D - http://localhost:5000/v3/auth/tokens

Setting Identity service as Identity Provider

Configuration options
Before attempting to federate multiple Identity service deployments, you
must setup certain configuration options in the keystone.conf file.
Within the keystone.conf assign values to the [saml] related fields,
for example:
[saml]
certfile=/etc/keystone/ssl/certs/ca.pem
keyfile=/etc/keystone/ssl/private/cakey.pem
idp_entity_id=https://keystone.example.com/v3/OS-FEDERATION/
saml2/idp
idp_sso_endpoint=https://keystone.example.com/v3/OS-FEDERATION/
saml2/sso
idp_metadata_path=/etc/keystone/saml2_idp_metadata.xml

It is recommended that the following Organization configuration options be setup.

idp_organization_name=example_company
idp_organization_display_name=Example Corp.
idp_organization_url=example.com

It is also recommended the following Contact options are set.

OpenStack Security Guide

May 1, 2015

current

idp_contact_company=example_company
idp_contact_name=John
idp_contact_surname=Smith
[email protected]
idp_contact_telephone=555-55-5555
idp_contact_type=technical

Generate metadata
In order to create a trust between the Identity Provider and the Service
Provider, metadata must be exchanged. To create metadata for your Identity service, run the keystone-manage command and pipe the output to a
file. For example:
$ keystone-manage saml_idp_metadata > /etc/keystone/
saml2_idp_metadata.xml

Note
The file location should match the value of the configuration
option idp_metadata_path that was assigned in the list of
[saml] updates.

Create a region for the Service Provider

Create a new region for the service provider, for example, create a
new region with an ID of BETA, and URL of https://beta.com/
Shibboleth.sso/SAML2/POST. This URL will be used when creating
a SAML assertion for BETA, and signed by the current keystone Identity
Provider.
$ curl -s -X PUT \
-H "X-Auth-Token: $OS_TOKEN" \
-H "Content-Type: application/json" \
-d '{"region": {"url": "http://beta.com/Shibboleth.sso/SAML2/
POST"}}' \
http://localhost:5000/v3/regions/BETA | python -mjson.tool

Testing it all out

Lastly, if a scoped token and a Service Provider region are presented to
keystone, the result will be a full SAML Assertion, signed by the IdP keystone, specifically intended for the Service Provider keystone.
$ curl -s -X POST \
-H "Content-Type: application/json" \

OpenStack Security Guide

May 1, 2015

current

-d '{"auth": {"scope": {"region": {"id": "BETA"}}, "identity":

{"token": {"id": "d793d935b9c343f783955cf39ee7dc3c"},
"methods": ["token"]}}}' \
http://localhost:5000/v3/auth/OS-FEDERATION/saml2

At this point the SAML Assertion can be sent to the Service Provider keystone, and a valid OpenStack token, issued by a Service Provider keystone,
will be returned.

Future
Currently, the CLI supports the Enhanced Client or Proxy (ECP), (the nonbrowser) support for keystoneclient from an API perspective. So, if
you are using the keystoneclient, you can create a client instance and
use the SAML authorization plugin. There is no support for dashboard
available presently. With the upcoming OpenStack releases, Federated
Identity should be supported with both CLI and the dashboard.

Checklist
Check-Identity-01: Is user and group ownership of
Identity configuration files set to keystone?
Configuration files contain critical parameters and information required
for smooth functioning of the component. If an unprivileged user, either
intentionally or accidently modifies or deletes any of the parameters or the
file itself then it would cause severe availability issue causing denial of service to the other end users. Thus user and group ownership of such critical
configuration files must be set to that component owner.
Run the following commands:
$ stat -L -c "%U %G" /etc/keystone/keystone.conf | egrep
"keystone keystone"
$ stat -L -c "%U %G" /etc/keystone/keystone-paste.ini | egrep
"keystone keystone"
$ stat -L -c "%U %G" /etc/keystone/policy.json | egrep "keystone
keystone"
$ stat -L -c "%U %G" /etc/keystone/logging.conf | egrep
"keystone keystone"
$ stat -L -c "%U %G" /etc/keystone/ssl/certs/signing_cert.pem |
egrep "keystone keystone"
$ stat -L -c "%U %G" /etc/keystone/ssl/private/signing_key.pem |
egrep "keystone keystone"
$ stat -L -c "%U %G" /etc/keystone/ssl/certs/ca.pem | egrep
"keystone keystone"

OpenStack Security Guide

May 1, 2015

current

Pass: If user and group ownership of all these config files is set to keystone. The above commands show output of keystone keystone.
Fail: If the above commands does not return any output as the user or
group ownership might have set to any user other than keystone.
Recommended in: the section called Internally implemented authentication methods [66]

Check-Identity-02: Are strict permissions set for

Identity configuration files?
Similar to previous check, it is recommended to set strict access permissions
for such configuration files.
Run the following commands:
$
$
$
$
$
$
$

stat
stat
stat
stat
stat
stat
stat

-L
-L
-L
-L
-L
-L
-L

-c
-c
-c
-c
-c
-c
-c

"%a"
"%a"
"%a"
"%a"
"%a"
"%a"
"%a"

/etc/keystone/keystone.conf
/etc/keystone/keystone-paste.ini
/etc/keystone/policy.json
/etc/keystone/logging.conf
/etc/keystone/ssl/certs/signing_cert.pem
/etc/keystone/ssl/private/signing_key.pem
/etc/keystone/ssl/certs/ca.pem

7. Dashboard
Basic web server configuration .............................................................
HTTPS ..................................................................................................
HTTP Strict Transport Security (HSTS) ...................................................
Front end caching ................................................................................
Domain names .....................................................................................
Static media .........................................................................................
Secret key ............................................................................................
Session back end .................................................................................
Allowed hosts ......................................................................................
Cross Site Request Forgery (CSRF) ........................................................
Cookies ................................................................................................
Cross Site Scripting (XSS) ......................................................................
Cross Origin Resource Sharing (CORS) ..................................................
Horizon image upload .........................................................................
Upgrading ...........................................................................................
Debug ..................................................................................................

89
90
90
90
90
91
92
92
92
93
93
93
93
94
94
94

Horizon is the OpenStack dashboard that provides users a self-service portal to provision their own resources within the limits set by administrators.
These include provisioning users, defining instance flavors, uploading VM
images, managing networks, setting up security groups, starting instances,
and accessing the instances through a console.
The dashboard is based on the Django web framework, therefore secure
deployment practices for Django apply directly to horizon. This guide provides a popular set of Django security recommendations. Further information can be found by reading the Django documentation.
The dashboard ships with reasonable default security settings, and has
good deployment and configuration documentation.

Basic web server configuration

The dashboard should be deployed as a Web Services Gateway Interface
(WSGI) application behind an HTTPS proxy such as Apache or nginx. If
Apache is not already in use, we recommend nginx since it is lightweight
and easier to configure correctly.
When using nginx, we recommend gunicorn as the WSGI host with an appropriate number of synchronous workers. When using Apache, we recommend mod_wsgi to host the dashboard.
89

OpenStack Security Guide

May 1, 2015

Session back end

Horizon's default session back end
(django.contrib.sessions.backends.signed_cookies) stores
user data in signed but unencrypted cookies stored in the browser. This
approach allows the most simple session back-end scaling since each dashboard instance is stateless, but it comes at the cost of storing sensitive access tokens in the client browser and transmitting them with every request.
This back end ensures that session data has not been tampered with, but
the data itself is not encrypted other than the encryption provided by
HTTPS.
If your architecture allows it, we recommend using
django.contrib.sessions.backends.cache as your session back
end with memcache as the cache. Memcache must not be exposed publicly, and should communicate over a secured private channel. If you
choose to use the signed cookies back end, refer to the Django documentation understand the security trade-offs.
For further details, see the Django documentation.

Allowed hosts
Configure the ALLOWED_HOSTS setting with the domain or domains
where the dashboard is available. Failure to configure this setting (especially if not following the recommendation above regarding second level domains) opens the dashboard to a number of serious attacks. Wild card domains should be avoided.
For further details, see the Django documentation.

OpenStack Security Guide

May 1, 2015

current

Cross Site Request Forgery (CSRF)

Django has dedicated middleware for cross-site request forgery (CSRF). For
further details, see the Django documentation.
Dashboard is designed to discourage developers from introducing crosssite scripting vulnerabilities with custom dashboards. However, it is important to audit custom dashboards, especially ones that are JavaScript-heavy
for inappropriate use of the @csrf_exempt decorator. Dashboards which
do not follow these recommended security settings should be carefully
evaluated before restrictions are relaxed.

Cookies
Session Cookies should be set to HTTPONLY:
SESSION_COOKIE_HTTPONLY = True

Never configure CSRF or session cookies to have a wild card domain with a
leading dot. Horizon's session and CSRF cookie should be secured when deployed with HTTPS:
CSRF_COOKIE_SECURE = True
SESSION_COOKIE_SECURE = True

Cross Site Scripting (XSS)

Unlike many similar systems, the OpenStack dashboard allows the entire
Unicode character set in most fields. This means developers have less latitude to make escaping mistakes that open attack vectors for cross-site
scripting (XSS).
Dashboard provides tools for developers to avoid creating XSS vulnerabilities, but they only work if developers use them correctly. Audit any custom
dashboards, paying particular attention to use of the mark_safe function, use of is_safe with custom template tags, the safe template tag,
anywhere auto escape is turned off, and any JavaScript which might evaluate improperly escaped data.

Cross Origin Resource Sharing (CORS)

Configure your web server to send a restrictive CORS header with each response, allowing only the dashboard domain and protocol:
93

OpenStack Security Guide

May 1, 2015

current

Access-Control-Allow-Origin: https://example.com/

Never allow the wild card origin.

Horizon image upload

We recommend that implementers disable
HORIZON_IMAGES_ALLOW_UPLOAD unless they have implemented a
plan to prevent resource exhaustion and denial of service.

Upgrading
Django security releases are generally well tested and aggressively backwards compatible. In almost all cases, new major releases of Django are
also fully backwards compatible with previous releases. Dashboard implementers are strongly encouraged to run the latest stable release of Django
with up-to-date security releases.

Debug
Make sure DEBUG is set to False in production. In Django, DEBUG displays
stack traces and sensitive web server state information on any exception.

OpenStack Security Guide

May 1, 2015

current

8. Compute
How to select virtual consoles .............................................................. 95
The Compute service (nova) is one of the more complex OpenStack services. It runs in many locations throughout the cloud and interacts with a
variety of internal services. For this reason, most of our recommendations
regarding best practices for Compute service configuration are distributed
throughout this book. We provide specific details in the sections on Management, API Endpoints, Messaging, and Database.

How to select virtual consoles

One decision a cloud architect will need to make regarding Compute service configuration is whether to use VNC or SPICE. Below we provide some
details on the differences between these options.

Virtual Network Computer (VNC)

OpenStack can be configured to provide remote desktop console access
to instances for tenants and/or administrators using the Virtual Network
Computer (VNC) protocol.

Capabilities
The OpenStack dashboard (horizon) can provide a VNC console for instances directly on the web page using the HTML5 noVNC client. This
requires the nova-novncproxy service to bridge from the public network to the management network.
The nova command-line utility can return a URL for the VNC console for access by the nova Java VNC client. This requires the nova-xvpvncproxy service to bridge from the public network to the
management network.

Security considerations
The nova-novncproxy and nova-xvpvncproxy services by default
open public-facing ports that are token authenticated.
By default, the remote desktop traffic is not encrypted. TLS can be enabled to encrypt the VNC traffic. Please refer to Introduction to TLS and
SSL for appropriate recommendations.
95

OpenStack Security Guide

May 1, 2015

current

Bibliography
blog.malchuk.ru, OpenStack VNC Security. 2013. Secure Connections to
VNC ports
OpenStack Mailing List, [OpenStack] nova-novnc SSL configuration - Havana. 2014. OpenStack nova-novnc SSL Configuration
Redhat.com/solutions, Using SSL Encryption with OpenStack nova-novacproxy. 2014. OpenStack nova-novncproxy SSL encryption

Simple Protocol for Independent Computing Environments (SPICE)

As an alternative to VNC, OpenStack provides remote desktop access to
guest virtual machines using the Simple Protocol for Independent Computing Environments (SPICE) protocol.

Capabilities
SPICE is supported by the OpenStack dashboard (horizon) directly on
the instance web page. This requires the nova-spicehtml5proxy service.
The nova command-line utility can return a URL for SPICE console for access by a SPICE-html client.

Limitations
Although SPICE has many advantages over VNC, the spice-html5 browser integration currently doesn't really allow admins to take advantage
of any of the benefits. To take advantage of SPICE features like multi-monitor, USB pass through, etc. admins are recommended to use a
standalone SPICE client within the Management Network.

Security considerations
The nova-spicehtml5proxy service by default opens public-facing
ports that are token authenticated.
The functionality and integration are still evolving. We will access the
features in the next release and make recommendations.
96

OpenStack Security Guide

May 1, 2015

current

As is the case for VNC, at this time we recommend using SPICE from the
management network in addition to limiting use to few individuals.

Bibliography
OpenStack Configuration Reference - Havana. SPICE Console. SPICE Console
bugzilla.redhat.com, Bug 913607 - RFE: Support Tunnelling SPICE over
websockets. 2013. Red Hat bug 913607

OpenStack Security Guide

May 1, 2015

current

9. Object Storage
First thing to secure: the network ......................................................
Securing services: general ...................................................................
Securing storage services ....................................................................
Securing proxy services .......................................................................
Object Storage authentication ............................................................
Other notable items ...........................................................................

100
102
103
104
105
106

OpenStack Object Storage (swift) is a service that provides software that

current

Figure9.2.Object Storage network architecture with a

management node (OSAM)

Securing services: general

Run services as non-root user
It is recommended that you configure each Object Storage service to run
under a non-root (UID 0) service account. One recommendation is the user name "swift" with the primary group "swift." Object Storage services include, for example, 'proxy-server', 'container-server', 'account-server'. Detailed steps for setup and configuration can be found in the Add Object
Storage chapter of the Installation Guide in the OpenStack Documentation
index. (The link defaults to the Ubuntu version.)

File permissions
The /etc/swift directory contains information about the ring topology and environment configuration. The following permissions are recommended:
# chown -R root:swift /etc/swift/*
# find /etc/swift/ -type f -exec chmod 640 {} \;

102

OpenStack Security Guide

May 1, 2015

current

# find /etc/swift/ -type d -exec chmod 750 {} \;

This restricts only root to be able to modify configuration files while allowing the services to read them through their group membership in the
'swift' group.

Securing storage services

The following are the default listening ports for the various storage services:
Service name

Port

Type

Account service

6002

TCP

Container service

6001

TCP

Object service

6000

TCP

873

TCP

Rsync
a

If ssync is used instead of rsync, the Object service port is used for maintaining durability.

Authentication does not take place at the storage nodes. If someone was
able to connect to a storage node on one of these ports they could access
or modify data without authentication. In order to secure against this issue you should follow the recommendations given previously about using
a private storage network.

Object Storage "account" terminology

An Object Storage "account" is not a user account or credential. The following explains the relations:
OpenStack Object Storage account

Collection of containers; not user accounts or

authentication. Which users are associated with
the account and how they may access it depends on the authentication system used. See
the section called Object Storage authentication [105].

OpenStack Object Storage containers

Collection of objects. Metadata on the container is available for ACLs. The meaning of ACLs is
dependent on the authentication system used.

OpenStack Object Storage objects

The actual data objects. ACLs at the object level

are also possible with metadata and are dependent on the authentication system used.

Tip
Another way of thinking about the above would be: A single
shelf (account) holds zero or more buckets (containers) which
103

OpenStack Security Guide

May 1, 2015

current

each hold zero or more objects. A garage (Object Storage cluster) may have multiple shelves (accounts) with each shelf belonging to zero or more users.
At each level you may have ACLs that dictate who has what type of access. ACLs are interpreted based on what authentication system is in use.
The two most common types of authentication providers used are Identity service (keystone) and TempAuth. Custom authentication providers
are also possible. Please see the section called Object Storage authentication [105] for more information.

Securing proxy services

A proxy node should have at least two interfaces (physical or virtual): one
public and one private. Firewalls or service binding might protect the public interface. The public facing service is an HTTP web server that processes
end-point client requests, authenticates them, and performs the appropriate action. The private interface does not require any listening services but
is instead used to establish outgoing connections to storage nodes on the
private storage network.

HTTP listening port

You should configure your web service as a non-root (no UID 0) user such as "swift" mentioned before. The use of a port greater than
1024 is required to make this easy and avoid running any part of the
web container as root. Doing so is not a burden as end-point clients
are not typically going to type in the URL manually into a web browser to browse around in the object storage. Additionally, for clients using the HTTP REST API and performing authentication they will normally automatically grab the full REST API URL they are to use as provided by the authentication response. OpenStack's REST API allows
for a client to authenticate to one URL and then be told to use a completely different URL for the actual service. Example: Client authenticates to https://identity.cloud.example.org:55443/
v1/auth and gets a response with their authentication key and Storage URL (the URL of the proxy nodes or load balancer) of https://
swift.cloud.example.org:44443/v1/AUTH_8980.
The method for configuring your web server to start and run as a non-root
user varies by web server and OS.

104

OpenStack Security Guide

May 1, 2015

current

Load balancer
If the option of using Apache is not feasible or for performance you wish
to offload your TLS work you may employ a dedicated network device
load balancer. This is also the common way to provide redundancy and
load balancing when using multiple proxy nodes.
If you choose to offload your TLS, ensure that the network link between
the load balancer and your proxy nodes are on a private (V)LAN segment
such that other nodes on the network (possibly compromised) cannot
wiretap (sniff) the unencrypted traffic. If such a breach were to occur the
attacker could gain access to end-point client or cloud administrator credentials and access the cloud data.
The authentication service you use, such as Identity service (keystone) or
TempAuth, will determine how you configure a different URL in the responses to end-point clients so they use your load balancer instead of an
individual proxy node.

Object Storage authentication

Object Storage uses a WSGI model to provide for a middleware capability
that not only provides general extensibility but is also used for authentication of end-point clients. The authentication provider defines what roles
and user types exist. Some use traditional user name and password credentials while others may leverage API key tokens or even client-side x.509 certificates. Custom providers can be integrated in using custom middleware.
Object Storage comes with two authentication middleware modules by default, either of which can be used as sample code for developing a custom
authentication middleware.

TempAuth
TempAuth is the default authentication for Object Storage. In contrast to
Identity it stores the user accounts, credentials, and metadata in object
storage itself. More information can be found in the section The Auth System of the Object Storage (swift) documentation.

Keystone
Keystone is the commonly used Identity provider in OpenStack. It may also be used for authentication in Object Storage. Coverage of securing keystone is already provided in Chapter6, Identity [65].
105

OpenStack Security Guide

May 1, 2015

current

Other notable items

In /etc/swift/swift.conf on every node there is a
swift_hash_path_prefix setting and a swift_hash_path_suffix
setting. These are provided to reduce the chance of hash collisions for objects being stored and avert one user overwriting the data of another user.
This value should be initially set with a cryptographically secure random
number generator and consistent across all nodes. Ensure that it is protected with proper ACLs and that you have a backup copy to avoid data loss.

106

OpenStack Security Guide

May 1, 2015

current

10. Case studies: Identity

management
Alice's private cloud ........................................................................... 107
Bob's public cloud .............................................................................. 107
Earlier in the section called Introduction to case studies [21] we introduced the Alice and Bob case studies where Alice is deploying a private
government cloud and Bob is deploying a public cloud each with different security requirements. Here we discuss how Alice and Bob would address configuration of OpenStack core services. These include the Identity service, dashboard, and Compute services. Alice will be concerned with
integration into the existing government directory services, while Bob will
need to provide access to the public.

Alice's private cloud

Alice's enterprise has a well-established directory service with two-factor
authentication for all users. She configures the Identity service to support
an external authentication service supporting authentication with government-issued access cards. She also uses an external LDAP server to provide
role information for the roles that are integrated with the access control
policy. Due to FedRAMP compliance requirements, Alice implements twofactor authentication on the management network for all administrator
access.
Alice also deploys the dashboard to manage many aspects of the cloud.
She deploys the dashboard with HSTS to ensure that only HTTPS is used.
The dashboard resides within an internal subdomain of the private network domain name system.
Alice decides to use SPICE instead of VNC for the virtual console. She wants
to take advantage of the emerging capabilities in SPICE.

Bob's public cloud

Because Bob must support authentication for the general public, he decides to use authentication based on a user name and password. He has
concerns about brute force attacks attempting to crack user passwords, so
he also uses an external authentication extension that throttles the num107

OpenStack Security Guide

May 1, 2015

current

ber of failed login attempts. Bob's management network is separate from

the other networks within his cloud, but can be reached from his corporate network through ssh. As recommended earlier, Bob requires administrators to use two-factor authentication on the Management network to
reduce the risk of compromised administrator passwords.
Bob also deploys the dashboard to manage many aspects of the cloud.
He deploys the dashboard with HSTS to ensure that only HTTPS is used.
He has ensured that the dashboard is deployed on a second-level domain due to the limitations of the same-origin policy. He also disables
HORIZON_IMAGES_ALLOW_UPLOAD to prevent resource exhaustion.
Bob decides to use VNC for his virtual console for its maturity and security
features.

108

OpenStack Security Guide

May 1, 2015

current

11. Networking
Networking architecture ....................................................................
Networking services ...........................................................................
Securing OpenStack Networking services ............................................
Networking services security best practices .........................................
Case studies .......................................................................................

109
113
117
119
121

OpenStack Networking enables the end-user or tenant to define, utilize,

and consume networking resources. OpenStack Networking provides a
tenant-facing API for defining network connectivity and IP addressing for
instances in the cloud in addition to orchestrating the network configuration. With the transition to an API-centric networking service, cloud architects and administrators should take into consideration best practices to
secure physical and virtual network infrastructure and services.
OpenStack Networking was designed with a plug-in architecture that provides extensibility of the API through open source community or third-party services. As you evaluate your architectural design requirements, it is important to determine what features are available in OpenStack Networking core services, any additional services that are provided by third-party
products, and what supplemental services are required to be implemented
in the physical infrastructure.
This section is a high-level overview of what processes and best practices
should be considered when implementing OpenStack Networking. We will
talk about the current state of services that are available, what future services will be implemented, and the current limitations in this project.

Networking architecture
OpenStack Networking is a standalone service that often deploys several processes across a number of nodes. These processes interact with each
other and other OpenStack services. The main process of the OpenStack
Networking service is neutron-server, a Python daemon that exposes the OpenStack Networking API and passes tenant requests to a suite of
plug-ins for additional processing.
The OpenStack Networking components are:
neutron server (neutron-server and neutron-*-plugin)

This service runs on the network node

to service the Networking API and its

109

OpenStack Security Guide

May 1, 2015

current

extensions. It also enforces the network model and IP addressing of each

port. The neutron-server and plugin
agents require access to a database for
persistent storage and access to a message queue for inter-communication.
plugin agent (neutron-*agent)

Runs on each compute node to manage local virtual switch (vswitch) configuration. The plug-in that you use determine which agents run. This service requires message queue access. This service requires message queue access and
depends on the plugin used.

DHCP agent (neutron-dhcpagent)

Provides DHCP services to tenant networks. This agent is the same across
all plug-ins and is responsible for maintaining DHCP configuration. The neutron-dhcp-agent requires message
queue access.

L3 agent (neutron-l3agent)

Provides L3/NAT forwarding for external network access of VMs on tenant

networks. Requires message queue access. Optional depending on plug-in.

network provider services (SDN

server/services)

Provides additional networking services

to tenant networks. These SDN services
may interact with neutron-server,
neutron-plugin, and plugin-agents
through communication channels such
as REST APIs.

The following figure shows an architectural and networking flow diagram

of the OpenStack Networking components:

110

OpenStack Security Guide

May 1, 2015

current

OpenStack Networking service placement on

physical servers
This guide focuses on a standard architecture that includes a cloud controller host, a network host, and a set of compute hypervisors for running
VMs.

111

OpenStack Security Guide

May 1, 2015

current

Network connectivity of physical servers

A standard OpenStack Networking setup has up to four distinct physical

data center networks:
Management network

Used for internal communication between

OpenStack Components. The IP addresses on
this network should be reachable only within
the data center and is considered the Management Security Domain.

Guest network

Used for VM data communication within the

cloud deployment. The IP addressing requirements of this network depend on the OpenStack Networking plug-in in use and the network configuration choices of the virtual networks made by the tenant. This network is considered the Guest Security Domain.

External network

Used to provide VMs with Internet access in

some deployment scenarios. The IP addresses
on this network should be reachable by anyone
on the Internet. This network is considered to
be in the Public Security Domain.
112

OpenStack Security Guide

API network

May 1, 2015

current

Exposes all OpenStack APIs, including the OpenStack Networking API, to tenants. The IP addresses on this network should be reachable by
anyone on the Internet. This may be the same
network as the external network, as it is possible to create a subnet for the external network that uses IP allocation ranges to use only
less than the full range of IP addresses in an IP
block. This network is considered the Public Security Domain.

For additional information see the Networking chapter in the OpenStack

Cloud Administrator Guide.

Networking services
In the initial architectural phases of designing your OpenStack Network infrastructure it is important to ensure appropriate expertise is available to
assist with the design of the physical networking infrastructure, to identify
proper security controls and auditing mechanisms.
OpenStack Networking adds a layer of virtualized network services which
gives tenants the capability to architect their own virtual networks. Currently, these virtualized services are not as mature as their traditional networking counterparts. Consider the current state of these virtualized services before adopting them as it dictates what controls you may have to
implement at the virtualized and traditional network boundaries.

L2 isolation using VLANs and tunneling

OpenStack Networking can employ two different mechanisms for traffic
segregation on a per tenant/network combination: VLANs (IEEE 802.1Q
tagging) or L2 tunnels using GRE encapsulation. The scope and scale of
your OpenStack deployment determines which method you should utilize
for traffic segregation or isolation.

VLANs
VLANs are realized as packets on a specific physical network containing
IEEE 802.1Q headers with a specific VLAN ID (VID) field value. VLAN networks sharing the same physical network are isolated from each other at
L2, and can even have overlapping IP address spaces. Each distinct physical
network supporting VLAN networks is treated as a separate VLAN trunk,
with a distinct space of VID values. Valid VID values are 1 through 4094.
113

OpenStack Security Guide

May 1, 2015

current

VLAN configuration complexity depends on your OpenStack design requirements. In order to allow OpenStack Networking to efficiently use
VLANs, you must allocate a VLAN range (one for each tenant) and turn
each compute node physical switch port into a VLAN trunk port.

Note
NOTE: If you intend for your network to support more than
4094 tenants VLAN is probably not the correct option for you
as multiple 'hacks' are required to extend the VLAN tags to
more than 4094 tenants.

L2 tunneling
Network tunneling encapsulates each tenant/network combination with
a unique "tunnel-id" that is used to identify the network traffic belonging
to that combination. The tenant's L2 network connectivity is independent
of physical locality or underlying network design. By encapsulating traffic
inside IP packets, that traffic can cross Layer-3 boundaries, removing the
need for preconfigured VLANs and VLAN trunking. Tunneling adds a layer
of obfuscation to network data traffic, reducing the visibility of individual
tenant traffic from a monitoring point of view.
OpenStack Networking currently supports both GRE and VXLAN encapsulation.
The choice of technology to provide L2 isolation is dependent upon the
scope and size of tenant networks that will be created in your deployment. If your environment has limited VLAN ID availability or will have a
large number of L2 networks, it is our recommendation that you utilize
tunneling.

Network services
The choice of tenant network isolation affects how the network security
and control boundary is implemented for tenant services. The following
additional network services are either available or currently under development to enhance the security posture of the OpenStack network architecture.

Access control lists

OpenStack Compute supports tenant network traffic access controls directly when deployed with the legacy nova-network service, or may defer access control to the OpenStack Networking service.
114

OpenStack Security Guide

May 1, 2015

current

Note, legacy nova-network security groups are applied to all virtual interface ports on an instance using iptables.
Security groups allow administrators and tenants the ability to specify
the type of traffic, and direction (ingress/egress) that is allowed to pass
through a virtual interface port. Security groups rules are stateful L2-L4
traffic filters.
When using the Networking service, we recommend that you enable security groups in this service and disable it in the Compute service.

L3 routing and NAT

OpenStack Networking routers can connect multiple L2 networks, and can
also provide a gateway that connects one or more private L2 networks to
a shared external network, such as a public network for access to the Internet.
The L3 router provides basic Network Address Translation (NAT) capabilities on gateway ports that uplink the router to external networks. This
router SNATs (Static NAT) all traffic by default, and supports floating IPs,
which creates a static one-to-one mapping from a public IP on the external network to a private IP on one of the other subnets attached to the
router.
It is our recommendation to leverage per tenant L3 routing and Floating
IPs for more granular connectivity of tenant VMs.

Quality of Service (QoS)

The ability to set QoS on the virtual interface ports of tenant instances is
a current deficiency for OpenStack Networking. The application of QoS
for traffic shaping and rate-limiting at the physical network edge device is
insufficient due to the dynamic nature of workloads in an OpenStack deployment and can not be leveraged in the traditional way. QoS-as-a-Service
(QoSaaS) is currently in development for the OpenStack Networking Icehouse release as an experimental feature. QoSaaS is planning to provide
the following services:
Traffic shaping through DSCP markings
Rate-limiting on a per port/network/tenant basis.
Port mirroring (through open source or third-party plug-ins)
Flow analysis (through open source or third-party plug-ins)
115

OpenStack Security Guide

May 1, 2015

current

Tenant traffic port mirroring or Network Flow monitoring is currently not

an exposed feature in OpenStack Networking. There are third-party plugin extensions that do provide Port Mirroring on a per port/network/tenant basis. If Open vSwitch is used on the networking hypervisor, it is possible to enable sFlow and port mirroring, however it will require some operational effort to implement.

Load balancing
Another feature in OpenStack Networking is Load-Balancer-as-a-service
(LBaaS). The LBaaS reference implementation is based on HA-Proxy. There
are third-party plug-ins in development for extensions in OpenStack Networking to provide extensive L4-L7 functionality for virtual interface ports.

Firewalls
FW-as-a-Service (FWaaS) is considered an experimental feature for the Kilo
release of OpenStack Networking. FWaaS addresses the need to manage
and leverage the rich set of security features provided by typical firewall
products which are typically far more comprehensive than what is currently provided by security groups. Both Freescale and Intel developed thirdparty plug-ins as extensions in OpenStack Networking to support this component in the Kilo release. Documentation for administration of FWaaS is
located at
It is critical during the design of an OpenStack Networking infrastructure
to understand the current features and limitations of network services that
are available. Understanding where the boundaries of your virtual and
physical networks will help you add the required security controls in your
environment.

Network services extensions

A list of known plug-ins provided by the open source community or by SDN
companies that work with OpenStack Networking is available at OpenStack Neutron Plugins and Drivers wiki page.

Networking services limitations

OpenStack Networking has the following known limitations:
Overlapping IP addresses

If nodes that run either neutron-l3agent or neutron-dhcp-agent

116

OpenStack Security Guide

May 1, 2015

current

use overlapping IP addresses, those

nodes must use Linux network namespaces. By default, the DHCP and L3
agents use Linux network namespaces.
However, if the host does not support
these namespaces, run the DHCP and
L3 agents on different hosts.
If network namespace support is not
present, a further limitation of the L3
agent is that only a single logical router
is supported.
Multi-host DHCP-agent

OpenStack Networking supports multiple L3 and DHCP agents with load balancing. However, tight coupling of the
location of the virtual machine is not
supported.

No IPv6 support for L3 agents

The neutron-l3-agent, used by many

plug-ins to implement L3 forwarding,
supports only IPv4 forwarding.

Securing OpenStack Networking services

To secure OpenStack Networking, you must understand how the workflow
process for tenant instance creation needs to be mapped to security domains.
There are four main services that interact with OpenStack Networking. In a
typical OpenStack deployment these services map to the following security
domains:
OpenStack dashboard: Public and management
OpenStack Identity: Management
OpenStack compute node: Management and guest
OpenStack network node: Management, guest, and possibly public depending upon neutron-plugin in use.
SDN services node: Management, guest and possibly public depending
upon product used.
117

OpenStack Security Guide

May 1, 2015

current

To isolate sensitive data communication between the OpenStack Networking services and other OpenStack core services, configure these communication channels to only allow communication over an isolated management network.

OpenStack Networking service configuration

Restrict bind address of the API server: neutron-server
To restrict the interface or IP address on which the OpenStack Networking
API service binds a network socket for incoming client connections, specify
the bind_host and bind_port in the neutron.conf file as shown:
# Address to bind the API server
bind_host = IP ADDRESS OF SERVER
# Port the bind the API server to
bind_port = 9696

118

OpenStack Security Guide

May 1, 2015

current

Restrict DB and RPC communication of the OpenStack Networking services

Various components of the OpenStack Networking services use either the
messaging queue or database connections to communicate with other
components in OpenStack Networking.
It is recommended that you follow the guidelines provided in the section
called Database authentication and access control [144] for all components which require direct DB connections.
It is recommended that you follow the guidelines provided in the section
called Queue authentication and access control [125] for all components which require RPC communication.

Networking services security best practices

This section discusses OpenStack Networking configuration best practices
as they apply to tenant network security within your OpenStack deployment.

Tenant network services workflow

OpenStack Networking provides users self services of network resources
and configurations. It is important that cloud architects and operators
evaluate their design use cases in providing users the ability to create, update, and destroy available network resources.

Networking resource policy engine

A policy engine and its configuration file, policy.json, within OpenStack Networking provides a method to provide finer grained authorization of users on tenant networking methods and objects. It is important
that cloud architects and operators evaluate their design and use cases in
providing users and tenants the ability to create, update, and destroy available network resources as it has a tangible effect on tenant network availability, network security, and overall OpenStack security. For a more detailed explanation of OpenStack Networking policy definition, please refer
to the Authentication and authorization section in the OpenStack Cloud
Administrator Guide.
119

OpenStack Security Guide

May 1, 2015

current

Note
It is important to review the default networking resource policy and modify the policy appropriately for your security posture.
If your deployment of OpenStack provides multiple external access points
into different security domains it is important that you limit the tenant's
ability to attach multiple vNICs to multiple external access pointsthis
would bridge these security domains and could lead to unforeseen security
compromise. It is possible mitigate this risk by utilizing the host aggregates
functionality provided by OpenStack Compute or through splitting the tenant VMs into multiple tenant projects with different virtual network configurations.

Security groups
The OpenStack Networking service provides security group functionality using a mechanism that is more flexible and powerful than the security group capabilities built into OpenStack Compute. Thus, nova.conf
should always disable built-in security groups and proxy all security group
calls to the OpenStack Networking API when using OpenStack Networking. Failure to do so results in conflicting security policies being simultaneously applied by both services. To proxy security groups to OpenStack Networking, use the following configuration values:
firewall_driver must be set to
nova.virt.firewall.NoopFirewallDriver so that nova-compute does not perform iptables-based filtering itself.
security_group_api must be set to neutron so that all security
group requests are proxied to the OpenStack Networking service.
A security group is a container for security group rules. Security groups and
their rules allow administrators and tenants the ability to specify the type
of traffic and direction (ingress/egress) that is allowed to pass through a
virtual interface port. When a virtual interface port is created in OpenStack
Networking it is associated with a security group. If a security group is not
specified, the port will be associated with a 'default' security group. By default this group will drop all ingress traffic and allow all egress. Rules can
be added to this group in order to change the behaviour.
When using the security group API through OpenStack Compute, security
groups are applied to all virtual interface ports on an instance. The reason
120

OpenStack Security Guide

RabbitMQ server SSL configuration

The following lines should be added to the system-wide RabbitMQ configuration file, typically /etc/rabbitmq/rabbitmq.config:
[
{rabbit, [
{tcp_listeners, [] },
{ssl_listeners, [{"<IP address or hostname of management
network interface>", 5671}] },
{ssl_options, [{cacertfile,"/etc/ssl/cacert.pem"},
{certfile,"/etc/ssl/rabbit-server-cert.
pem"},
{keyfile,"/etc/ssl/rabbit-server-key.pem"},
{verify,verify_peer},
{fail_if_no_peer_cert,true}]}
]}
].

Note, the tcp_listeners option is set to [] to prevent it from listening

an on non-SSL port. The ssl_listeners option should be restricted to
only listen on the management network for the services.
For more information on RabbitMQ SSL configuration see:
124

OpenStack Security Guide

May 1, 2015

current

RabbitMQ Configuration
RabbitMQ SSL

Qpid server SSL configuration

The Apache Foundation has a messaging security guide for Qpid. See:
Apache Qpid SSL

Queue authentication and access control

RabbitMQ and Qpid offer authentication and access control mechanisms
for controlling access to queues. ZeroMQ offers no such mechanisms.
Simple Authentication and Security Layer (SASL) is a framework for authentication and data security in Internet protocols. Both RabbitMQ and
Qpid offer SASL and other pluggable authentication mechanisms beyond
simple user names and passwords that allow for increased authentication
security. While RabbitMQ supports SASL, support in OpenStack does not
currently allow for requesting a specific SASL authentication mechanism.
RabbitMQ support in OpenStack allows for either user name and password
authentication over an unencrypted connection or user name and password in conjunction with X.509 client certificates to establish the secure
TLS connection.
We recommend configuring X.509 client certificates on all the OpenStack
service nodes for client connections to the messaging queue and where
possible (currently only Qpid) perform authentication with X.509 client certificates. When using user names and passwords, accounts should be created per-service and node for finer grained auditability of access to the
queue.
Before deployment, consider the TLS libraries that the queuing servers use.
Qpid uses Mozilla's NSS library, whereas RabbitMQ uses Erlang's TLS module which uses OpenSSL.

Authentication configuration example: RabbitMQ

On the RabbitMQ server, delete the default guest user:
# rabbitmqctl delete_user quest

On the RabbitMQ server, for each OpenStack service or node that communicates with the message queue set up user accounts and privileges:
125

OpenStack Security Guide

May 1, 2015

current

# rabbitmqctl add_user compute01 RABBIT_PASS

# rabbitmqctl set_permissions compute01 ".*" ".*" ".*"

Replace RABBIT_PASS with a suitable password.

For additional configuration information see:
RabbitMQ Access Control
RabbitMQ Authentication
RabbitMQ Plugins
RabbitMQ SASL External Auth

OpenStack service configuration: RabbitMQ

[DEFAULT]
rpc_backend=nova.openstack.common.rpc.impl_kombu
rabbit_use_ssl=True
rabbit_host=
rabbit_port=5671
rabbit_user=compute01
rabbit_password=RABBIT_PASS
kombu_ssl_keyfile=/etc/ssl/node-key.pem
kombu_ssl_certfile=/etc/ssl/node-cert.pem
kombu_ssl_ca_certs=/etc/ssl/cacert.pem

Authentication configuration example: Qpid

For configuration information see:
Apache Qpid Authentication
Apache Qpid Authorization

OpenStack service configuration: Qpid

[DEFAULT]
rpc_backend=nova.openstack.common.rpc.impl_qpid
qpid_protocol=ssl
qpid_hostname=<IP or hostname of management network interface of
messaging server>
qpid_port=5671
qpid_username=compute01
qpid_password=QPID_PASS

126

OpenStack Security Guide

May 1, 2015

current

Optionally, if using SASL with Qpid specify the SASL mechanisms in use by
adding:
qpid_sasl_mechanisms=<space separated list of SASL mechanisms to
use for auth>

Message queue process isolation and policy

Each project provides a number of services which send and consume messages. Each binary which sends a message is expected to consume messages, if only replies, from the queue.
Message queue service processes should be isolated from each other and
other processes on a machine.

Namespaces
Network namespaces are highly recommended for all services running on
OpenStack Compute Hypervisors. This will help prevent against the bridging of network traffic between VM guests and the management network.
When using ZeroMQ messaging, each host must run at least one ZeroMQ
message receiver to receive messages from the network and forward messages to local processes through IPC. It is possible and advisable to run an
independent message receiver per project within an IPC namespace, along
with other services within the same project.

Network policy
Queue servers should only accept connections from the management network. This applies to all implementations. This should be implemented
through configuration of services and optionally enforced through global
network policy.
When using ZeroMQ messaging, each project should run a separate ZeroMQ receiver process on a port dedicated to services belonging to that
project. This is equivalent to the AMQP concept of control exchanges.

Mandatory access controls

Use both mandatory access controls (MACs) and discretionary access controls (DACs) to restrict the configuration for processes to only those processes. This restriction prevents these processes from being isolated from
other processes that run on the same machine(s).
127

OpenStack Security Guide

May 1, 2015

current

Case studies
Earlier in the section called Introduction to case studies [21] we introduced the Alice and Bob case studies where Alice is deploying a private
government cloud and Bob is deploying a public cloud each with different
security requirements. Here we discuss how Alice and Bob would address
security concerns around the messaging service.
The message queue is a critical piece of infrastructure that supports a number of OpenStack services but is most strongly associated with the Compute service. Due to the nature of the message queue service, Alice and
Bob have similar security concerns. One of the larger concerns that remains
is that many systems have access to this queue and there is no way for a
consumer of the queue messages to verify which host or service placed the
messages on the queue. An attacker who is able to successfully place messages on the queue is able to create and delete VM instances, attach the
block storage of any tenant and a myriad of other malicious actions. There
are a number of solutions anticipated in the near future, with several proposals for message signing and encryption making their way through the
OpenStack development process.

Alice's private cloud

In this case, Alice's controls are the same as Bob's controls, which are described below.

Bob's public cloud

Bob assumes the infrastructure or networks underpinning the Compute
service could become compromised, therefore he recognizes the importance of hardening the system by restricting access to the message queue.
In order to accomplish this task Bob deploys his RabbitMQ servers with TLS
and X.509 client auth for access control. Hardening activities assists in limiting the capabilities of a malicious user that has compromised the system by
disallowing queue access, provided that this user does not have valid credentials to override the controls.
Additionally, Bob adds strong network ACL rulesets to enforce which endpoints can communicate with the message servers. This second control provides some additional assurance should the other protections fail.

128

OpenStack Security Guide

May 1, 2015

current

13. Data processing

Introduction to Data processing .........................................................
Deployment .......................................................................................
Configuration and hardening .............................................................
Case studies .......................................................................................

129
132
134
138

The Data processing service for OpenStack (sahara) provides a platform

for the provisioning and management of instance clusters using processing frameworks such as Hadoop and Spark. Through the OpenStack dashboard or REST API, users will be able to upload and execute framework applications which may access data in object storage or external providers.
The data processing controller uses the Orchestration service to create clusters of instances which may exist as long-running groups that can grow
and shrink as requested, or as transient groups created for a single workload.

usage in the section called Configuration and hardening [134].

Configuration and hardening

There are several configuration options and deployment strategies that
can improve security in the Data processing service. The service controller is
configured through a main configuration file and one or more policy files.
Installations that are using the data-locality features will also have two additional files to specify the physical location of Compute and Object Storage nodes.

TLS
The Data processing service controller, like many other OpenStack controllers, can be configured to require TLS connections.
Pre-Kilo releases will require a TLS proxy as the controller does not allow
direct TLS connections. Configuring TLS proxies is covered in the section
called TLS proxies and HTTP services [48], and we recommend following
the advice there to create this type of installation.
From the Kilo release onward the data processing controller allows direct
TLS connections. Enabling this behavior requires some small adjustments
to the controller configuration file. For any post-Juno installation we recommend enabling the direct TLS connections in the controller configuration.

Example. Configuring TLS access to the controller

[ssl]
ca_file = cafile.pem
cert_file = certfile.crt
key_file = keyfile.key

Role-based access control policies

The Data processing service uses a policy file, as described in the section
called Policies [70], to configure role-based access control. Using the policy file an operator can restrict a groups access to specific data processing
functionality.
The reasons for doing this will change depending on the organizational requirements of the installation. In general, these fine grained controls are
used in situations where an operator needs to restrict the creation, dele134

OpenStack Security Guide

May 1, 2015

current

tion, and retrieval of the Data processing service resources. Operators who
need to restrict access within a project should be fully aware that there will
need to be alternative means for users to gain access to the core functionality of the service (for example, provisioning clusters).

Example. Allow all methods to all users (default policy)

{
"default": ""
}

Example. Disallow image registry manipulations to non-admin users

{
"default": "",
"images:register": "role:admin",
"images:unregister": "role:admin",
"images:add_tags": "role:admin",
"images:remove_tags": "role:admin"
}

Security groups
The Data processing service allows for the association of security groups
with instances provisioned for its clusters. With no additional configuration the service will use the default security group for any project that provisions clusters. A different security group may be used if requested, or
an automated option exists which instructs the service to create a security
group based on ports specified by the framework being accessed.
For production environments we recommend controlling the security
groups manually and creating a set of group rules that are appropriate
for the installation. In this manner the operator can ensure that the default security group will contain all the appropriate rules. For an expanded discussion of security groups please see the section called Security
groups [120].

Proxy domains
When using the Object Storage service in conjunction with data processing it is necessary to add credentials for the store access. With proxy domains the Data processing service can instead use a delegated trust from
the Identity service to allow store access via a temporary user created in
the domain. For this delegation mechanism to work the Data processing
135

OpenStack Security Guide

May 1, 2015

current

service must be configured to use proxy domains and the operator must
configure an identity domain for the proxy users.
The data processing controller retains temporary storage of the username
and password provided for object store access. When using proxy domains
the controller will generate this pair for the proxy user, and the access of
this user will be limited to that of the identity trust. We recommend using
proxy domains in any installation where the controller or its database have
routes to or from public networks.

Example. Configuring for a proxy domain named

dp_proxy
[DEFAULT]
use_domain_for_proxy_users = true
proxy_user_domain_name = dp_proxy
proxy_user_role_names = Member

Custom network topologies

The data processing controller can be configured to use proxy commands
for accessing its cluster instances. In this manner custom network topologies can be created for installations which will not use the networks provided directly by the Networking service. We recommend using this option
for installations which require limiting access between the controller and
the instances.

Example. Access instances through a specified relay machine

[DEFAULT]
proxy_command='ssh relay-machine-{tenant_id} nc {host} {port}'

Example. Access instances through a custom network

namespace
[DEFAULT]
proxy_command='ip netns exec ns_for_{network_id} nc {host}
{port}'

Indirect access
For installations in which the controller will have limited access to all the instances of a cluster, due to limits on floating IP addresses or security rules,
indirect access may be configured. This allows some instances to be designated as proxy gateways to the other instances of the cluster.
136

OpenStack Security Guide

May 1, 2015

current

This configuration can only be enabled while defining the node group templates that will make up the data processing clusters. It is provided as a run
time option to be enabled during the cluster provisioning process.

Rootwrap
When creating custom topologies for network access it can be necessary to
allow non-root users the ability to run the proxy commands. For these situations the oslo rootwrap package is used to provide a facility for non-root
users to run privileged commands. This configuration requires the user associated with the data processing controller application to be in the sudoers list and for the option to be enabled in the configuration file. Optionally, an alternative rootwrap command can be provided.

Example. Enabling rootwrap usage and showing the default

command
[DEFAULT]
use_rootwrap=True
rootwrap_command=sudo sahara-rootwrap /etc/sahara/rootwrap.
conf

For more information on the rootwrap project, please see the official documentation:
https://wiki.openstack.org/wiki/Rootwrap

Logging
Monitoring the output of the service controller is a powerful forensic
tool, as described more thoroughly in Chapter18, Monitoring and logging [193]. The Data processing service controller offers a few options
for setting the location and level of logging.

Example. Setting the log level higher than warning and

specifying an output file.
[DEFAULT]
verbose = true
log_file = /var/log/data-processing.log

References
Sahara project documentation: http://docs.openstack.org/developer/sahara
137

OpenStack Security Guide

May 1, 2015

current

Hadoop project: https://hadoop.apache.org/

Hadoop secure mode docs: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SecureMode.html
Hadoop HDFS documentation: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
Spark project: https://spark.apache.org/
Spark security documentation: https://spark.apache.org/docs/latest/security.html
Storm project: https://storm.apache.org/
Zookeeper project: https://zookeeper.apache.org/
Oozie project: https://oozie.apache.org/
Hive https://hive.apache.org/
Pig https://pig.apache.org/
Cloudera CDH documentation: https://www.cloudera.com/content/cloudera/en/documentation.html#CDH
Hortonworks Data Platform documentation: http://
docs.hortonworks.com/
MapR project: https://www.mapr.com/products/mapr-distribution-including-apache-hadoop

Case studies
Continuing with the studies described in the section called Introduction to
case studies [21], we present Alice and Bob's approaches to deploying the
Data processing service for their users.

Alice's private cloud

Alice is deploying the Data processing service for a group of users that are
trusted members of a collaboration. They are all placed in a single project
and share the clusters, jobs, and data within. She deploys the controller
with TLS enabled, using a certificate signed by the organization's root certificate. She configures the controller to provide floating IP addresses to
138

OpenStack Security Guide

May 1, 2015

current

the cluster instances allowing for users to gain access to the instances in
the event of errors. She enables the use of proxy domains to prevent the
users from needing to enter their credentials into the Data processing service.

Bob's public cloud

Bob's public cloud contains users that will not necessarily know or trust
each other. He puts all users into separate projects. Each user has their
own clusters, jobs, and data which cannot be accessed by other users. He
deploys the controller with TLS enabled, using a certificate signed by a well
known public certificate authority. He configures a custom topology to ensure that access to the provisioned cluster instances will flow through a
controlled gateway. He creates a security group that opens only the ports
needed for the controller to access the frameworks deployed. He enables
the use of proxy domains to prevent the users from needing to enter their
credentials into the Data processing service. He configures the rootwrap
command to allow the data processing controller user to run the proxy
commands.

139

OpenStack Security Guide

May 1, 2015

current

14. Databases
Database back end considerations .....................................................
Database access control .....................................................................
Database transport security ................................................................
Case studies .......................................................................................

141
142
147
149

The choice of database server is an important consideration in the security

of an OpenStack deployment. Multiple factors should be considered when
deciding on a database server, however for the scope of this book only
security considerations will be discussed. OpenStack supports PostgreSQL
and MySQL database types.

Database back end considerations

PostgreSQL has a number of desirable security features such as Kerberos
authentication, object-level security, and encryption support. The PostgreSQL community has done well to provide solid guidance, documentation, and tooling to promote positive security practices.
MySQL has a large community, widespread adoption, and provides high
availability options. MySQL also has the ability to provide enhanced client
authentication by way of plug-in authentication mechanisms. Forked distributions in the MySQL community provide many options for consideration.
It is important to choose a specific implementation of MySQL based on a
thorough evaluation of the security posture and the level of support provided for the given distribution.

Security references for database back ends

Those deploying MySQL or PostgreSQL are advised to refer to existing security guidance. Some references are listed below:
MySQL:
OWASP MySQL Hardening
MySQL Pluggable Authentication
Security in MySQL
PostgreSQL:
141

OpenStack Security Guide

May 1, 2015

current

OWASP PostgreSQL Hardening

Total security in a PostgreSQL database

Database access control

Each of the core OpenStack services (Compute, Identity, Networking, Block
Storage) store state and configuration information in databases. In this
chapter, we discuss how databases are used currently in OpenStack. We
also explore security concerns, and the security ramifications of database
back end choices.

OpenStack database access model

All of the services within an OpenStack project access a single database.
There are presently no reference policies for creating table or row based
access restrictions to the database.
There are no general provisions for granular control of database operations in OpenStack. Access and privileges are granted simply based on
whether a node has access to the database or not. In this scenario, nodes
with access to the database may have full privileges to DROP, INSERT, or
UPDATE functions.

Granular access control

By default, each of the OpenStack services and their processes access the
database using a shared set of credentials. This makes auditing database
operations and revoking access privileges from a service and its processes
to the database particularly difficult.

142

OpenStack Security Guide

May 1, 2015

current

Nova-conductor
The compute nodes are the least trusted of the services in OpenStack because they host tenant instances. The nova-conductor service has been
introduced to serve as a database proxy, acting as an intermediary between the compute nodes and the database. We discuss its ramifications
later in this chapter.
We strongly recommend:
All database communications be isolated to a management network
Securing communications using TLS
Creating unique database user accounts per OpenStack service endpoint
(illustrated below)

143

OpenStack Security Guide

May 1, 2015

current

Database authentication and access control

Given the risks around access to the database, we strongly recommend
that unique database user accounts be created per node needing access
to the database. Doing this facilitates better analysis and auditing for
ensuring compliance or in the event of a compromise of a node allows
you to isolate the compromised host by removing access for that node
to the database upon detection. When creating these per service endpoint database user accounts, care should be taken to ensure that they are
configured to require TLS. Alternatively, for increased security it is recommended that the database accounts be configured using X.509 certificate
authentication in addition to user names and passwords.

Privileges
A separate database administrator (DBA) account should be created and
protected that has full privileges to create/drop databases, create user accounts, and update user privileges. This simple means of separation of responsibility helps prevent accidental misconfiguration, lowers risk and lowers scope of compromise.
144

OpenStack Security Guide

May 1, 2015

current

The database user accounts created for the OpenStack services and for
each node should have privileges limited to just the database relevant to
the service where the node is a member.

Require user accounts to require SSL transport

Configuration example #1: (MySQL)
GRANT ALL ON dbname.* to 'compute01'@'hostname' IDENTIFIED BY
'NOVA_DBPASS' REQUIRE SSL;

Configuration example #2: (PostgreSQL)

In file pg_hba.conf:
hostssl dbname compute01 hostname md5

Note this command only adds the ability to communicate over SSL and is
non-exclusive. Other access methods that may allow unencrypted transport should be disabled so that SSL is the sole access method.
The md5 parameter defines the authentication method as a hashed password. We provide a secure authentication example in the section below.

OpenStack service database configuration

If your database server is configured for TLS transport, you will need to
specify the certificate authority information for use with the initial connection string in the SQLAlchemy query.

Example of a :sql_connection string to MySQL:

sql_connection = mysql://compute01:NOVA_DBPASS@localhost/nova?
charset=utf8&ssl_ca=/etc/mysql/cacert.pem

Authentication with X.509 certificates

Security may be enhanced by requiring X.509 client certificates for authentication. Authenticating to the database in this manner provides greater
identity assurance of the client making the connection to the database and
ensures that the communications are encrypted.

Configuration example #1: (MySQL)

GRANT ALL on dbname.* to 'compute01'@'hostname' IDENTIFIED BY
'NOVA_DBPASS' REQUIRE SUBJECT
'/C=XX/ST=YYY/L=ZZZZ/O=cloudycloud/CN=compute01' AND ISSUER
'/C=XX/ST=YYY/L=ZZZZ/O=cloudycloud/CN=cloud-ca';

145

OpenStack Security Guide

current

isolated management network. This is achieved by restricting the interface

or IP address on which the database server binds a network socket for incoming client connections.

Restricting bind address for MySQL

In my.cnf:
[mysqld]
...
bind-address <ip address or hostname of management network
interface>

Restricting listen address for PostgreSQL

In postgresql.conf:
listen_addresses = <ip address or hostname of management network
interface>

Database transport
In addition to restricting database communications to the management
network, we also strongly recommend that the cloud administrator configure their database back end to require TLS. Using TLS for the database
client connections protects the communications from tampering and
eavesdropping. As will be discussed in the next section, using TLS also
provides the framework for doing database user authentication through
X.509 certificates (commonly referred to as PKI). Below is guidance on
how TLS is typically configured for the two popular database back ends
MySQL and PostgreSQL.

Note
When installing the certificate and key files, ensure that the file
permissions are restricted, for example chmod 0600, and the
ownership is restricted to the database daemon user to prevent unauthorized access by other processes and users on the
database server.

MySQL SSL configuration

The following lines should be added in the system-wide MySQL configuration file:
148

OpenStack Security Guide

May 1, 2015

current

In my.cnf:
[[mysqld]]
...
ssl-ca=/path/to/ssl/cacert.pem
ssl-cert=/path/to/ssl/server-cert.pem
ssl-key=/path/to/ssl/server-key.pem

Optionally, if you wish to restrict the set of SSL ciphers used for the encrypted connection. See http://www.openssl.org/docs/apps/ciphers.html
for a list of ciphers and the syntax for specifying the cipher string:
ssl-cipher='cipher:list'

PostgreSQL SSL configuration

The following lines should be added in the system-wide PostgreSQL configuration file, postgresql.conf.
ssl = true

The server certificate, key, and certificate authority (CA) files should be
placed in the $PGDATA directory in the following files:
$PGDATA/server.crt - Server certificate
$PGDATA/server.key - Private key corresponding to server.crt
$PGDATA/root.crt - Trusted certificate authorities
$PGDATA/root.crl - Certificate revocation list

OpenStack Security Guide

May 1, 2015

current

Alice's private cloud

Alice's organization has high availability concerns and so she has selected
MySQL as the underlying database for the cloud services. She places the
database on the Management network, utilizing SSL/TLS with mutual authentication among the services to ensure secure access. Based on the assumption that external access of the database will not be facilitated, she
installs a certificate signed with the organization's root certificate on the
database and its access endpoints. Alice creates separate user accounts for
each database user then configures the database to use both passwords
and X.509 certificates for authentication. She elects not to use the nova-conductor sub-service due to the desire for fine-grained access control policies and audit support.

Bob's public cloud

Bob is concerned about strong separation of his tenants' data, so he has
elected to use the PostgreSQL database, known for its stronger security
features. The database resides on the Management network and uses SSL/
TLS with mutual authentication with the services. Since the database is
on the Management network, the database uses certificates signed with
the company's self-signed root certificate. Bob creates separate user accounts for each database user, and configures the database to use both
passwords and X.509 certificates for authentication. He elects not to use
the nova-conductor sub-service due to a desire for fine-grained access
control.

150

OpenStack Security Guide

May 1, 2015

current

15. Tenant data privacy

Data privacy concerns ........................................................................
Data encryption .................................................................................
Key management ...............................................................................
Case studies .......................................................................................

151
155
158
159

OpenStack is designed to support multitenancy and those tenants will

most probably have different data requirements. As a cloud builder and
operator you need to ensure your OpenStack environment can address
various data privacy concerns and regulations. In this chapter we will address data residency and disposal as it pertains to OpenStack implementations.

Data privacy concerns

Data residency
The privacy and isolation of data has consistently been cited as the primary barrier to cloud adoption over the past few years. Concerns over who
owns data in the cloud and whether the cloud operator can be ultimately
trusted as a custodian of this data have been significant issues in the past.
Numerous OpenStack services maintain data and metadata belonging to
tenants or reference tenant information.
Tenant data stored in an OpenStack cloud may include the following
items:
Object Storage objects
Compute instance ephemeral filesystem storage
Compute instance memory
Block Storage volume data
Public keys for Compute access
Virtual machine images in the Image service
Machine snapshots
Data passed to OpenStack Compute's configuration-drive extension
151

OpenStack Security Guide

May 1, 2015

current

Metadata stored by an OpenStack cloud includes the following non-exhaustive items:

Organization name
User's "Real Name"
Number or size of running instances, buckets, objects, volumes, and other quota-related items
Number of hours running instances or storing data
IP addresses of users
Internally generated private keys for compute image bundling

Data disposal
OpenStack operators should strive to provide a certain level of tenant data
disposal assurance. Best practices suggest that the operator sanitize cloud
system media (digital and non-digital) prior to disposal, release out of organization control or release for reuse. Sanitization methods should implement an appropriate level of strength and integrity given the specific security domain and sensitivity of the information.
"The sanitization process removes information from the
media such that the information cannot be retrieved or reconstructed. Sanitization techniques, including clearing,
purging, cryptographic erase, and destruction, prevent the
disclosure of information to unauthorized individuals when
such media is reused or released for disposal." NIST Special
Publication 800-53 Revision 4
General data disposal and sanitization guidelines as adopted from NIST
recommended security controls. Cloud operators should:
1. Track, document and verify media sanitization and disposal actions.
2. Test sanitation equipment and procedures to verify proper performance.
3. Sanitize portable, removable storage devices prior to connecting such
devices to the cloud infrastructure.
4. Destroy cloud system media that cannot be sanitized.
152

OpenStack Security Guide

May 1, 2015

A bare metal server driver for Compute was under development and has
since moved into a separate project called ironic. At the time of this writing, ironic does not appear to address sanitization of tenant data resident
the physical hardware.
Additionally, it is possible for tenants of a bare metal system to modify
system firmware. TPM technology, described in the section called Secure bootstrapping [32], provides a solution for detecting unauthorized
firmware changes.

Data encryption
The option exists for implementers to encrypt tenant data wherever it is
stored on disk or transported over a network, such as the OpenStack volume encryption feature described below. This is above and beyond the
general recommendation that users encrypt their own data before sending
it to their provider.
The importance of encrypting data on behalf of tenants is largely related to the risk assumed by a provider that an attacker could access tenant
data. There may be requirements here in government, as well as requirements per-policy, in private contract, or even in case law in regard to private contracts for public cloud providers. It is recommended that a risk assessment and legal consul advised before choosing tenant encryption policies.
Per-instance or per-object encryption is preferable over, in descending order, per-project, per-tenant, per-host, and per-cloud aggregations. This recommendation is inverse to the complexity and difficulty of implementation. Presently, in some projects it is difficult or impossible to implement
155

OpenStack Security Guide

Bibliography:
OpenStack.org, Welcome to Barbican's Developer Documentation!.
2014. Barbican developer documentation
oasis-open.org, OASIS Key Management Interoperability Protocol
(KMIP). 2014. KMIP
PyKMIP library https://github.com/OpenKMIP/PyKMIP

Alice's private cloud

Product or project maturity

The maturity of a given hypervisor product or project is critical to your security posture as well. Product maturity has a number of effects once you
have deployed your cloud:
Availability of expertise
Active developer and user communities
162

OpenStack Security Guide

May 1, 2015

current

Timeliness and availability of updates

Incidence response
One of the biggest indicators of a hypervisor's maturity is the size and vibrancy of the community that surrounds it. As this concerns security, the
quality of the community affects the availability of expertise if you need
additional cloud operators. It is also a sign of how widely deployed the hypervisor is, in turn leading to the battle readiness of any reference architectures and best practices.
Further, the quality of community, as it surrounds an open source hypervisor like KVM or Xen, has a direct impact on the timeliness of bug fixes and
security updates. When investigating both commercial and open source
hypervisors, you must look into their release and support cycles as well as
the time delta between the announcement of a bug or security issue and
a patch or response. Lastly, the supported capabilities of OpenStack compute vary depending on the hypervisor chosen. See the OpenStack Hypervisor Support Matrix for OpenStack compute feature support by hypervisor.

Certifications and attestations

One additional consideration when selecting a hypervisor is the availability of various formal certifications and attestations. While they may not be
requirements for your specific organization, these certifications and attestations speak to the maturity, production readiness, and thoroughness of
the testing a particular hypervisor platform has been subjected to.

Common criteria
Common Criteria is an internationally standardized software evaluation
process, used by governments and commercial companies to validate software technologies perform as advertised. In the government sector, NSTISSP No. 11 mandates that U.S. Government agencies only procure software
which has been Common Criteria certified, a policy which has been in place
since July 2002. It should be specifically noted that OpenStack has not undergone Common Criteria certification, however many of the available hypervisors have.
In addition to validating a technologies capabilities, the Common Criteria
process evaluates how technologies are developed.
How is source code management performed?
How are users granted access to build systems?
163

OpenStack Security Guide

May 1, 2015

current

Is the technology cryptographically signed before distribution?

The KVM hypervisor has been Common Criteria certified through the U.S.
Government and commercial distributions, which have been validated to
separate the runtime environment of virtual machines from each other,
providing foundational technology to enforce instance isolation. In addition to virtual machine isolation, KVM has been Common Criteria certified
to
"provide system-inherent separation mechanisms to the resources of virtual machines. This separation ensures that
large software component used for virtualizing and simulating devices executing for each virtual machine cannot interfere with each other. Using the SELinux multi-category mechanism, the virtualization and simulation software instances are isolated. The virtual machine management framework configures SELinux multi-category settings
transparently to the administrator"
While many hypervisor vendors, such as Red Hat, Microsoft, and VMware
have achieved Common Criteria Certification their underlying certified feature set differs. It is recommended to evaluate vendor claims to ensure
they minimally satisfy the following requirements:
Identification and Authentication

Identification and authentication using pluggable authentication modules

(PAM) based upon user passwords.
The quality of the passwords used can
be enforced through configuration
options.

Audit

The system provides the capability to

audit a large number of events including individual system calls as well as
events generated by trusted processes. Audit data is collected in regular
files in ASCII format. The system provides a program for the purpose of
searching the audit records.
The system administrator can define
a rule base to restrict auditing to the
events they are interested in. This includes the ability to restrict auditing
to specific events, specific users, specific objects or a combination of all of
this.
Audit records can be transferred to a
remote audit daemon.

Discretionary Access Control

Discretionary Access Control (DAC)

restricts access to file system objects

164

OpenStack Security Guide

May 1, 2015
based on Access Control Lists (ACLs)
that include the standard UNIX permissions for user, group and others.
Access control mechanisms also protect IPC objects from unauthorized
access.
The system includes the ext4 file system, which supports POSIX ACLs. This
allows defining access rights to files
within this type of file system down
to the granularity of a single user.

Mandatory Access Control

Mandatory Access Control (MAC) restricts access to objects based on labels assigned to subjects and objects.
Sensitivity labels are automatically attached to processes and objects. The
access control policy enforced using
these labels is derived from the BellLaPadula access control model.
SELinux categories are attached to
virtual machines and its resources.
The access control policy enforced using these categories grant virtual machines access to resources if the category of the virtual machine is identical to the category of the accessed resource.
The TOE implements non-hierarchical
categories to control access to virtual
machines.

Role-Based Access Control

Role-based access control (RBAC) allows separation of roles to eliminate

the need for an all-powerful system
administrator.

Object Reuse

File system objects and memory and

IPC objects are cleared before they
can be reused by a process belonging
to a different user.

Security Management

The management of the security critical parameters of the system is performed by administrative users. A set
of commands that require root privileges (or specific roles when RBAC
is used) are used for system management. Security parameters are stored
in specific files that are protected by
the access control mechanisms of the
system against unauthorized access
by users that are not administrative
users.

Secure Communication

The system supports the definition

of trusted channels using SSH. Pass-

165

current

OpenStack Security Guide

May 1, 2015

current

word based authentication is supported. Only a restricted number of cipher

suites are supported for those protocols in the evaluated configuration.
Storage Encryption

The system supports encrypted block

devices to provide storage confidentiality via dm_crypt.

TSF Protection

While in operation, the kernel software and data are protected by the
hardware memory protection mechanisms. The memory and process management components of the kernel
ensure a user process cannot access
kernel storage or storage belonging
to other processes.
Non-kernel TSF software and data
are protected by DAC and process
isolation mechanisms. In the evaluated configuration, the reserved user
ID root owns the directories and files
that define the TSF configuration. In
general, files and directories containing internal TSF data, such as configuration files and batch job queues, are
also protected from reading by DAC
permissions.
The system and the hardware and
firmware components are required
to be physically protected from unauthorized access. The system kernel
mediates all access to the hardware
mechanisms themselves, other than
program visible CPU instruction functions.
In addition, mechanisms for protection against stack overflow attacks
are provided.

Cryptography standards
Several cryptography algorithms are available within OpenStack for identification and authorization, data transfer and protection of data at rest.
When selecting a hypervisor, the following are recommended algorithms
and implementation standards to ensure the virtualization layer supports:
Algorithm

Key length

Intended pur- Security func- Implementapose

tion
tion standard

AES

128, 192, or
256 bits

Encryption /
decryption

166

Protected da- RFC 4253

ta transfer,
protection for
data at rest

OpenStack Security Guide

May 1, 2015

current

Algorithm

Key length

Intended pur- Security func- Implementapose

tion
tion standard

TDES

168 bits

Encryption /
decryption

Protected data transfer

RFC 4253

RSA

1024, 2048, or Authentica3072 bits

tion, key exchange

Identification
and authentication, protected data
transfer

U.S. NIST FIPS

PUB 186-3

DSA

L=1024,
N=160 bits

Authentication, key exchange

Identification
and authentication, protected data
transfer

U.S. NIST FIPS

PUB 186-3

Serpent

128, 192, or
256 bits

Encryption /
decryption

Protection of
data at rest

http://
www.cl.cam.ac.uk/
~rja14/Papers/serpent.pdf

Twofish

128, 192, or
256 bit

Encryption /
decryption

Protection of
data at rest

http://
www.schneier.com/
paper-twofishpaper.html

SHA-1

Message Digest

Protection of U.S. NIST FIPS

data at rest,
PUB 180-3
protected data transfer

SHA-2 (224,
256, 384, or
512 bits)

Message Digest

Protection for U.S. NIST FIPS

data at rest,
PUB 180-3
identification
and authentication

FIPS 140-2
In the United States the National Institute of Science and Technology
(NIST) certifies cryptographic algorithms through a process known the
Cryptographic Module Validation Program. NIST certifies algorithms for
conformance against Federal Information Processing Standard 140-2 (FIPS
140-2), which ensures:
Products validated as conforming to FIPS 140-2 are accepted by the Federal agencies of both countries [United States
and Canada] for the protection of sensitive information
(United States) or Designated Information (Canada). The
goal of the CMVP is to promote the use of validated cryptographic modules and provide Federal agencies with a security metric to use in procuring equipment containing validated cryptographic modules.
167

OpenStack Security Guide

May 1, 2015

current

When evaluating base hypervisor technologies, consider if the hypervisor

has been certified against FIPS 140-2. Not only is conformance against FIPS
140-2 mandated per U.S. Government policy, formal certification indicates
that a given implementation of a cryptographic algorithm has been reviewed for conformance against module specification, cryptographic module ports and interfaces; roles, services, and authentication; finite state
model; physical security; operational environment; cryptographic key management; electromagnetic interference/electromagnetic compatibility
(EMI/EMC); self-tests; design assurance; and mitigation of other attacks.

Hardware concerns
Further, when you evaluate a hypervisor platform, consider the supportability of the hardware on which the hypervisor will run. Additionally, consider the additional features available in the hardware and how those features are supported by the hypervisor you chose as part of the OpenStack
deployment. To that end, hypervisors each have their own hardware compatibility lists (HCLs). When selecting compatible hardware it is important
to know in advance which hardware-based virtualization technologies are
important from a security perspective.
Description

Technology

Explanation

I/O MMU

VT-d / AMD-Vi

Required for protecting

PCI-passthrough

Intel Trusted Execution

Technology

Intel TXT / SEM

Required for dynamic attestation services

PCI-SIG I/O virtualization SR-IOV, MR-IOV, ATS

Required to allow secure

sharing of PCI Express
devices

Network virtualization

Improves performance
of network I/O on hypervisors

VT-c

Hypervisor vs. baremetal

It is important to recognize the difference between using LXC (Linux Containers) or baremetal systems vs using a hypervisor like KVM. Specifically, the focus of this security guide is largely based on having a hypervisor
and virtualization platform. However, should your implementation require
the use of a baremetal or LXC environment, you must pay attention to the
particular differences in regard to deployment of that environment.
In particular, you must assure your end users that the node has been properly sanitized of their data prior to re-provisioning. Additionally, prior to
168

OpenStack Security Guide

May 1, 2015

current

reusing a node, you must provide assurances that the hardware has not
been tampered or otherwise compromised.

Note
While OpenStack has a baremetal project, a discussion of the
particular security implications of running baremetal is beyond
the scope of this book.
Finally, due to the time constraints around a book sprint, the team chose
to use KVM as the hypervisor in our example implementations and architectures.

Note
There is an OpenStack Security Note pertaining to the use of
LXC in Compute.

Hypervisor memory optimization

Many hypervisors use memory optimization techniques to overcommit
memory to guest virtual machines. This is a useful feature that allows you
to deploy very dense compute clusters. One way to achieve this is through
de-duplication or "sharing" of memory pages. When two virtual machines
have identical data in memory, there are advantages to having them reference the same memory.
Typically this is achieved through Copy-On-Write (COW) mechanisms.
These mechanisms have been shown to be vulnerable to side-channel attacks where one VM can infer something about the state of another and
might not be appropriate for multi-tenant environments where not all tenants are trusted or share the same levels of trust.

KVM Kernel Samepage Merging

Introduced into the Linux kernel in version 2.6.32, Kernel Samepage Merging (KSM) consolidates identical memory pages between Linux processes.
As each guest VM under the KVM hypervisor runs in its own process, KSM
can be used to optimize memory use between VMs.

XEN transparent page sharing

XenServer 5.6 includes a memory overcommitment feature named Transparent Page Sharing (TPS). TPS scans memory in 4 KB chunks for any dupli169

OpenStack Security Guide

May 1, 2015

current

cates. When found, the Xen Virtual Machine Monitor (VMM) discards one
of the duplicates and records the reference of the second one.

Security considerations for memory optimization

Traditionally, memory de-duplication systems are vulnerable to side channel attacks. Both KSM and TPS have demonstrated to be vulnerable to
some form of attack. In academic studies attackers were able to identify
software packages and versions running on neighboring virtual machines
as well as software downloads and other sensitive information through analyzing memory access times on the attacker VM.
If a cloud deployment requires strong separation of tenants, as is the situation with public clouds and some private clouds, deployers should consider
disabling TPS and KSM memory optimizations.

Additional security features

Another thing to look into when selecting a hypervisor platform is the
availability of specific security features. In particular, we are referring to
features like Xen Server's XSM or Xen Security Modules, sVirt, Intel TXT,
and AppArmor. The presence of these features increase your security profile as well as provide a good foundation.
The following table calls out these features by common hypervisor platforms.
XSM

sVirt

TXT

AppArmor

cgroups

MAC Policy

KVM
Xen
ESXi
Hyper-V

MAC Policy: Mandatory Access Control; may be implemented with SELinux

or other operating systems
* Features in this table might not be applicable to all hypervisors or directly
mappable between hypervisors.

Bibliography
Sunar, Eisenbarth, Inci, Gorka Irazoqui Apecechea. Fine Grain CrossVM Attacks on Xen and VMware are possible!. 2014. https://
eprint.iacr.org/2014/248.pdf
170

OpenStack Security Guide

May 1, 2015

current

Artho, Yagi, Iijima, Kuniyasu Suzaki. Memory Deduplication as a Threat

to the Guest OS. 2011. https://staff.aist.go.jp/c.artho/papers/EuroSec2011-suzaki.pdf
KVM: Kernal-based Virtual Machine. Kernal Samepage Merging. 2010.
KVM: Kernel Samepage Merging
Xen Project, Xen Security Modules: XSM-FLASK. 2014. XSM: Xen Security
Modules
SELinux Project, SVirt. 2011. xVirt: Mandatory Access Control for Linux-based virtualization
Intel.com, Trusted Compute Pools with Intel Trusted Execution Technology (Intel TXT). TXT: Intel Trusted Execution Technology
AppArmor.net, AppArmor Main Page. 2011. AppArmor: Linux security
module implementing MAC
Kernal.org, CGroups. 2004. cgroups: Linux kernel feature to control resource usage
Computer Security Resource Centre. Guide to Security for Full Virtualization Technologies. 2011. Guide to Security for Full Virtualization Technologies
National Information Assurance Partnership, National Security Telecommunications and Information Systems Security Policy. 2003. National Security Telecommunications and Information Systems Security Policy No.
11

Hardening the virtualization layers

In the beginning of this chapter we discuss the use of both physical and virtual hardware by instances, the associated security risks, and some recommendations for mitigating those risks. We conclude the chapter with a discussion of sVirt, an open source project for integrating SELinux mandatory
access controls with the virtualization components.

Physical hardware (PCI passthrough)

Many hypervisors offer a functionality known as PCI passthrough. This allows an instance to have direct access to a piece of hardware on the node.
For example, this could be used to allow instances to access video cards or
171

OpenStack Security Guide

May 1, 2015

current

GPUs offering the compute unified device architecture (CUDA) for high
performance computation. This feature carries two types of security risks:
direct memory access and hardware infection.
Direct memory access (DMA) is a feature that permits certain hardware devices to access arbitrary physical memory addresses in the host computer.
Often video cards have this capability. However, an instance should not be
given arbitrary physical memory access because this would give it full view
of both the host system and other instances running on the same node.
Hardware vendors use an input/output memory management unit (IOMMU) to manage DMA access in these situations. Therefore, cloud architects
should ensure that the hypervisor is configured to utilize this hardware feature.
KVM: How to assign devices with VT-d in KVM
Xen: VTd Howto

Note
The IOMMU feature is marketed as VT-d by Intel and AMD-Vi
by AMD.
A hardware infection occurs when an instance makes a malicious modification to the firmware or some other part of a device. As this device is used
by other instances or the host OS, the malicious code can spread into those
systems. The end result is that one instance can run code outside of its security domain. This is a significant breach as it is harder to reset the state of
physical hardware than virtual hardware, and can lead to additional exposure such as access to the management network.
Solutions to the hardware infection problem are domain specific. The strategy is to identify how an instance can modify hardware state then determine how to reset any modifications when the instance is done using the
hardware. For example, one option could be to re-flash the firmware after use. Clearly there is a need to balance hardware longevity with security as some firmwares will fail after a large number of writes. TPM technology, described in the section called Secure bootstrapping [32], provides
a solution for detecting unauthorized firmware changes. Regardless of the
strategy selected, it is important to understand the risks associated with
this kind of hardware sharing so that they can be properly mitigated for a
given deployment scenario.
Additionally, due to the risk and complexities associated with PCI
passthrough, it should be disabled by default. If enabled for a specific
172

OpenStack Security Guide

May 1, 2015

current

need, you will need to have appropriate processes in place to ensure the
hardware is clean before re-issue.

Virtual hardware (QEMU)

When running a virtual machine, virtual hardware is a software layer that
provides the hardware interface for the virtual machine. Instances use this
functionality to provide network, storage, video, and other devices that
may be needed. With this in mind, most instances in your environment will
exclusively use virtual hardware, with a minority that will require direct
hardware access. The major open source hypervisors use QEMU for this
functionality. While QEMU fills an important need for virtualization platforms, it has proven to be a very challenging software project to write and
maintain. Much of the functionality in QEMU is implemented with low-level code that is difficult for most developers to comprehend. Furthermore,
the hardware virtualized by QEMU includes many legacy devices that have
their own set of quirks. Putting all of this together, QEMU has been the
source of many security problems, including hypervisor breakout attacks.
Therefore, it is important to take proactive steps to harden QEMU. Three
specific steps are recommended: minimizing the code base, using compiler
hardening, and using mandatory access controls such as sVirt, SELinux, or
AppArmor.
Additionally, ensure iptables has the default policy filtering network traffic, and consider examining the existing rule set to understand each rule
and determine if the policy needs to be expanded upon.

Minimizing the QEMU code base

The first recommendation is to minimize the QEMU code base by removing
unused components from the system. QEMU provides support for many
different virtual hardware devices, however only a small number of devices
are needed for a given instance. The most common hardware devices are
the virtio devices. Some legacy instances will need access to specific hardware, which can be specified using glance metadata:
$ glance image-update \
--property hw_disk_bus=ide \
--property hw_cdrom_bus=ide \
--property hw_vif_model=e1000 \
f16-x86_64-openstack-sda

A cloud architect should decide what devices to make available to cloud

users. Anything that is not needed should be removed from QEMU. This
step requires recompiling QEMU after modifying the options passed to the
173

OpenStack Security Guide

May 1, 2015

current

QEMU configure script. For a complete list of up-to-date options simply run
./configure --help from within the QEMU source directory. Decide what is
needed for your deployment, and disable the remaining options.

Compiler hardening
The next step is to harden QEMU using compiler hardening options. Modern compilers provide a variety of compile time options to improve the security of the resulting binaries. These features, which we will describe in
more detail below, include relocation read-only (RELRO), stack canaries,
never execute (NX), position independent executable (PIE), and address
space layout randomization (ASLR).
Many modern Linux distributions already build QEMU with compiler hardening enabled, so you may want to verify your existing executable before
proceeding with the information below. One tool that can assist you with
this verification is called checksec.sh.
RELocation Read-Only (RELRO)

Hardens the data sections of an executable. Both full and partial RELRO
modes are supported by gcc. For QEMU full RELRO is your best choice. This
will make the global offset table readonly and place various internal data
sections before the program data section in the resulting executable.

Stack canaries

Places values on the stack and verifies

their presence to help prevent buffer
overflow attacks.

Never eXecute (NX)

Also known as Data Execution Prevention (DEP), ensures that data sections
of the executable can not be executed.

Position Independent Executable (PIE)

Produces a position independent executable, which is necessary for ASLR.

Address Space Layout Randomization (ASLR)

This ensures that placement of both

code and data regions will be randomized. Enabled by the kernel (all modern
Linux kernels support ASLR), when the
executable is built with PIE.

The following compiler options are recommend for GCC when compiling
QEMU:
174

OpenStack Security Guide

May 1, 2015

current

CFLAGS="-arch x86_64 -fstack-protector-all -Wstack-protector \

--param ssp-buffer-size=4 -pie -fPIE -ftrapv -D_FORTIFY_SOURCE=
2 -O2 \
-Wl,-z,relro,-z,now"

We recommend testing your QEMU executable file after it is compiled to

ensure that the compiler hardening worked properly.
Most cloud deployments will not want to build software such as QEMU by
hand. It is better to use packaging to ensure that the process is repeatable
and to ensure that the end result can be easily deployed throughout the
cloud. The references below provide some additional details on applying
compiler hardening options to existing packages.
DEB packages: Hardening Walkthrough
RPM packages: How to create an RPM package

Mandatory access controls

Compiler hardening makes it more difficult to attack the QEMU process.
However, if an attacker does succeed, we would like to limit the impact
of the attack. Mandatory access controls accomplish this by restricting the
privileges on QEMU process to only what is needed. This can be accomplished using sVirt / SELinux or AppArmor. When using sVirt, SELinux is
configured to run each QEMU process under a separate security context.
AppArmor can be configured to provide similar functionality. We provide
more details on sVirt and instance isolation in the section below.

sVirt: SELinux and virtualization

With unique kernel-level architecture and National Security Agency (NSA)
developed security mechanisms, KVM provides foundational isolation
technologies for multi-tenancy. With developmental origins dating back
to 2002, the Secure Virtualization (sVirt) technology is the application of
SELinux against modern day virtualization. SELinux, which was designed to
apply separation control based upon labels, has been extended to provide
isolation between virtual machine processes, devices, data files and system
processes acting upon their behalf.
OpenStack's sVirt implementation aspires to protect hypervisor hosts and
virtual machines against two primary threat vectors:
Hypervisor threats

A compromised application running

within a virtual machine attacks the hy175

OpenStack Security Guide

May 1, 2015

current

pervisor to access underlying resources.

For example, when a virtual machine is
able to access the hypervisor OS, physical devices, or other applications. This
threat vector represents considerable
risk as a compromise on a hypervisor
can infect the physical hardware as well
as exposing other virtual machines and
network segments.
Virtual Machine (multi-tenant)
threats

A compromised application running

within a VM attacks the hypervisor to
access or control another virtual machine and its resources. This is a threat
vector unique to virtualization and represents considerable risk as a multitude
of virtual machine file images could be
compromised due to vulnerability in a
single application. This virtual network
attack is a major concern as the administrative techniques for protecting real
networks do not directly apply to the
virtual environment.

Each KVM-based virtual machine is a process which is labeled by SELinux,

effectively establishing a security boundary around each virtual machine.
This security boundary is monitored and enforced by the Linux kernel, restricting the virtual machine's access to resources outside of its boundary
such as host machine data files or other VMs.

176

OpenStack Security Guide

May 1, 2015

current

As shown above, sVirt isolation is provided regardless of the guest Operating System running inside the virtual machineLinux or Windows VMs can
be used. Additionally, many Linux distributions provide SELinux within the
operating system, allowing the virtual machine to protect internal virtual
resources from threats.

Labels and categories

KVM-based virtual machine instances are labelled with their own SELinux
data type, known as svirt_image_t. Kernel level protections prevent unauthorized system processes, such as malware, from manipulating the virtual machine image files on disk. When virtual machines are powered off, images are stored as svirt_image_t as shown below:
system_u:object_r:svirt_image_t:SystemLow
system_u:object_r:svirt_image_t:SystemLow
system_u:object_r:svirt_image_t:SystemLow
system_u:object_r:svirt_image_t:SystemLow

image1
image2
image3
image4

The svirt_image_t label uniquely identifies image files on disk, allowing for the SELinux policy to restrict access. When a KVM-based Compute
image is powered on, sVirt appends a random numerical identifier to the
image. sVirt is capable of assigning numeric identifiers to a maximum of
524,288 virtual machines per hypervisor node, however most OpenStack
deployments are highly unlikely to encounter this limitation.
This example shows the sVirt category identifier:
system_u:object_r:svirt_image_t:s0:c87,c520 image1
system_u:object_r:svirt_image_t:s0:419,c172 image2

SELinux users and roles

SELinux can also manage user roles. These can be viewed through the -Z
flag, or with the semanage command. On the hypervisor, only administrators should be able to access the system, and should have an appropriate
context around both the administrative users and any other users that are
on the system.
SELinux users documentation: SELinux.org Users and Roles Overview

Booleans
To ease the administrative burden of managing SELinux, many enterprise
Linux platforms utilize SELinux Booleans to quickly change the security posture of sVirt.
177

OpenStack Security Guide

May 1, 2015

current

Red Hat Enterprise Linux-based KVM deployments utilize the following

sVirt booleans:
sVirt SELinux Boolean

Description

virt_use_common

Allow virt to use serial/parallel communication ports.

virt_use_fusefs

Allow virt to read FUSE mounted files.

virt_use_nfs

Allow virt to manage NFS mounted

files.

virt_use_samba

Allow virt to manage CIFS mounted

files.

virt_use_sanlock

Allow confined virtual guests to interact with the sanlock.

virt_use_sysfs

Allow virt to manage device configuration (PCI).

virt_use_usb

Allow virt to use USB devices.

virt_use_xserver

Allow virtual machine to interact with

the X Window System.

Alice's private cloud

Alice chooses Xen for the hypervisor in her cloud due to a strong internal
knowledge base and a desire to use the Xen security modules (XSM) for
fine-grained policy enforcement.
Alice is willing to apply a relatively large amount of resources to software
packaging and maintenance. She will use these resources to build a highly
customized version of QEMU that has many components removed, thereby reducing the attack surface. She will also ensure that all compiler hardening options are enabled for QEMU. Alice accepts that these decisions will
increase long-term maintenance costs.
Alice writes XSM policies (for Xen) and SELinux policies (for Linux domain 0, and device domains) to provide stronger isolation between the in178

OpenStack Security Guide

May 1, 2015

current

stances. Alice also uses the Intel TXT support in Xen to measure the hypervisor launch in the TPM.

Bob's public cloud

These filters will create an instance

based on the utilizations of the hypervisor host sets and can trigger on free
or used properties such as RAM, IO, or
CPU utilization

Image based filters

This delegates instance creation based

on the image used, such as the operating system of the VM or type of image
used.

Environment based filters

This filter will create an instance based

on external details such as in a specific
IP range, across availability zones, or on
the same host as another instance.

Custom criteria

This filter will delegate instance creation based on user or administrator

provided criteria such as trusts or metadata parsing.

Multiple filters can be applied at once, such as the ServerGroupAffinity filter to ensure an instance is created on a member of a specific set
of hosts and ServerGroupAntiAffinity filter to ensure that same instance is not created on another specific set of hosts. These filters should
be analyzed carefully to ensure they do not conflict with each other and
result in rules that prevent the creation of instances.

183

OpenStack Security Guide

May 1, 2015

current

The GroupAffinity and GroupAntiAffinity filters conflict and

should not both be enabled at the same time.
The DiskFilter filter is capable of oversubscribing disk space. While not
normally an issue, this can be a concern on storage devices that are thinly
provisioned, and this filter should be used with well-tested quotas applied.
We recommend you disable filters that parse things that are provided by
users or are able to be manipulated such as metadata.

Trusted images
In a cloud environment, users work with either pre-installed images or images they upload themselves. In both cases, users should be able to ensure
the image they are utilizing has not been tampered with. This requires a
method of validation, such as a checksum for the known good image as
well as verification of a running instance. While there are current best practices around these actions there are also several gaps in the process.

Image creation process

The OpenStack Documentation provides guidance on how to create and
upload an image to the Image service. Additionally it is assumed that you
184

OpenStack Security Guide

May 1, 2015

current

have a process by which you install and harden operating systems. Thus,
the following items will provide additional guidance on how to ensure
your images are transferred securely into OpenStack. There are a variety of
options for obtaining images. Each has specific steps that help validate the
image's provenance.
The first option is to obtain boot media from a trusted source.
$ mkdir -p /tmp/download_directorycd /tmp/download_directory
$ wget http://mirror.anl.gov/pub/ubuntu-iso/CDs/precise/
ubuntu-12.04.2-server-amd64.iso
$ wget http://mirror.anl.gov/pub/ubuntu-iso/CDs/precise/
SHA256SUMS
$ wget http://mirror.anl.gov/pub/ubuntu-iso/CDs/precise/
SHA256SUMS.gpg
$ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys
0xFBB75451
$ gpg --verify SHA256SUMS.gpg SHA256SUMSsha256sum -c SHA256SUMS
2>&1 | grep OK

The second option is to use the OpenStack Virtual Machine Image Guide
. In this case, you will want to follow your organizations OS hardening
guidelines or those provided by a trusted third-party such as the Linux
STIGs.
The final option is to use an automated image builder. The following example uses the Oz image builder. The OpenStack community has recently
created a newer tool worth investigating: disk-image-builder. We have not
evaluated this tool from a security perspective.
Example of RHEL 6 CCE-26976-1 which will help implement NIST 800-53
SectionAC-19(d) in Oz.
<template>
<name>centos64</name>
<os>
<name>RHEL-6</name>
<version>4</version>
<arch>x86_64</arch>
<install type='iso'>
<iso>http://trusted_local_iso_mirror/isos/x86_64/RHEL-6.4x86_64-bin-DVD1.iso</iso>
</install>
<rootpw>CHANGE THIS TO YOUR ROOT PASSWORD</rootpw>
</os>
<description>RHEL 6.4 x86_64</description>
<repositories>
<repository name='epel-6'>
<url>http://download.fedoraproject.org/pub/epel/6/
$basearch</url>

May 1, 2015

current

2. An encrypted tunnel is created between libvirtd processes on both

source and destination hosts.
3. Destination libvirtd host copies the instances back to an underlying hypervisor.

Monitoring, alerting, and reporting

As an OpenStack virtual machine is a server image able to be replicated
across hosts, best practice in logging applies similarly between physical and
virtual hosts. Operating system-level and application-level events should be
logged, including access events to hosts and data, user additions and removals, changes in privilege, and others as dictated by the environment.
Ideally, you can configure these logs to export to a log aggregator that collects log events, correlates them for analysis, and stores them for reference
or further action. One common tool to do this is an ELK stack, or Elasticsearch, Logstash, and Kibana .
These logs should be reviewed at a regular cadence such as a live view
by a network operations center (NOC), or if the environment is not large
enough to necessitate a NOC, then logs should undergo a regular log review process.
Many times interesting events trigger an alert which is sent to a responder for action. Frequently this alert takes the form of an email with the
messages of interest. An interesting event could be a significant failure,
or known health indicator of a pending failure. Two common utilities for
managing alerts are Nagios and Zabbix .

Updates and patches

A hypervisor runs independent virtual machines. This hypervisor can run
in an operating system or directly on the hardware (called baremetal). Updates to the hypervisor are not propogated down to the virtual machines.
For example, if a deployment is using XenServer and has a set of Debian
virtual machines, an update to XenServer will not update anything running
on the Debian virtual machines.
Therefore, we recommend that clear ownership of virtual machines be assigned, and that those owners be responsible for the hardening, deployment, and continued functionality of the virtual machines. We also recommend that updates be deployed on a regular schedule. These patches should be tested in an environment as closely resembling production
189

OpenStack Security Guide

current

ing process therefore she should continue to define use cases and alerts in
order to have a better understanding of the network traffic activity and
usage over time.

Bob's public cloud

When it comes to logging, as a public cloud provider, Bob is interested in
the activities for situational awareness as well as compliance. In the aspect
of compliance, as a provider, Bob is subject to adherence to various rules
and regulations to include activities such as providing timely, relevant logs
or reports to customers to meet the requirements of their compliance programs. With that in mind, Bob configures all of his instances, nodes, and
infrastructure devices to perform time synchronization with an external,
validated time device. Additionally, Bob's team has built a Django based
web application for his customers to perform self-service log retrieval from
the SIEM tool. Bob also uses this SIEM tool along with a robust set of alerts
and integration with his CMDB to provide operational awareness to both
customers and cloud administrators.

196

OpenStack Security Guide

May 1, 2015

current

19. Compliance
Compliance overview .........................................................................
Understanding the audit process ........................................................
Compliance activities ..........................................................................
Certification and compliance statements ............................................
Privacy ...............................................................................................
Case studies .......................................................................................

197
201
203
206
211
211

An OpenStack deployment may require compliance activities for many purposes, such as regulatory and legal requirements, customer need, privacy
considerations, and security best practices. The Compliance function is important for the business and its customers. Compliance means adhering
to regulations, specifications, standards and laws. It is also used when describing an organizations status regarding assessments, audits, and certifications. Compliance, when done correctly, unifies and strengthens the other security topics discussed in this guide.
This chapter has several objectives:
Review common security principles.
Discuss common control frameworks and certification resources to
achieve industry certifications or regulator attestations.
Act as a reference for auditors when evaluating OpenStack deployments.
Introduce privacy considerations specific to OpenStack and cloud environments.

Compliance overview
Security principles
Industry standard security principles provide a baseline for compliance certifications and attestations. If these principles are considered and referenced throughout an OpenStack deployment, certification activities may
be simplified.
Layered defenses

Identify where risks exist in a cloud architecture

and apply controls to mitigate the risks. In areas
197

OpenStack Security Guide

May 1, 2015

current

of significant concern, layered defences provide

multiple complementary controls to manage
risk down to an acceptable level. For example,
to ensure adequate isolation between cloud
tenants, we recommend hardening QEMU, using a hypervisor with SELinux support, enforcing mandatory access control policies, and reducing the overall attack surface. The foundational principle is to harden an area of concern
with multiple layers of defense such that if any
one layer is compromised, other layers will exist
to offer protection and minimize exposure.
Fail securely

In the case of failure, systems should be configured to fail into a closed secure state. For example, TLS certificate verification should fail
closed by severing the network connection if
the CNAME doesn't match the server's DNS
name. Software often fails open in this situation, allowing the connection to proceed without a CNAME match, which is less secure and
not recommended.

Least privilege

Only the minimum level of access for users and

system services is granted. This access is based
upon role, responsibility and job function. This
security principal of least privilege is written into several international government security
policies, such as NIST 800-53 Section AC-6 within the United States.

Compartmentalize

Systems should be segregated in a such way

that if one machine, or system-level service, is
compromised the security of the other systems
will remain intact. Practically, the enablement
and proper usage of SELinux helps accomplish
this goal.

Promote privacy

The amount of information that can be gathered about a system and its users should be
minimized.

Logging capability

Appropriate logging is implemented to monitor for unauthorized use, incident response

198

OpenStack Security Guide

May 1, 2015

current

and forensics. It is highly recommended that

selected audit subsystems be Common Criteria
certified, which provides non-attestable event
records in most countries.

Common control frameworks

The following is a list of Control Frameworks that an organization can use
to build their security controls
Cloud Security Alliance (CSA)
Common Control Matrix (CCM)

The CSA CCM is specifically designed

to provide fundamental security principles to guide cloud vendors and to
assist prospective cloud customers in
assessing the overall security risk of a
cloud provider. The CSA CCM provides
a controls framework that are aligned
across 16 security domains. The foundation of the Cloud Controls Matrix
rests on its customized relationship to
other industry standards, regulations,
and controls frameworks such as: ISO
27001:2013, COBIT 5.0, PCI:DSS v3, AICPA 2014 Trust Service Principles and Criteria and augments internal control direction for service organization control
reports attestations.
The CSA CCM strengthens existing information security control environments by enabling the reduction of security threats and vulnerabilities in the
cloud, provides standardized security
and operational risk management, and
seeks to normalize security expectations, cloud taxonomy and terminology, and security measures implemented
in the cloud.

ISO 27001/2:2013

The ISO 27001 Information Security

standard and certification has been
used for many years to evaluate and
distinguish an organizations alignment
with information Security best prac199

OpenStack Security Guide

May 1, 2015

current

tices. The standard is comprised of two

parts: Mandatory Clauses that define
the Information Security Management
System (ISMS) and Annex A which contains a list of controls organized by domain.
The information security management
system preserves the confidentiality, integrity, and availability of information
by applying a risk management process
and gives confidence to interested parties that risks are adequately managed.
Trusted Security Principles

Trust Services are a set of professional

attestation and advisory services based
on a core set of principles and criteria
that address the risks and opportunities
of IT-enabled systems and privacy programs. Commonly known as the SOC
audits, the principles define what the
requirement is and it is the organizations responsibility to define the control that meets the requirement.

Audit reference
OpenStack is innovative in many ways however the process used to audit
an OpenStack deployment is fairly common. Auditors will evaluate a process by two criteria: Is the control designed effectively and if the control
is operating effectively. An understanding of how an auditor evaluates if
a control is designed and operating effectively will be discussed in the section called Understanding the audit process [201].
The most common frameworks for auditing and evaluating a cloud deployment include the previously mentioned ISO 27001/2 Information Security standard, ISACA's Control Objectives for Information and Related Technology (COBIT) framework, Committee of Sponsoring Organizations of
the Treadway Commission (COSO), and Information Technology Infrastructure Library (ITIL). It is very common for audits to include areas of focus
from one or more of these frameworks. Fortunately there is a lot of overlap between the frameworks, so an organization that adopts one will be in
a good position come audit time.
200

OpenStack Security Guide

May 1, 2015

current

Understanding the audit process

Information system security compliance is reliant on the completion of two
foundational processes:
1. Implementation and operation of security controls. Aligning the information system with in-scope standards and regulations involves internal
tasks which must be conducted before a formal assessment. Auditors
may be involved at this state to conduct gap analysis, provide guidance,
and increase the likelihood of successful certification.
2. Independent verification and validation. Demonstration to a neutral
third-party that system security controls are implemented and operating
effectively, in compliance with in-scope standards and regulations, is required before many information systems achieve certified status. Many
certifications require periodic audits to ensure continued certification,
considered part of an overarching continuous monitoring practice.

Determining audit scope

how well an organization's ISMS is performing.

Risk assessment
A risk assessment framework identifies risks within an organization or service, and specifies ownership of these risks, along with implementation
and mitigation strategies. Risks apply to all areas of the service, from technical controls to environmental disaster scenarios and human elements, for
example a malicious insider (or rogue employee). Risks can be rated using
a variety of mechanisms, for example likelihood vs impact. An OpenStack
deployment risk assessment can include control gaps that are described in
this book.

Access and log reviews

Periodic access and log reviews are required to ensure authentication, authorization, and accountability in a service deployment. Specific guidance
for OpenStack on these topics are discussed in-depth in the logging section.

Backup and disaster recovery

Disaster Recovery (DR) and Business Continuity Planning (BCP) plans are
common requirements for ISMS and compliance activities. These plans
must be periodically tested as well as documented. In OpenStack key areas
are found in the management security domain, and anywhere that single
points of failure (SPOFs) can be identified. See the section on secure backup and recovery for additional details.

Security training
Annual, role-specific, security training is a mandatory requirement for almost all compliance certifications and attestations. To optimize the effectiveness of security training, a common method is to provide role specific
training, for example to developers, operational personnel, and non-technical employees. Additional cloud security or OpenStack security training
based on this hardening guide would be ideal.

Security reviews
As OpenStack is a popular open source project, much of the codebase and
architecture has been scrutinized by individual contributors, organizations
204

OpenStack Security Guide

May 1, 2015

current

and enterprises. This can be advantageous from a security perspective,

however the need for security reviews is still a critical consideration for service providers, as deployments vary, and security is not always the primary concern for contributors. A comprehensive security review process may
include architectural review, threat modelling, source code analysis and
penetration testing. There are many techniques and recommendations for
conducting security reviews that can be found publicly posted. A well-tested example is the Microsoft SDL, created as part of the Microsoft Trustworthy Computing Initiative.

Vulnerability management
Security updates are critical to any IaaS deployment, whether private or
public. Vulnerable systems expand attack surfaces, and are obvious targets
for attackers. Common scanning technologies and vulnerability notification services can help mitigate this threat. It is important that scans are authenticated and that mitigation strategies extend beyond simple perimeter
hardening. Multi-tenant architectures such as OpenStack are particularly
prone to hypervisor vulnerabilities, making this a critical part of the system
for vulnerability management. See the section on instance isolation for additional details.

Data classification
Data Classification defines a method for classifying and handling information, often to protect customer information from accidental or deliberate
theft, loss, or inappropriate disclosure. Most commonly this involves classifying information as sensitive or non-sensitive, or as personally identifiable information (PII). Depending on the context of the deployment various other classifying criteria may be used (government, health-care etc).
The underlying principle is that data classifications are clearly defined and
in-use. The most common protective mechanisms include industry standard
encryption technologies. See the data security section for additional details.

Exception process
An exception process is an important component of an ISMS. When certain actions are not compliant with security policies that an organization
has defined, they must be logged. Appropriate justification, description
and mitigation details need to be included, and signed off by appropriate
authorities. OpenStack default configurations may vary in meeting various compliance criteria, areas that fail to meet compliance requirements
205

OpenStack Security Guide

May 1, 2015

current

should be logged, with potential fixes considered for contribution to the

community.

Certification and compliance statements

Compliance and security are not exclusive, and must be addressed together. OpenStack deployments are unlikely to satisfy compliance requirements
without security hardening. The listing below provides an OpenStack architect foundational knowledge and guidance to achieve compliance against
commercial and government certifications and standards.

Commercial standards
For commercial deployments of OpenStack, it is recommended that SOC
1/2 combined with ISO 2700 1/2 be considered as a starting point for
OpenStack certification activities. The required security activities mandated
by these certifications facilitate a foundation of security best practices and
common control criteria that can assist in achieving more stringent compliance activities, including government attestations and certifications.
After completing these initial certifications, the remaining certifications
are more deployment specific. For example, clouds processing credit card
transactions will need PCI-DSS, clouds storing health care information require HIPAA, and clouds within the federal government may require FedRAMP/FISMA, and ITAR, certifications.

SOC 1 (SSAE 16) / ISAE 3402

OpenStack Security Guide

May 1, 2015

current

ty around security controls, and the volume of documentation required to

meet government standards.
For more details see http://www.gsa.gov/portal/category/102371.

ITAR
The International Traffic in Arms Regulations (ITAR) is a set of United
States government regulations that control the export and import of defense-related articles and services on the United States Munitions List
(USML) and related technical data. ITAR is often approached by cloud
providers as an "operational alignment" rather than a formal certification.
This typically involves implementing a segregated cloud environment following practices based on the NIST 800-53 framework, as per FISMA requirements, complemented with additional controls restricting access to
"U.S. Persons" only and background screening.
For more details see https://www.pmddtc.state.gov/regulations_laws/
itar.html.

FISMA
The Federal Information Security Management Act requires that government agencies create a comprehensive plan to implement numerous government security standards, and was enacted within the E-Government
Act of 2002. FISMA outlines a process, which utilizing multiple NIST publications, prepares an information system to store and process government
data.
This process is broken apart into three primary categories:
System categorization: The information system will receive a security
category as defined in Federal Information Processing Standards Publication 199 (FIPS 199). These categories reflect the potential impact of system compromise.
Control selection:Based upon system security category as defined in FIPS
199, an organization utilizes FIPS 200 to identify specific security control
requirements for the information system. For example, if a system is categorized as "moderate" a requirement may be introduced to mandate
"secure passwords".
Control tailoring: Once system security controls are identified, an OpenStack architect will utilize NIST 800-53 to extract tailored control selection. For example, specification of what constitutes a "secure password".
210

OpenStack Security Guide

May 1, 2015

current

Privacy
Privacy is an increasingly important element of a compliance program.
Businesses are being held to a higher standard by their customers, who
have increased interest in understanding how their data is treated from a
privacy perspective.
An OpenStack deployment will likely need to demonstrate compliance
with an organization's Privacy Policy, with the U.S.-E.U. Safe Harbor framework, the ISO/IEC 29100:2011 privacy framework or with other privacy-specific guidelines. In the U.S. the AICPA has defined 10 privacy areas of
focus, OpenStack deployments within a commercial environment may desire to attest to some or all of these principles.
To aid OpenStack architects in the protection of personal data, it is recommended that OpenStack architects review the NIST publication 800-122, titled "Guide to Protecting the Confidentiality of Personally Identifiable Information (PII)." This guide steps through the process of protecting:
"any information about an individual maintained by an
agency, including (1) any information that can be used to
distinguish or trace an individual's identity, such as name,
social security number, date and place of birth, mother's
maiden name, or biometric records; and (2) any other information that is linked or linkable to an individual, such as
medical, educational, financial, and employment information"
Comprehensive privacy management requires significant preparation,
thought and investment. Additional complications are introduced when
building global OpenStack clouds, for example navigating the differences
between U.S. and more restrictive E.U. privacy laws. In addition, extra care
needs to be taken when dealing with sensitive PII that may include information such as credit card numbers or medical records. This sensitive data is not only subject to privacy laws but also regulatory and governmental
regulations. By deferring to established best practices, including those published by governments, a holistic privacy management policy may be created and practiced for OpenStack deployments.

Case studies
Earlier in the section called Introduction to case studies [21] we introduced the Alice and Bob case studies where Alice is deploying a private
211

OpenStack Security Guide

May 1, 2015

current

government cloud and Bob is deploying a public cloud each with different
security requirements. Here we discuss how Alice and Bob would address
common compliance requirements. The preceding chapter refers to a wide
variety of compliance certifications and standards. Alice will address compliance in a private cloud, while Bob will be focused on compliance for a
public cloud.

Alice's private cloud

Alice is building an OpenStack private cloud for the United States government, specifically to provide elastic compute environments for signal processing. Alice has researched government compliance requirements, and
has identified that her private cloud will be required to certify against FISMA and follow the FedRAMP accreditation process, which is required for
all federal agencies, departments and contractors to become a Certified
Cloud Provider (CCP). In this particular scenario for signal processing, the
FISMA controls required will most likely be FISMA High, which indicates
possible "severe or catastrophic adverse effects" should the information
system become compromised. In addition to FISMA Moderate controls Alice must ensure her private cloud is FedRAMP certified, as this is a requirement for all agencies that currently utilize, or host federal information
within a cloud environment.
To meet these strict government regulations Alice undertakes a number of
activities. Scoping of requirements is particularly important due to the volume of controls that must be implemented, which will be defined in NIST
Publication 800-53.
All technology within her private cloud must be FIPS certified technology,
as mandated within NIST 800-53 and FedRAMP. As the U.S. Department of
Defense is involved, Security Technical Implementation Guides (STIGs) will
come into play, which are the configuration standards for DOD IA and IAenabled devices / systems. Alice notices a number of complications here as
there is no STIG for OpenStack, so she must address several underlying requirements for each OpenStack service; for example, the networking SRG
and Application SRG will both be applicable (list of SRGs). Other critical
controls include ensuring that all identities in the cloud use PKI, that SELinux is enabled, that encryption exists for all wire-level communications, and
that continuous monitoring is in place and clearly documented. Alice is not
concerned with object encryption, as this will be the tenants responsibility
rather than the provider.
If Alice has adequately scoped and executed these compliance activities,
she may begin the process to become FedRAMP compliant by hiring an
212

OpenStack Security Guide

May 1, 2015

current

approved third-party auditor. Typically this process takes up to 6 months,

after which she will receive an Authority to Operate and can offer OpenStack cloud services to the government.

Bob's public cloud

Bob is tasked with compliance for a new OpenStack public cloud deployment, that is focused on providing cloud services to both small developers
and startups, as well as large enterprises. Bob recognizes that individual
developers are not necessarily concerned with compliance certifications,
but to larger enterprises certifications are critical. Specifically Bob desires
to achieve SOC 1, SOC 2 Security, as well as ISO 27001/2 as quickly as possible. Bob references the Cloud Security Alliance Cloud Control Matrix (CCM)
to assist in identifying common controls across these three certifications
(such as periodic access reviews, auditable logging and monitoring services,
risk assessment activities, security reviews, etc). Bob then engages an experienced audit team to conduct a gap analysis on the public cloud deployment, reviews the results and fills any gaps identified. Bob works with other team members to ensure that these security controls and activities are
regularly conducted for a typical audit period (~6-12 months).
At the end of the audit period Bob has arranged for an external audit
team to review in-scope security controls at randomly sampled points of
time over a 6 month period. The audit team provides Bob with an official
report for SOC 1 and SOC 2, and separately for ISO 27001/2. As Bob has
been diligent in ensuring security controls are in place for his OpenStack
public cloud, there are no additional gaps exposed on the report. Bob can
now provide these official reports to his customers under NDA, and advertise that he is SOC 1, SOC 2 and ISO 27001/2 compliant on his website.

213

OpenStack Security Guide

May 1, 2015

current

AppendixA.Community support
Table of Contents
Documentation ..................................................................................
ask.openstack.org ..............................................................................
OpenStack mailing lists .......................................................................
The OpenStack wiki ...........................................................................
The Launchpad Bugs area ..................................................................
The OpenStack IRC channel ................................................................
Documentation feedback ...................................................................
OpenStack distribution packages ........................................................

215
216
217
217
217
218
219
219

The following resources are available to help you run and use OpenStack.
The OpenStack community constantly improves and adds to the main features of OpenStack, but if you have any questions, do not hesitate to ask.
Use the following resources to get OpenStack support, and troubleshoot
your installations.

Documentation
For the available OpenStack documentation, see docs.openstack.org.
To provide feedback on documentation, join and use the
<[email protected]> mailing list at OpenStack
Documentation Mailing List, or report a bug.
The following books explain how to install an OpenStack cloud and its associated components:
Installation Guide for openSUSE 13.2 and SUSE Linux Enterprise Server
12
Installation Guide for Red Hat Enterprise Linux 7, CentOS 7, and Fedora
21
Installation Guide for Ubuntu 14.04 (LTS)
The following books explain how to configure and run an OpenStack
cloud:
215

OpenStack Security Guide

May 1, 2015

current

Architecture Design Guide

Cloud Administrator Guide
Configuration Reference
Operations Guide
Networking Guide
High Availability Guide
Security Guide
Virtual Machine Image Guide
The following books explain how to use the OpenStack dashboard and
command-line clients:
API Quick Start
End User Guide
Admin User Guide
Command-Line Interface Reference
The following documentation provides reference and guidance information for the OpenStack APIs:
OpenStack API Complete Reference (HTML)
API Complete Reference (PDF)
The Training Guides offer software training for cloud administration and
management.

ask.openstack.org
During the set up or testing of OpenStack, you might have questions
about how a specific task is completed or be in a situation where a feature
does not work correctly. Use the ask.openstack.org site to ask questions
and get answers. When you visit the http://ask.openstack.org site, scan
the recently asked questions to see whether your question has already
been answered. If not, ask a new question. Be sure to give a clear, concise
216

OpenStack Security Guide

May 1, 2015

current

summary in the title and provide as much detail as possible in the description. Paste in your command output or stack traces, links to screen shots,
and any other information which might be useful.

OpenStack mailing lists

A great way to get answers and insights is to post your question or
problematic scenario to the OpenStack mailing list. You can learn from
and help others who might have similar issues. To subscribe or view the
archives, go to http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack. You might be interested in the other mailing lists for specific projects
or development, which you can find on the wiki. A description of all mailing lists is available at http://wiki.openstack.org/MailingLists.

The OpenStack wiki

The OpenStack wiki contains a broad range of topics but some of the information can be difficult to find or is a few pages deep. Fortunately, the
wiki search feature enables you to search by title or content. If you search
for specific information, such as about networking or nova, you can find
a large amount of relevant material. More is being added all the time, so
be sure to check back often. You can find the search box in the upper-right
corner of any OpenStack wiki page.

The Launchpad Bugs area

The OpenStack community values your set up and testing efforts and
wants your feedback. To log a bug, you must sign up for a Launchpad account at https://launchpad.net/+login. You can view existing bugs and
report bugs in the Launchpad Bugs area. Use the search feature to determine whether the bug has already been reported or already been fixed. If
it still seems like your bug is unreported, fill out a bug report.
Some tips:
Give a clear, concise summary.
Provide as much detail as possible in the description. Paste in your command output or stack traces, links to screen shots, and any other information which might be useful.
Be sure to include the software and package versions that
you are using, especially if you are using a development
217

OpenStack Security Guide

May 1, 2015

current

branch, such as, "Juno release" vs git commit

bc79c3ecc55929bac585d04a03475b72e06a3208.
Any deployment-specific information is helpful, such as whether you are
using Ubuntu 14.04 or are performing a multi-node installation.
The following Launchpad Bugs areas are available:
Bugs: OpenStack Block Storage (cinder)
Bugs: OpenStack Compute (nova)
Bugs: OpenStack Dashboard (horizon)
Bugs: OpenStack Identity (keystone)
Bugs: OpenStack Image service (glance)
Bugs: OpenStack Networking (neutron)
Bugs: OpenStack Object Storage (swift)
Bugs: Bare metal service (ironic)
Bugs: Data processing service (sahara)
Bugs: Database service (trove)
Bugs: Orchestration (heat)
Bugs: Telemetry (ceilometer)
Bugs: Message Service (zaqar)
Bugs: OpenStack API Documentation (developer.openstack.org)
Bugs: OpenStack Documentation (docs.openstack.org)

The OpenStack IRC channel

The OpenStack community lives in the #openstack IRC channel on the
Freenode network. You can hang out, ask questions, or get immediate
feedback for urgent and pressing issues. To install an IRC client or use
a browser-based client, go to https://webchat.freenode.net/. You can
also use Colloquy (Mac OS X, http://colloquy.info/), mIRC (Windows,
http://www.mirc.com/), or XChat (Linux). When you are in the IRC chan218

OpenStack Security Guide

May 1, 2015

current

nel and want to share code or command output, the generally accepted
method is to use a Paste Bin. The OpenStack project has one at http://
paste.openstack.org. Just paste your longer amounts of text or logs in
the web form and you get a URL that you can paste into the channel. The
OpenStack IRC channel is #openstack on irc.freenode.net. You can
find a list of all OpenStack IRC channels at https://wiki.openstack.org/wiki/IRC.

Documentation feedback
To provide feedback on documentation, join and use the
<[email protected]> mailing list at OpenStack
Documentation Mailing List, or report a bug.

May 1, 2015

current

Virtual Network Computing (VNC)

Open source GUI and CLI tools used for remote console access to VMs. Supported
by Compute.
ZeroMQ
Message queue software supported by OpenStack. An alternative to RabbitMQ.
Also spelled 0MQ.

225

OpenStack Installation Guide For (RHEL, CentOS, Fedora)
100% (1)
OpenStack Installation Guide For (RHEL, CentOS, Fedora)
140 pages
Ubuntu OpenStack Fundamentals Training
No ratings yet
Ubuntu OpenStack Fundamentals Training
6 pages
Openstack
No ratings yet
Openstack
18 pages
Ovn-Architecture 7 Openvswitch-Manual
No ratings yet
Ovn-Architecture 7 Openvswitch-Manual
16 pages
Learning OpenStack Networking (Neutron) Sample Chapter
No ratings yet
Learning OpenStack Networking (Neutron) Sample Chapter
18 pages
2 Day Bootcamp For OpenStack - Mirantis
No ratings yet
2 Day Bootcamp For OpenStack - Mirantis
66 pages
Rally
100% (1)
Rally
305 pages
Openstack Centos KVM Glusterfs Guide
No ratings yet
Openstack Centos KVM Glusterfs Guide
49 pages
OpenStack NVM 2021
No ratings yet
OpenStack NVM 2021
120 pages
Openstack Comp 1
No ratings yet
Openstack Comp 1
28 pages
Production Ready OpenStack - Recipes For Successful Environments - Sample Chapter
No ratings yet
Production Ready OpenStack - Recipes For Successful Environments - Sample Chapter
52 pages
RedHat Certified Engineer Notes
No ratings yet
RedHat Certified Engineer Notes
20 pages
Install Your Own OpenStack Cloud
100% (1)
Install Your Own OpenStack Cloud
11 pages
Backup and Restore Logical Volume Using LVM Snapshot
No ratings yet
Backup and Restore Logical Volume Using LVM Snapshot
25 pages
Install Ntopng Network Traffic Monitoring Tool On CentOS 7
100% (1)
Install Ntopng Network Traffic Monitoring Tool On CentOS 7
6 pages
TXLF2011 OpenStack
No ratings yet
TXLF2011 OpenStack
83 pages
Same Same, But Better: Comparing Artifactory To Other Binary Repository Managers
No ratings yet
Same Same, But Better: Comparing Artifactory To Other Binary Repository Managers
20 pages
Red Hat Ceph Storage Hardware Selection Guide
No ratings yet
Red Hat Ceph Storage Hardware Selection Guide
23 pages
Red Hat Virtualization-4.4-Installing Red Hat Virtualization As A Self-Hosted Engine Using The Cockpit Web interface-en-US
No ratings yet
Red Hat Virtualization-4.4-Installing Red Hat Virtualization As A Self-Hosted Engine Using The Cockpit Web interface-en-US
82 pages
Kolla Ansible
No ratings yet
Kolla Ansible
210 pages
OVA and OVF
No ratings yet
OVA and OVF
3 pages
Ebook OpenStack Made Easy 20160726 PDF
No ratings yet
Ebook OpenStack Made Easy 20160726 PDF
28 pages
Complete Guide To Setting Up An OpenStack Development Environment
No ratings yet
Complete Guide To Setting Up An OpenStack Development Environment
4 pages
Troubleshooting Open Stack
No ratings yet
Troubleshooting Open Stack
20 pages
Terraform
No ratings yet
Terraform
6 pages
What Is Cloud Computing
No ratings yet
What Is Cloud Computing
21 pages
OpenStack Networking Cookbook - Sample Chapter
No ratings yet
OpenStack Networking Cookbook - Sample Chapter
28 pages
Installing OpenStack On Ubuntu 12
No ratings yet
Installing OpenStack On Ubuntu 12
13 pages
Complete Download (Ebook PDF) Django For APIs: Build Web APIs With Python and Django PDF All Chapters
100% (5)
Complete Download (Ebook PDF) Django For APIs: Build Web APIs With Python and Django PDF All Chapters
51 pages
Linux Network Namespace Introduction - Docker Kubernetes Lab 0.1
No ratings yet
Linux Network Namespace Introduction - Docker Kubernetes Lab 0.1
10 pages
Veritas Cluster Commands
No ratings yet
Veritas Cluster Commands
8 pages
Linux Lab Manual
No ratings yet
Linux Lab Manual
4 pages
Veritas Clustering On Linux
No ratings yet
Veritas Clustering On Linux
714 pages
Introduction To SELinux
No ratings yet
Introduction To SELinux
11 pages
Red Hat Openstack Platform-17.1-Installing and Managing Red Hat Openstack Platform With Director-En-Us222
No ratings yet
Red Hat Openstack Platform-17.1-Installing and Managing Red Hat Openstack Platform With Director-En-Us222
192 pages
Red Hat Satellite 6.2 ArchitectureGuide
100% (1)
Red Hat Satellite 6.2 ArchitectureGuide
35 pages
Artifactory-Configuring With Jenkins
No ratings yet
Artifactory-Configuring With Jenkins
9 pages
How To Set Up Private DNS Servers With BIND On CentOS 8
No ratings yet
How To Set Up Private DNS Servers With BIND On CentOS 8
5 pages
BP 2105 Linux On AHV
No ratings yet
BP 2105 Linux On AHV
34 pages
Fuel For OpenStack 3.1 UserGuide
100% (1)
Fuel For OpenStack 3.1 UserGuide
126 pages
Openstack Manual
No ratings yet
Openstack Manual
111 pages
Install FreeIPA Server On CentOS 8 - CentLinux
No ratings yet
Install FreeIPA Server On CentOS 8 - CentLinux
16 pages
Red Hat Gluster Storage 3.3 Administration Guide en US
No ratings yet
Red Hat Gluster Storage 3.3 Administration Guide en US
517 pages
Cloud Openstack
No ratings yet
Cloud Openstack
5 pages
OpenShift Online 2.0 User Guide en US
100% (1)
OpenShift Online 2.0 User Guide en US
80 pages
Red Hat OpenStack Platform-11-High Availability For Compute Instances-En-US
No ratings yet
Red Hat OpenStack Platform-11-High Availability For Compute Instances-En-US
14 pages
OpenStack Installation Checklist 1.1
No ratings yet
OpenStack Installation Checklist 1.1
85 pages
Red Hat System Administration I 3.9 Lab PDF
No ratings yet
Red Hat System Administration I 3.9 Lab PDF
11 pages
LDAP Directories Explained
No ratings yet
LDAP Directories Explained
291 pages
Red Hat Enterprise Linux-9-Building Running and Managing Containers-En-Us
No ratings yet
Red Hat Enterprise Linux-9-Building Running and Managing Containers-En-Us
163 pages
Openstack Technical 101
100% (2)
Openstack Technical 101
31 pages
Mastering Openstack: Controller Nodes
No ratings yet
Mastering Openstack: Controller Nodes
21 pages
Red Hat OpenStack Platform-9-Director Installation and Usage-En-US
No ratings yet
Red Hat OpenStack Platform-9-Director Installation and Usage-En-US
190 pages
Ceph Reference Architecture
100% (1)
Ceph Reference Architecture
12 pages
Getting Started With Satellite 6
No ratings yet
Getting Started With Satellite 6
54 pages
OpenStack Networking Essentials
From Everand
OpenStack Networking Essentials
Denton James
No ratings yet
DRBD-Cookbook: How to create your own cluster solution, without SAN or NAS!
From Everand
DRBD-Cookbook: How to create your own cluster solution, without SAN or NAS!
Joerg Christian Seubert
No ratings yet
OpenStack Networking Cookbook
From Everand
OpenStack Networking Cookbook
Subramanian Sriram
No ratings yet
Unix / Linux FAQ: with Tips to Face Interviews
From Everand
Unix / Linux FAQ: with Tips to Face Interviews
Prof. N.B. Venkateswarlu
No ratings yet
Learning SaltStack - Second Edition
From Everand
Learning SaltStack - Second Edition
Colton Myers
No ratings yet
Zealand Between Inthe AND Newzealandmadein 21st
No ratings yet
Zealand Between Inthe AND Newzealandmadein 21st
4 pages
Client-Side Web Security PDF
No ratings yet
Client-Side Web Security PDF
208 pages
Rest Security by Example Frank Kim
No ratings yet
Rest Security by Example Frank Kim
62 pages
IS5111 Project Proposal
No ratings yet
IS5111 Project Proposal
1 page
Fntdb07 Architecture of A Database System
100% (4)
Fntdb07 Architecture of A Database System
119 pages
APOSTOLIC LETTER Issued Motu Proprio
No ratings yet
APOSTOLIC LETTER Issued Motu Proprio
3 pages
Presentation of Evidence - Video Conferencing PDF
No ratings yet
Presentation of Evidence - Video Conferencing PDF
8 pages
1.1 Blotter Report
No ratings yet
1.1 Blotter Report
38 pages
Case Bond Request
83% (6)
Case Bond Request
3 pages
Military Law
No ratings yet
Military Law
43 pages
Pertinent Provisions of PD 603
No ratings yet
Pertinent Provisions of PD 603
14 pages
Instant Download Criminal Dismemberment: Forensic and Investigative Analysis 1st Edition Sue Black PDF All Chapter
100% (6)
Instant Download Criminal Dismemberment: Forensic and Investigative Analysis 1st Edition Sue Black PDF All Chapter
52 pages
Convictions Vacated in Case of Richmond Heights Man Accused of Trying To Kill His Wife
No ratings yet
Convictions Vacated in Case of Richmond Heights Man Accused of Trying To Kill His Wife
15 pages
Mock Trial Notes 1.2
No ratings yet
Mock Trial Notes 1.2
3 pages
VTMB Transcripts
No ratings yet
VTMB Transcripts
21 pages
The Black Tulip, by Alexandre Dumas
No ratings yet
The Black Tulip, by Alexandre Dumas
138 pages
Dimat vs. People
No ratings yet
Dimat vs. People
3 pages
Struggle For Memory Delano Nienass
No ratings yet
Struggle For Memory Delano Nienass
7 pages
A.5 CLJ 2022 no answer
No ratings yet
A.5 CLJ 2022 no answer
16 pages
Police Log May 29, 2016
No ratings yet
Police Log May 29, 2016
12 pages
"Lot A Is Nothing More Than A Tidal Pool Area ... " The Doctor 2001 On Scribd JNP
No ratings yet
"Lot A Is Nothing More Than A Tidal Pool Area ... " The Doctor 2001 On Scribd JNP
24 pages
Rape Essay
No ratings yet
Rape Essay
1 page
International Law Int Class
50% (2)
International Law Int Class
194 pages
Key Macbeth Quotes - Scene by Scene
No ratings yet
Key Macbeth Quotes - Scene by Scene
4 pages
Cases After Ra 9262
100% (1)
Cases After Ra 9262
6 pages
RPD 6-18-24
No ratings yet
RPD 6-18-24
5 pages
The Madonnas of Echo Park Character Chart
No ratings yet
The Madonnas of Echo Park Character Chart
2 pages
Gordon Shiraishi Complaint
No ratings yet
Gordon Shiraishi Complaint
14 pages
Moot Problem 1 1 PDF
No ratings yet
Moot Problem 1 1 PDF
5 pages
Syria SST Hmun
No ratings yet
Syria SST Hmun
2 pages
Sex Crimes in California
No ratings yet
Sex Crimes in California
2 pages
(HC) Ekwueme v. Chertoff Et Al - Document No. 6
No ratings yet
(HC) Ekwueme v. Chertoff Et Al - Document No. 6
2 pages
STATE OF THE COUNTERCULTURE UNION ADDRESS 2013 by Ryan Bartek
100% (1)
STATE OF THE COUNTERCULTURE UNION ADDRESS 2013 by Ryan Bartek
23 pages
Potsdam Village Police Dept. Blotter Nov. 10, 2018
No ratings yet
Potsdam Village Police Dept. Blotter Nov. 10, 2018
4 pages
An Outlook On Sexual Violence Case Handling in Indonesia 18 May 2022
No ratings yet
An Outlook On Sexual Violence Case Handling in Indonesia 18 May 2022
159 pages