How to configure or build ceph storage cluster

in Openstack ( CentOS 7 )
Table of Contents

o Monitors:
o Managers:
o Ceph OSDs:
• My infrastructure detail.
• About ceph-deploy tool
• Install pre-requisite rpms
• Configure NTP
• Create ceph user
• Enable Passwordless SSH
• Validate connectivity
• Open firewall ports
o Handle Selinux
• Steps to build ceph storage cluster
o Create the cluster:
o Install Ceph packages from your admin node:
o Add OSDs:
o OSD dump
o CRUSH dump
o OSD list
o Get the quorum status

In my last article I shared the steps to configure controller node in OpenStack manually, now in this
article I will share the steps to configure and build ceph storage cluster using CentOS 7. Ceph is an
open source, scalable, and software-defined object store system, which provides object, block, and
file system storage in a single platform. Ceph has a capability to self-heal, self-manage, and does not
have a single point of failure. It is a perfect replacement for a traditional storage system and an
efficient storage solution for the object and block storage of cloud environments.
 Before we start with the steps to build ceph storage cluster, let us understand some basic
terminologies

Monitors:

A Ceph Monitor (ceph-mon) maintains maps of the cluster state, including the monitor map,
manager map, the OSD map, and the CRUSH map. These maps are critical cluster state required for
Ceph daemons to coordinate with each other. Monitors are also responsible for managing
authentication between daemons and clients. At least three monitors are normally required for
redundancy and high availability.

Managers:

A Ceph Manager daemon (ceph-mgr) is responsible for keeping track of runtime metrics and the
current state of the Ceph cluster, including storage utilization, current performance metrics, and
system load. The Ceph Manager daemons also host python-based plugins to manage and expose
Ceph cluster information, including a web-based dashboard and REST API. At least two managers
are normally required for high availability.

Ceph OSDs:

A Ceph OSD (object storage daemon, ceph-osd) stores data, handles data replication, recovery,
rebalancing, and provides some monitoring information to Ceph Monitors and Managers by
checking other Ceph OSD Daemons for a heartbeat. At least 3 Ceph OSDs are normally required for
redundancy and high availability.

My infrastructure detail.
We will build ceph storage cluster with two nodes ceph storage with one OSD per storage node and
one admin node where we will perform most of our tasks. So in total we have three virtual machines
running on Oracle Virtual Box on top of my Windows laptop.

Below are the configuration I have used for my storage and admin nodes

admin storage1 storage2

OS CentOS 7 CentOS 7 CentOS 7
Disk 1 10 GB 10 GB 10 GB
Disk 2 4 GB 4 GB 4 GB
RAM 4 GB 4 GB 4 GB
vCPU 2 2 2
Network 10.0.2.10 10.0.2.13 10.0.2.14
Hostname controller storage1 storage2
About ceph-deploy tool
ceph-deploy is the official tool to deploy Ceph clusters. It works on the principle of having an
admin node with SSH access (without password) to all machines in your Ceph cluster; it also holds a
copy of the Ceph configuration file. Every time you carry out a deployment action, it uses SSH to
connect to your Ceph nodes to carry out the necessary steps. Although the ceph-deploy tool is an
entirely supported method, which will leave you with a perfectly functioning Ceph cluster, ongoing
management of Ceph will not be as easy as desired. Larger scale Ceph clusters will also cause a lot
of management overheads if ceph-deploy is to be used. For this reason, it is recommended that ceph-
deploy is limited to test or small-scale production clusters, although as you will see, an orchestration
tool allows the rapid deployment of Ceph and is probably better suited for test environments where
you might need to continually build new Ceph clusters.

Install pre-requisite rpms

To get the required rpms to build ceph storage cluster we need to install epel repo and enable ceph
repo
Install the latest available epel repo on your admin node

# yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-

7.noarch.rpm

Add the Ceph repository to your yum configuration file at /etc/yum.repos.d/ceph.repo with the
following command. Replace {ceph-stable-release} with a stable Ceph release (e.g., mimic.)

# cat /etc/yum.repos.d/ceph.repo
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-mimic/el7/noarch/
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

Next install ceph-deploy:

# yum -y install ceph-deploy

Configure NTP
To build ceph storage cluster it is very important that all the nodes part of the cluster are in time sync
with each other. So NTP is very important. Install the ntp package and configure your ntp.conf with
your nearest server

# yum -y install ntp

My server pool list are as below which I have added in my ntp.conf

server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
Next start and enable your ntp daemon on both the storage node

# systemctl start ntpd

# systemctl enable ntpd

Create ceph user

The ceph-deploy utility must login to a Ceph node as a user that has passwordless sudo privileges,
because it needs to install software and configuration files without prompting for passwords.

We will create a new user ceph on both the storage nodes

# useradd ceph
# echo redhat | passwd --stdin ceph

Give sudo permission to the “ceph” user

# echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph

# chmod 0440 /etc/sudoers.d/ceph

Enable Passwordless SSH

Since ceph-deploy will not prompt for a password, you must generate SSH keys on the admin node
and distribute the public key to each Ceph node. ceph-deploy will attempt to generate the SSH keys
for initial monitors.

Generate ssh-keys on the admin node

# ssh-keygen -t rsa

Next copy the public key to the target storage nodes

# ssh-copy-id ceph@storage1
# ssh-copy-id ceph@storage2

Modify the ~/.ssh/config file of your ceph-deploy admin node so that ceph-deploy can log in to
Ceph nodes as the user you created without requiring you to specify --username {username} each
time you execute ceph-deploy. This has the added benefit of streamlining ssh and scp usage.

# cat ~/.ssh/config
Host storage1
Hostname storage1
User ceph
Host storage2
Hostname storage2
User ceph
Validate connectivity
Now since our password less is setup it is time to validate the connectivity

# ssh ceph@storage1
Last login: Sat Nov 17 12:48:38 2018 from 10.0.2.10
[ceph@storage1 ~]$ logout
Connection to storage1 closed.

# ssh ceph@storage2
Last login: Sat Nov 17 12:30:31 2018 from 10.0.2.13
[ceph@storage2 ~]$ logout
Connection to storage2 closed.

So all looks good.

Open firewall ports

During your POC stage you can also stop and disable the firewall to make sure the config works. But
before going to production the firewall must be enabled again

On monitors

# firewall-cmd --zone=public --add-service=ceph-mon --permanent

On OSDs

# firewall-cmd --zone=public --add-service=ceph --permanent

Once you have finished configuring firewalld with the --permanent flag, you can make the changes
live immediately without rebooting:

# firewall-cmd --reload

Handle Selinux

On CentOS and RHEL, SELinux is set to Enforcing by default. To streamline your installation, we
recommend setting SELinux to Permissive or disabling it entirely and ensuring that your installation
and cluster are working properly before hardening your configuration. To set SELinux to Permissive,
execute the following:

# setenforce 0

Ensure that your package manager has priority/preferences packages installed and enabled. On
CentOS, you may need to install EPEL. On RHEL, you may need to enable optional repositories.

# yum install yum-plugin-priorities

Create a directory on your admin node for maintaining the configuration files and keys that ceph-
deploy generates for your cluster.
# mkdir my-cluster

and navigate inside this directory as the ceph-deploy tool will create all the required configuration
file in your current working directory

# cd my-cluster

Steps to build ceph storage cluster

The ceph-deploy tool operates out of a directory on an admin node.

On your admin node from the directory you created for holding your configuration details, perform
the following steps using ceph-deploy.

Create the cluster:

# ceph-deploy new storage1 storage2

Check the output of ceph-deploy with ls and cat in the current directory. You should see a Ceph
configuration file (ceph.conf), a monitor secret keyring (ceph.mon.keyring), and a log file for the
new cluster.

If you have more than one network interface, add the public network setting under the [global]
section of your Ceph configuration file. See the Network Configuration Reference for details.

public network = 10.1.2.0/24

to use IPs in the 10.1.2.0/24 (or 10.1.2.0/255.255.255.0) network

Install Ceph packages from your admin node:

# ceph-deploy install storage1 storage2
<< Output trimmed >>
[storage2][DEBUG ]
[storage2][DEBUG ] Complete!
[storage2][INFO ] Running command: sudo ceph --version
[storage2][DEBUG ] ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126)
mimic (stable)
NOTE:
The ceph-deploy utility will install Ceph on each node.

Check the ceph version from your storage nodes

[root@storage1 ~]# ceph --version

ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)

Deploy the initial monitor(s) and gather the keys:

# ceph-deploy mon create-initial

<< Output trimmed >>
[storage1][DEBUG ] fetch remote file
[storage1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --
cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.storage1.asok mon_status
[storage1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --
cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-storage1/keyring auth
get client.admin
[storage1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --
cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-storage1/keyring auth
get client.bootstrap-mds
[storage1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --
cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-storage1/keyring auth
get client.bootstrap-mgr
[storage1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --
cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-storage1/keyring auth
get client.bootstrap-osd
[storage1][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --
cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-storage1/keyring auth
get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmpYahdve

Once you complete the process, your local directory should have the following keyrings:

ceph.client.admin.keyring
ceph.bootstrap-mgr.keyring
ceph.bootstrap-osd.keyring
ceph.bootstrap-mds.keyring
ceph.bootstrap-rgw.keyring
ceph.bootstrap-rbd.keyring

Use ceph-deploy to copy the configuration file and admin key to your admin node and your Ceph
Nodes so that you can use the ceph CLI without having to specify the monitor address and
ceph.client.admin.keyring each time you execute a command.

# ceph-deploy admin storage1 storage2

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy admin storage1
storage2
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf :
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : ['storage1',
'storage2']
[ceph_deploy.cli][INFO ] func :
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to storage1
[storage1][DEBUG ] connection detected need for sudo
[storage1][DEBUG ] connected to host: storage1
[storage1][DEBUG ] detect platform information from remote host
[storage1][DEBUG ] detect machine type
[storage1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to storage2
[storage2][DEBUG ] connection detected need for sudo
[storage2][DEBUG ] connected to host: storage2
[storage2][DEBUG ] detect platform information from remote host
[storage2][DEBUG ] detect machine type
[storage2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

Deploy a manager daemon. (Required only for luminous+ builds):

# ceph-deploy mgr create storage1

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy mgr create
storage1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] mgr : [('storage1',
'storage1')]
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf :
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func :
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts storage1:storage1
[storage1][DEBUG ] connection detected need for sudo
[storage1][DEBUG ] connected to host: storage1
[storage1][DEBUG ] detect platform information from remote host
[storage1][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to storage1
[storage1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[storage1][WARNIN] mgr keyring does not exist yet, creating one
[storage1][DEBUG ] create a keyring file
[storage1][DEBUG ] create path recursively if it doesn't exist
[storage1][INFO ] Running command: sudo ceph --cluster ceph --name
client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-
or-creat
e mgr.storage1 mon allow profile mgr osd allow * mds allow * -o
/var/lib/ceph/mgr/ceph-storage1/keyring
[storage1][INFO ] Running command: sudo systemctl enable ceph-mgr@storage1
[storage1][WARNIN] Created symlink from /etc/systemd/system/ceph-
mgr.target.wants/[email protected] to /usr/lib/systemd/system/ceph-
[email protected].
[storage1][INFO ] Running command: sudo systemctl start ceph-mgr@storage1
[storage1][INFO ] Running command: sudo systemctl enable ceph.target

Add OSDs:

NOTE:
For the purposes of these instructions, we assume you have an unused disk in each node called
/dev/sdc. Be sure that the device is not currently in use and does not contain any important data.
# ceph-deploy osd create --data /dev/sdc storage1
<< Output trimmed >>
[storage1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --
format=json
[ceph_deploy.osd][DEBUG ] Host storage1 is now ready for osd use.
# ceph-deploy osd create --data /dev/sdc storage2
<< Output trimmed >>

[storage2][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --

format=json
[ceph_deploy.osd][DEBUG ] Host storage2 is now ready for osd use.

So we are all done now here.

Now we brought up a functioning Ceph cluster from scratch and incrementally scaled it without
taking it down. Next we’ll explore a number of operations we can perform to customise, populate,
and manage Ceph clusters.

To check the cluster health

[root@storage2 ~]# ceph health

HEALTH_OK

Check the OSD and cluster status

[root@storage1 ~]# ceph -s

cluster:
id: 454f796c-ed9f-4242-89b4-e0d43f740ffd
health: HEALTH_OK

services:
mon: 2 daemons, quorum storage1,storage2
mgr: storage1(active)
osd: 2 osds: 2 up, 2 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 2.0 GiB used, 6.0 GiB / 8.0 GiB avail
pgs:

OSD dump

The ceph osd dump command shows a wealth of lower-level information about our clusters. This
includes a list of pools with their attributes and a list of OSDs each including reweight adjustment,
up/in status, and more. This command is mostly used in unusual troubleshooting situations.

[root@storage2 ~]# ceph osd dump

epoch 15
fsid 454f796c-ed9f-4242-89b4-e0d43f740ffd
created 2018-11-17 13:19:39.387450
modified 2018-11-17 16:14:46.813272
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 6
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release mimic
max_osd 3
osd.0 up in weight 1 up_from 11 up_thru 0 down_at 10 last_clean_interval [5,9)
10.0.2.13:6800/1236 10.0.2.13:6801/1236 10.
osd.1 up in weight 1 up_from 13 up_thru 0 down_at 12 last_clean_interval [0,0)
10.0.2.14:6800/1079 10.0.2.14:6801/1079 10.

CRUSH dump

This command presents much the same information as ceph osd tree, though in a different, JSON,
format.

# ceph osd crush dump

OSD list

The ceph osd ls command simply returns a list of the OSD numbers currently deployed within the
cluster.

[root@storage2 ~]# ceph osd ls

0
1

Get the quorum status

[root@storage2 ~]# ceph quorum_status --format json-pretty

{
"election_epoch": 8,
"quorum": [
0,
1
],
"quorum_names": [
"storage1",
"storage2"
],
"quorum_leader_name": "storage1",
"monmap": {
"epoch": 1,
"fsid": "454f796c-ed9f-4242-89b4-e0d43f740ffd",
"modified": "2018-11-17 13:17:57.395486",
"created": "2018-11-17 13:17:57.395486",
"features": {
"persistent": [
"kraken",
"luminous",
 "mimic",
"osdmap-prune"
],
"optional": []
},
"mons": [
{
"rank": 0,
"name": "storage1",
"addr": "10.0.2.13:6789/0",
"public_addr": "10.0.2.13:6789/0"
},
{
"rank": 1,
"name": "storage2",
"addr": "10.0.2.14:6789/0",
"public_addr": "10.0.2.14:6789/0"
}
]
}
}

Reference:

Official Ceph Page

Lastly I hope the steps from the article to build ceph storage cluster in Openstack using CentOS 7
Linux was helpful. So, let me know your suggestions and feedback using the comment section
Cinder vs Swift storage in OpenStack – Basic
Difference and Comparison
Earlier I had shared an article with the steps to configure ceph storage cluster in OpenStack. Now
let me give you some brief overview on comparison and difference between cinder vs swift storage
in OpenStack.

Let me give some brief overview and examples to explain you the difference between Cinder vs
Swift Storage.

Comparison and Difference – Cinder vs Swift Storage

Cinder (Block Storage)

• Cinder is a block storage device.

• Let’s suppose it has been a year since Mr. Neo started using his workstation. His daily task
saves loads and loads of data onto his hard disk. Mr. Neo noticed that he would run out of his
500 GB internal hard drive space in the next few days. So, he decided to demand a new
external hard drive (portable HDD) from the storage team and raised the new request for an
external hard drive of 1 TB in size.
• When the storage team received the request, it will verify the token ID of Mr. Neo with the
access management team to confirm that the user is authorized to receive an additional
storage.
• Once the authorization is verified, the storage team will provide him with an external 1 TB
hard disk.
• We all know the advantage of using the portable hard drive. Mr. Neo has the flexibility to
connect and mount the external HDD permanently and use it as a secondary internal hard
drive for his workstation or optionally, he could also mount the external HDD temporarily to
backup all necessary files and detach the external HDD from his workstation:

• Similarly, the cinder service will do the storage team’s job in OpenStack. The Cinder block
storage service will take care of providing the additional disk volume to the user. Once the
new disk volume is allocated to the user tenant (project), the user has the flexibility to map
(attach) the volume storage to any VMs between the same project. A Cinder volume is very
similar to an external HDD; you cannot attach the single cinder volume to two or more
virtual machines at the same time, however we could connect the single cinder volume to
any virtual machine, one at a time.
• The logical architecture diagram shows how various processes and daemons work together
to perform the block storage service (Cinder) in OpenStack, and the interconnection between
them:
Swift (Object storage)
• Swift is a object storage device.
• Object storage is very similar to cloud storage, such as Google Drive or Dropbox. As an
end user, we can only experience the end user window of Google Drive or Dropbox for
storing and retrieving the files. However, the actions carried out by the group of processes
and daemons behind the end user screen to save and recover the file has very complex object
storage architecture.
• The Swift service is quite different from other OpenStack services because we can configure
Swift services as standalone services to provide only the object storage services to the end
users, without setting IAAS features.
• For OpenStack, the Swift service is rated under the additional services, not the core one
because the primary purpose of OpenStack is to provide Infrastructure-as-a-Service
(IAAS) to end users. However, Swift is not a mandatory service to bring up the IAAS
feature.
• As an additional service, the Swift service can be configured with other OpenStack services
like Glance and Cinder for storing/retrieving the Glance images and to back up the cinder-
volumes respectively.
 • The following logical architecture diagram shows how various processes and daemons
work together to bring up the object storage service (Swift) in OpenStack, and the
interconnection between the services:

Lastly I hope the steps from the article to understand the difference and comparison between Cinder
vs Swift storage in OpenStack was helpful. So, let me know your suggestions and feedback using the
comment section.
Understanding nova compute architecture
basics in Openstack (flow chart)
Table of Contents

• Understanding nova compute architecture (Flow Chart)

o Understanding nova compute services
o nova-api service
o nova-api-metadata service
o nova-compute service
o nova-scheduler service
o nova-conductor module
o nova-consoleauth daemon
o nova-cert module
o The queue (AMQP message broker)
o Database

Earlier I had shared an article with the steps to configure ceph storage cluster in OpenStack. Now
let me give you some brief overview on the nova compute architecture and how all the services
within nova compute communicate with each other. What is the role of each service under nova
compute? What is the role of nova-api, nova-scheduler, nova-console, nova-conductor, nova-
console-auth, nova-compute service in OpenStack?

Understanding nova compute architecture (Flow Chart)

The compute service consists of various services and daemons including the nova-api service,
which accepts the API requests and forwards them to the other components of the service.
The following diagram shows how various processes and daemons work together to form the
compute service (Nova) and interlink between them:

Understanding nova compute services

Below are the list of services which are part of nova compute service:

nova-api service

• All services in OpenStack have at least one API process that accepts and responds to end user
API calls, pre-processes them, and passes the request to the appropriate process within the
service.
• For the OpenStack compute service, we have the nova-api service, which listens and
responds to the end user compute API calls. The nova-api service takes care of initiating
most orchestration activities, such as provisioning new virtual machines.

nova-api-metadata service
 • The metadata service delivers the instance-specific data to the virtual machine instances. The
instance-specific data includes hostname, instance-id, ssh-keys, and so on. The virtual
machine accesses the metadata service via the special IP address at
http://169.254.169.254.

nova-compute service

• Underneath, the entire lifecycle of the virtual machine is managed by the hypervisors.
Whenever the end user submits the instance creation API call to the nova-api service, the
nova-api service processes it and passes the request to the nova-compute service. The nova-
compute service processes the nova-api call for new instance creation and triggers the
appropriate API request for virtual machine creation in a way that the underlying hypervisor
can understand.
• For example, if we choose to use the KVM hypervisor in the OpenStack setup, when the end
user submits the virtual machine creation request via the OpenStack dashboard, the nova-api
calls will get sent to the nova-api service. The nova-api service will pass the APIs for
instance creation to the nova-compute service. The nova-compute service knows what API
the underlying KVM hypervisor will support. Now, pointing to the underlying KVM
hypervisor, the nova-compute will trigger the libvirt-api for virtual machine creation.
Then, the KVM hypervisor processes the libvirt-api request and creates a new virtual
machine.
• OpenStack has the flexibility to use multi-hypervisor environments in the same setup, that is,
we could configure different hypervisors like KVM and VMware in the same OpenStack
setup. The nova-compute service will take care of triggering the suitable APIs for the
hypervisors to manage the virtual machine lifecycle.

nova-scheduler service

• When we have more than one compute node in your OpenStack environment, the nova-
scheduler service will take care of determining where the new virtual machine will
provision. Based on the various resource filters, such as RAM/CPU/Disk/Availability Zone,
the nova-scheduler will filter the suitable compute host for the new instance:
nova-conductor module

• The nova-compute service running on the compute host has no direct access to the database
because if one of your compute nodes is compromised, then the attacker has (almost) full
access to the database. With the nova-conductor daemon, the compromised node cannot
access the database directly, and all the communication can only go through the nova-
conductor daemon. So, the compromised node is now limited to the extent that the conductor
APIs allow it.
• The nova-conductor module should not be deployed on any compute nodes, or else the
purpose of removing direct database access for the nova-compute will become invalid.

nova-consoleauth daemon

The nova-consoleauth daemon takes care of authorizing the tokens for the end users, to access a
remote console of the guest virtual machines provided by the following control proxies:

• The nova-novncproxy daemon provides a proxy for accessing running instances through a
VNC connection.
• The nova-spicehtml5proxy daemon provides a proxy through a SPICE connection

nova-cert module

• Used to generate X509 certificates for euca-bundle-image, and only needed for the EC2 API.
The queue (AMQP message broker)

• Usually, the AMQP message queue is implemented with RabbitMQ or ZeroMQ. In

OpenStack, an AMQP message broker is used for all communication between the processes
and daemons of one service. However, the communication between the two different services
in OpenStack uses service endpoints.
• For example, the nova-api and nova-scheduler will communicate through the AMQP
message broker. However, the communication between the nova-api service and cinder-
api service will carry through the service endpoints.

Database

• Most of the OpenStack services use an SQL database to store the build-time, and run-time
states for a cloud infrastructure, such as instance status, networks, projects, and the list goes
on. In short, we could say the database is the brain of OpenStack.
• The most tested and preferable databases to use in OpenStack are MySQL, MariaDB, and
PostgreSQL.

Lastly I hope this article on understanding nova compute architecture on OpenStack was helpful. So,
let me know your suggestions and feedback using the comment section.
OpenStack Command Line Cheat Sheet
(Beginner’s Guide)
Table of Contents

• Identity (Keystone)
• Flavor
• Security Group
• Keypair
• Glance Image
• Neutron (Network)
• Server (Instance)
• Block Storage
• Snapshot
• Guestfish

In this article I will share some of the basic OpenStack command line cheat sheet which can be
useful for beginners starting with OpenStack.

Identity (Keystone)
To list the installed users

openstack user list

Show provided user’s properties

openstack user show <username>

Create a user Willian with password redhat, email as [email protected] and is part of project
Production

openstack user create --project Production --email [email protected] --password

redhat --enable William

Assign admin role to user William

openstack role add --user William --project Production admin

Check the assigned role to user William

openstack role assignment list --user William --project Production

openstack role assignment list --user William --project Production --names

Enable or disable user William

openstack user set --disable William

openstack user set --enable William

Flavor
Create flavor named m1.petite with 1 vcpu, 1 GB RAM, 10 GB Disk and must not be publicly
accessible

openstack flavor create --id auto --vcpus 1 --ram 1024 --disk 10 --private
m1.petite

Assign flavor m1.petite to Engineering project

openstack flavor set --project Engineering m1.petite

Security Group
Create security group with name ssh

openstack security group create ssh

Add a rule to allow ssh and icmp in the ssh security group

openstack security group rule create --ingress --protocol tcp --dst-port 22 ssh

openstack security group rule create --ingress --protocol tcp --protocol icmp ssh

Keypair
Create a keypair with name webkey in your home folder

openstack keypair create webkey > ~/webkey.pem

Glance Image
Create a glance image webimage using a file osp-small.qcow2 available inside /tmp

openstack image create --disk-format qcow2 --file /tmp/osp-small.qcow2 webimage

Neutron (Network)
Create a public and private network under Engineering project

openstack network create --external --provider-network-type flat --provider-

physical-network datacentre --project Engineering --enable --no-share public

openstack network create --internal --project Engineering --enable --no-share

private

Create external network with subnet 172.25.250.20/24, gateway as 172.25.250.254, and an

allocation pool between 172.25.250.100-150

openstack subnet create --network public --no-dhcp --project Engineering --

subnet-range 172.25.250.0/24 --gateway 172.25.250.254 --allocation-pool
start=172.25.250.100,end=172.25.250.150 external

Create internal network with subnet range 192.168.1.0/24

openstack subnet create --network private --project Engineering --subnet-range

192.168.1.0/24 internal

Create and configure a router with name Router1

openstack router add subnet Router1 internal

neutron router-gateway-set Router1 public

Server (Instance)
Create an instance/server using the flavor m1.petite, key as webkey, security group as ssh and
Router1 network

openstack server create --image webimage --flavor m1.petite --key-name webkey --

security-group ssh --nic net-id=private webserver

Create a floating IP for the instance

openstack ip floating create public

Assign floating IP to the webserver instance

openstack ip floating add 172.25.250.100 webserver

Block Storage
Create a 2GB block storage volume named storage

openstack volume create --size 2 --project Engineering storage

Attach the storage to the webserver instance as /dev/sdb

openstack server add volume --device /dev/sdb webserver storage

Snapshot
Before creating snapshot detach the volume from webserver instance

openstack server remove volume webserver storage

Here strgsnap is the snapshot name and storage is the name of the volume attached

openstack snapshot create --name strgsnap storage

Attach volume back to webserver after taking the snapshot

openstack server add volume --device /dev/sdb webserver storage

Guestfish
Edit an image using guestfish

yum install -y libguestfs-tools-c

Here osd-webserver.qcow2 is the image which we will edit

$ guestfish -i --network -a /root/osd-webserver.qcow2

><fs> command "yum -y install httpd"

><fs> command "systemctl enable httpd"

><fs> command "systemctl is-enabled httpd"

enabled

><fs> command "touch /var/www/html/index.html"

><fs> edit /var/www/html/index.html
><fs> command "ls -l /var/www/html/index.html"
-rw-r--r-- 1 root root 20 Oct 18 16:14 /var/www/html/index.html

><fs> command "sudo useradd Sheila"

><fs> command "sudo grep Sheila /etc/shadow"

><fs> selinux-relabel /etc/selinux/targeted/contexts/files/file_contexts /

><fs> exit

Above we are installing httpd, enabling the service, creating a password less user Sheila, creating a
dummy index.html file and updating the selinux context which is the most important part. Without
this your image will not work.

Lastly I hope this article with OpenStack command line cheat sheet was helpful. So, let me know
your suggestions and feedback using the comment section.
What is the difference between Root Disk vs
Ephemeral Disk in OpenStack
Before starting to understand about different disk types and the difference between root disk vs
ephemeral disk in OpenStack, you must understand what are Flavors and how to create a flavor.

Below list of flavors exist on my undercloud node in OpenStack

$ openstack flavor list

+--------------------------------------+---------------+------+------+-----------
+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral
| VCPUs | Is Public |
+--------------------------------------+---------------+------+------+-----------
+-------+-----------+
| 3cd8285a-7f9a-4176-91c5-6cc8d4b69d10 | compute | 4096 | 20 | 0
| 1 | True |
| 48c4e5fc-dba5-49e9-98a8-8fca11db781a | control | 4096 | 20 | 0
| 1 | True |
| c238e44a-d74e-4d08-91ae-33effb657005 | baremetal | 4096 | 40 | 0
| 1 | True |
| e3e95dfd-8a64-4cc0-a57f-bd8a133f5673 | block-storage | 4096 | 40 | 0
| 1 | True |
| f95ea5d8-d776-4359-96ae-c5a099ddbdc1 | ceph-storage | 4096 | 15 | 0
| 1 | True |
| fd31fb4f-0ae5-47ce-a68f-3e5f412b503c | swift-storage | 4096 | 40 | 0
| 1 | True |
+--------------------------------------+---------------+------+------+-----------
+-------+-----------+

OpenStack recognises two types of storage: ephemeral and persistent.

Ephemeral storage is storage that is associated only to a specific Compute instance. Once that
instance is terminated, so is its ephemeral storage. This type of storage is useful for basic runtime
requirements, such as storing the instance’s operating system.

Persistent storage, on the other hand, is designed to survive (“persist”) independent of any
running instance. This storage is used for any data that needs to be reused, either by different
instances or beyond the life of a specific instance.

Each instance needs at least one root disk (that contains the bootloader and core operating system
files), and may have optional ephemeral disk. The content for the root disk either comes from an
image stored within the Glance repository (and copied to storage attached to the destination
hypervisor) or from a persistent block storage volume (via Cinder).

Difference between Root Disk vs Ephemeral Disk in OpenStack

Root Disk:

• The root disk contains the operating system for the image, and it is created using an image as
a template.
• If the size of the root disk is smaller than the minimum disk defined by the image,
the instance is not deployed.
• The OpenStack compute service supports the use of root disks based on OpenStack volumes
in order to make root disks persistent.

Ephemeral Disk:

• The ephemeral disk in a flavor defines how much additional disk space is going to be
available to the instance.
• To make this space available, a device is created in the instance.
• The size of that device is the size defined for the ephemeral disk in the flavor.
• This device is a raw device, so it needs to be partitioned and formatted with a file system.
• By default, the size of an ephemeral disk in a flavor is 0 GB.

How to create a flavor with root disk and ephemeral disk?

Here below I am creating a flavor m1.exercise with 10 GB root disk and 2 GB Ephemeral Disk

$ openstack flavor create --vcpus 1 --ram 1024 --disk 10 --ephemeral 2 --swap

1024 m1.exercise
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 2 |
| disk | 10 |
| id | c8be1fd2-c3e5-4cc4-b6ec-c61a6f544f8c |
| name | m1.exercise |
| os-flavor-access:is_public | True |
| properties | |
| ram | 1024 |
| rxtx_factor | 1.0 |
| swap | 1024 |
| vcpus | 1 |
+----------------------------+--------------------------------------+

Lastly I hope the this article on difference between root disk vs ephemeral disk in OpenStack was
helpful. So, let me know your suggestions and feedback using the comment section.
Step by Step Instance Creation Flow in
OpenStack

Have you ever think what happened in the background whenever we create an instance or virtual
machine either via Openstack cli or dashboard, In this this article we will elaborate step by step VM
or instance creation in Openstack. All the Openstack components communicate with each other with
API using Auth-token.

Creating a Virtual Machine in OpenStack via OpenStack CLI

Source the credential file and then execute the nova-boot command,

# nova boot --flavor m1.small --image centos7 --nic net-id={private_network_id} -

-security-group norprod_sec_grp --key-name my_key stack_testvm

Once nova boot command is executed then following are steps are executed behind the scene,

Step:1) The Horizon Dashboard or OpenStack CLI gets user credentials and authenticates with
identity service via REST API

• The identity service (Keystone) authenticate the user with the user credentials and then generates
and send back an auth-token, that auth-token which will be used for sending the request to other
components through REST-Call

Step:2) The Dashboard or OpenStack CLI converts new instance request specified in launch instance
or nova boot command to a REST API request and sent it to nova-api

Step:3) Then nova-api service gets the request and send that request to the identity service
(Keystone) for validation of auth-token and access permission,

• Keystone service validates the token and send the updated authentication headers with roles along
with the permissions
Step:4) After getting the repsonse from keystone, then nova-api checks for conflicts with nova-
database and then it creates initial database entry for new instance or VM.

Step:5) nova-api sends the rpc.call request to nova-scheduler expecting to get updated instance entry
with host id specified

Step:6) Now nova-scheduler picks the request from the queue

Step:7) nova-scheduler talks to nova-database to locate an appropriate host using filtering and
weighing mechanism,

• nova-scheduler returns the updated instance entry with the appropriate host ID after filtering and
weighing
• nova-scheduler sends the rpc.cast request to nova compute for launching an instance on the
appropriate host

Step:8) nova-compute picks the request from the queue and it sends the rpc.call request to nova-
conductor to get the VM or instance info such as host id and flavor (RAM,CPU and Disk)

Step:9) nova-conductor takes the request from queue and communicate with nova-database,

• nova-conductor gets the instance information

• now nova-compute picks the instance information from the queue

Step:10) nova-compute connects to glance-api by making a REST Call using auth-token and then
nova-compute uses the image id to get the image URI from image service and loads the image from
image storage

Step:11) glance-api validates the auth-token with keystone and after that nova-compute gets the
image metadata

Step:12) Nova-compute make the REST-call by passing the auth-token to Network API (Neutron) to
allocate and configure network so that vm gets the IP address

Step:13) Neutron-server validates the auth-token with keystone and after that nova-compute
retrieves the network information.

Step:14) Nova-Compute makes the REST-call by passing the auth-token to Volume API to attach
the volume to the instance or VM.

Step:15) cinder-api validates the auth-token with keystone and then nova-compute gets the block
storage information.

Step:16) nova-compute generates data for the hypervisor driver and executes the request on the
hypervisor using libvirt or API and then finally a VM is created on the hypervior. We can see that
VM in Dashboard and also using “nova list” command.

That’s all from the article, i hope you find this article informative. Please do share your feedback and
comments.