Master
Thesis

Electrical
Engineering

09
2014

Performance of Disk I/O operations

during the Live Migration of a
Virtual Machine over WAN

Revanth Vemulapalli
Ravi Kumar Mada

Department of Communication Systems

Blekinge Institute of Technology
SE-371 79 Karlskrona
Sweden
This thesis is submitted to the School of Computing at Blekinge Institute of Technology
in partial fulfillment of the requirements for the degree of Master of Science in Software
Engineering. The thesis is equivalent to 40 weeks of full time studies.

Contact Information:
Author(s):
Revanth Vemulapalli,
Ravi Kumar Mada.
E-mail:
[email protected],
[email protected].

University advisor(s):
Dr. Dragos Ilie,
Department of Communication Systems.

University examiner(s):
Prof. Kurt Tutschku,
Department of Communication Systems.

School of Computing
Blekinge Institute of Technology Internet : www.bth.se/com
SE-371 79 Karlskrona Phone : +46 455 38 50 00
Sweden Fax : +46 455 38 50 57
 Abstract

Virtualization is a technique that allows several virtual machines (VMs)

to run on a single physical machine (PM) by adding a virtualization layer
above the physical host’s hardware. Many virtualization products allow a
VM be migrated from one PM to other PM without interrupting the services
running on the VM. This is called live migration and offers many potential
advantages like server consolidation, reduced energy consumption, disaster
recovery, reliability, and efficient workflows such as “Follow-the-Sun”. At
present, the advantages of VM live migration are limited to Local Area
Networks (LANs) as migrations over Wide Area Networks (WAN) offer
lower performance due to IP address changes in the migrating VMs and
also due to large network latency.
For scenarios which require migrations, shared storage solutions like
iSCSI (block storage) and NFS (file storage) are used to store the VM’s
disk to avoid the high latencies associated with disk state migration when
private storage is used. When using iSCSI or NFS, all the disk I/O op-
erations generated by the VM are encapsulated and carried to the shared
storage over the IP network. The underlying latency in WAN will effect the
performance of application requesting the disk I/O from the VM.
In this thesis our objective was to determine the performance of shared
and private storage when VMs are live migrated in networks with high la-
tency, with WANs as the typical case. To achieve this objective, we used
Iometer, a disk benchmarking tool, to investigate the I/O performance of
iSCSI and NFS when used as shared storage for live migrating Xen VMs
over emulated WANs. In addition, we have configured the Distributed Repli-
cated Block Device (DRBD) system to provide private storage for our VMs
through incremental disk replication. Then, we have studied the I/O per-
formance of the private storage solution in the context of live disk migration
and compared it to the performance of shared storage based on iSCSI and
NFS. The results from our testbed indicate that the DRBD-based solution
should be preferred over the considered shared storage solutions because
DRBD consumed less network bandwidth and has a lower maximum I/O
response time.

Keywords: Virtualization, XEN, DRBD, iSCSI, NFS, Iometer, Distributed

Replicated Block Device, file storage, block storage, virtual machine, VM
and I/O performance.
ii
 Acknowledgments

I would like to express gratitude to my supervisor Dr. Dragos Ilie who intro-
duced the concept of virtualization to us and for his encouragement and support
through valuable suggestions when required. Furthermore, I would like to thank
Prof. Dr. Kurt Tutschku for his motivation and useful suggestions through the
learning process of this master thesis. I would like to thank Dr. Patrik Arlos for
lending us hardware for our experiments.

I am fortunate to have loving and caring parents who supported my stay and
education against all odds. Without their support I couldn‘t have studied in
Sweden. In my daily work I have been blessed with a friendly and cheerful group
of fellow students (Chaitu, Gautam, Venu, Tushal, etc). I will forever be grateful
for all your love and help.

–Revanth Vemulapalli

iii
iv
 Acknowledgments

I would like to express my gratefulness towards my supervisor Dr. Dragos Ilie,

without whom this research would not have been possible. I’m grateful to Prof.
Kurt Tutschku for his consistent support and suggestions.

I would like to thank Dr. Patrik Arlos for providing required equipment and
environment. I appreciate the cooperation of my partner Revanth.V for being
consistent towards this research work.

I would like to thank my friends who have been great support. Last but not
the least; I would also like to thank my family for their endless support.

–Ravi Kumar M

v
vi
Contents

Abstract i

Acknowledgments iii

Acknowledgments v

List of Contents vii

List of Figures xi

List of Equations xiii

List of Tables xv

1 INTRODUCTION 1
1.1 Advantages of virtualization . . . . . . . . . . . . . . . . . . . . . 2
1.2 Components of Virtualization . . . . . . . . . . . . . . . . . . . . 2
1.3 Types of Virtualization . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3.1 Full Virtualization . . . . . . . . . . . . . . . . . . . . . . 3
1.3.2 Para Virtualization . . . . . . . . . . . . . . . . . . . . . . 3
1.3.3 Hardware Assisted Virtualization . . . . . . . . . . . . . . 4
1.4 Virtual Machine Migration . . . . . . . . . . . . . . . . . . . . . . 4
1.5 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.6 Advantages of Live Migration over WAN . . . . . . . . . . . . . . 7

2 Aim, Objectives, Research Questions & Related Work 9

2.1 Aim and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 XEN Hypervisor and Storage Solutions 17

3.1 Xen Hypervisor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Distributed Replicated Block Device (DRBD) . . . . . . . . . . . 19
3.2.1 DRBD disk replication algorithm . . . . . . . . . . . . . . 21
3.2.2 DRBD Replication Modes . . . . . . . . . . . . . . . . . . 22

vii
 3.2.3 Multiple Replication Transports . . . . . . . . . . . . . . . 23
3.2.4 Three Way Replication . . . . . . . . . . . . . . . . . . . . 24
3.2.5 Efficient Synchronization . . . . . . . . . . . . . . . . . . . 24
3.2.6 XEN with DRBD . . . . . . . . . . . . . . . . . . . . . . . 25
3.3 iSCSI Block Storage . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.1 iSCSI Architecture . . . . . . . . . . . . . . . . . . . . . . 27
3.4 Network File Storage (NFS) . . . . . . . . . . . . . . . . . . . . . 29

4 Research Methodology and Test-bed 31

4.1 Research Methodology . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1.1 Stage 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1.2 Stage 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.1.3 Stage 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2 Experiment Test-beds . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3 Hardware configuration . . . . . . . . . . . . . . . . . . . . . . . . 33
4.4 Benchmarking and other tools . . . . . . . . . . . . . . . . . . . . 34
4.4.1 Iometer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4.2 NetEm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.4.3 Wireshark . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.5 Experiment Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.5.1 Scenario 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.5.2 Scenario 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.5.3 Scenario 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.6 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.6.1 IOps performance . . . . . . . . . . . . . . . . . . . . . . . 37
4.6.2 I/O Throughput . . . . . . . . . . . . . . . . . . . . . . . 38
4.6.3 Maximum I/O Response Time . . . . . . . . . . . . . . . . 38
4.6.4 Network Load . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.7 Delay Characteristics of WAN . . . . . . . . . . . . . . . . . . . . 38

5 Experiment Results 40
5.1 Input Output per Second Performance . . . . . . . . . . . . . . . 41
5.1.1 No delay (0.02 ms) . . . . . . . . . . . . . . . . . . . . . . 41
5.1.2 Inter-Europe (30ms Delay) . . . . . . . . . . . . . . . . . . 42
5.1.3 Trans-Atlantic (90ms Delay) . . . . . . . . . . . . . . . . . 42
5.1.4 Trans-Pacific (160ms Delay) . . . . . . . . . . . . . . . . . 43
5.2 I/O Throughput during Migration . . . . . . . . . . . . . . . . . . 44
5.2.1 Campus Network (0.02 ms) . . . . . . . . . . . . . . . . . 45
5.2.2 Inter-Europe (30ms Delay) . . . . . . . . . . . . . . . . . . 45
5.2.3 Trans-Atlantic (90ms Delay) . . . . . . . . . . . . . . . . . 46
5.2.4 Trans-Pacific (160ms Delay) . . . . . . . . . . . . . . . . . 46
5.3 Maximum Response Time (MRT) . . . . . . . . . . . . . . . . . . 48
5.3.1 No delay (0.02 ms) . . . . . . . . . . . . . . . . . . . . . . 48

viii
 5.3.2 Inter-Europe (30ms Delay) . . . . . . . . . . . . . . . . . . 48
5.3.3 Trans-Atlantic (90ms Delay) . . . . . . . . . . . . . . . . . 49
5.3.4 Trans-Pacific (160ms Delay) . . . . . . . . . . . . . . . . . 50
5.4 Network Load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

6 Conclusion and Future work 53

6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Appendix 56

ix
x
List of Figures

1.1 Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Type-1 and Type-2 Hypervisor . . . . . . . . . . . . . . . . . . . 3
1.3 Types of Virtualization . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Xen Live Migration [9] . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 Three-Phase whole system live migration . . . . . . . . . . . . . . 11

2.2 Incremental Migration Algorithm . . . . . . . . . . . . . . . . . . 12

3.1 Xen Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.2 DRBD Position in I/O stack . . . . . . . . . . . . . . . . . . . . . 20
3.3 Block diagram of DRBD algorithm . . . . . . . . . . . . . . . . . 21
3.4 Asynchronous protocol . . . . . . . . . . . . . . . . . . . . . . . . 22
3.5 Memory or semi Synchronous protocol . . . . . . . . . . . . . . . 23
3.6 Synchronization protocol . . . . . . . . . . . . . . . . . . . . . . . 23
3.7 DRBD Three Way Replication . . . . . . . . . . . . . . . . . . . . 24
3.8 DRBD disk status during vm live migration. . . . . . . . . . . . . 26
3.9 Communication between initiator and target . . . . . . . . . . . . 28
3.10 iSCSI Block-Access Protocol . . . . . . . . . . . . . . . . . . . . . 29
3.11 NFS File-access protocol . . . . . . . . . . . . . . . . . . . . . . . 30

4.1 Experiment setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.2 Scenario 1 and 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.3 Scenario 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.1 Graph: IOps Vs Network Delay . . . . . . . . . . . . . . . . . . . 44

5.2 Graph: MBps Vs Network Delay . . . . . . . . . . . . . . . . . . 47
5.3 Graph: MRT vs Network delay . . . . . . . . . . . . . . . . . . . 50
5.4 Graph: Network Bandwidth vs Storage Solutions . . . . . . . . . 52

xi
xii
List of Equations

3.1 Synchronization time . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.1 Confidence Interval . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.2 Relative error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

xiii
xiv
List of Tables

3.1 Comparison of storage solutions . . . . . . . . . . . . . . . . . . . 27

5.1 IOps performance of different solutions on campus network . . . . 42

5.2 IOps performance of different solutions on Inter-Europe Network . 42
5.3 IOps performance of different solutions on Trans-Atlantic Network 43
5.4 IOps performance of different solutions on Trans-Pacific Network . 43
5.5 I/O throughput for different solutions on Campus Network . . . . 45
5.6 I/O throughput for different solutions on Inter-Europe Network . 45
5.7 I/O throughput for different solutions on Trand-Atlantic Network 46
5.8 I/O throughput for different solutions on Trans-Pacific network. . 47
5.9 MRT of different solutions on Campus network . . . . . . . . . . 48
5.10 MRT of different solutions on inter-Europe network . . . . . . . . 49
5.11 MRT of different solutions on trans-Atlantic network . . . . . . . 49
5.12 MRT of different solutions on trans-Pacific network . . . . . . . . 50

1 NFS address Details . . . . . . . . . . . . . . . . . . . . . . . . . 56

2 iSCSI address Details . . . . . . . . . . . . . . . . . . . . . . . . . 58

xv
xvi
 Acronyms

ABSS Activity Bases Sector Synchronization MBPS Mega Bytes per Second
AVG average page dirty rate MIPv6 Mobile Internet Protocol version 6
CBR Content Based Redundancy MRT Maximum Response Time
Challenge-Handshake Authentication
CHAP Protocol NAS Network Attached Storage
CPU Central Processing Unit NFS Network File Storage
DBT Dirty Block Tracking OS Operating System
DDNS Dynamic Domain Name Service PBA Proxy Binding Acknowledgement
DNS Domain Name Service PBU Proxy Binding Unit
Proxy Mobile Internet Protocol
DRBD Distributed Replicated Block Device PMIPv6 version 6
HIST history based page dirty rate RDMA Remote Direct Memory Access
IM Incremental Migration RPC Remote Procedure Call
IO Input Output RPD Rapid Page Dirtying
IOPS Input Outputs per Second SAN Storage Area Network
IPSec Internet Protocol Security SFHA Super-Fast Hash Algorithm
IPv4 Internet Protocol version 4 TCP Transmission Control Protocol
IPv6 Internet Protocol version 6 TOE TCP offload engine
internet Small Computer System
iSCSI Interface VM Virtual Machine
KVM Kernel-based Virtual Machine VMM Virtual Machine Monitor
LAN Local Area Network VMTC Virtual Machine Traffic Controller
LMA Local Mobility Anchor VPN Virtual Private Network
MAG Mobile Access Gateway WAN Wide Area Network
xviii
Chapter 1
INTRODUCTION

Hardware virtualization is one of the underlying techniques that makes cloud

computing possible [7]. Virtualization is a technique that allows a physical host
to run two or more operating systems simultaneously. This technique adds a
virtualization layer between the physical host’s operating system and hardware,
allowing virtualized hosts to utilize the physical host processor, memory and I/O
devices [1]. The physical host may be a desktop that contains a limited number
of virtual hosts or a Data Centre (DC) containing several virtual hosts. Virtual-
ization has many advantages like server consolidation, providing virtual desktops,
reducing energy consumption, hardware cost reduction, etc. The simple diagram
of virtualization where a single physical machine hosting three virtual machine is
shown in figure1.1.

Figure 1.1: Virtualization

This report is divided into six chapters. The concepts of virtualization and

1
Chapter 1. INTRODUCTION 2

advantages of VM migration over WAN are discussed in chapter one. Chapter two
describes our aim, research question and related work. We introduce the XEN
hypervisor, the disk replication scheme DRBD and shared storage approaches like
iSCSi and NFS in chapter three. We describe our research methodology and ex-
periment setups with detail description of hardware used in our experiment along
with delay characteristics of WAN in chapter four. The analysis of the results
observed during the experiment are presented in chapter five. We concluded our
thesis and discussed possible future work in chapter 6.

1.1 Advantages of virtualization

Some of the advantages of virtualization are hardware abstraction and server
consolidation where several small physical server machines can be efficiently re-
placed by a single physical machine with many virtual server machines. Virtual-
ization is cost efficient. It eliminates the need of several individual physical server
machines thereby reducing the physical space, energy utilization etc. Below is a
list of advantages [3] [51] associated with virtualization:

• Server consolidation and hardware abstraction,

• Proper resource utilization,
• Cost reduction,
• Low power consumption,
• Reduction of physical space,
• Global warming and corporate greenhouse gas reduction,
• Flexibility,
• High Availability, etc.

1.2 Components of Virtualization

Virtualization has two important components, namely the hypervisor or Vir-
tual Machine Monitor (VMM) and the guest. The hypervisor is responsible for
managing the virtualization layer. There are Type1 and Type 2 hypervisors.
Type-1 and type-2 hypervisors are shown in below figure 1.2.
Type-1 hypervisor, which is also known as bare metal hypervisor runs directly
above the host’s hardware. They have direct access to the hardware resources.
They provide greater flexibility and better performance due to its design [9]. Xen,
Microsoft Hyper-V and VMware ESX are the type-1 hypervisors.
Type-2 hypervisor works on top of host machine’s operating system as an
application. For every virtual machine there is one virtual machine monitor
to run and control them, for this reason there is one virtual platform for each
Chapter 1. INTRODUCTION 3

and every virtual machine to work. Virtual box, VMWare player and VMware
workstation are examples of type-2 hypervisors.
The guest is the virtual host that runs above the virtualization layer. The
virtual host has its own operating system (OS) and applications. This guest
operating system can be migrated from one physical host to other physical host.

Figure 1.2: Type-1 and Type-2 Hypervisor

1.3 Types of Virtualization

Virtualization can be classified into three types: full virtualization, para-
virtualization and hardware assisted virtualization [1]. These three virtualizations
are shown in figure 1.3.

1.3.1 Full Virtualization

In full virtualization, binary translation is performed over the virtual ma-
chine’s privileged instructions before sending them to the physical host’s CPU.
Full virtualization is the combination of binary translation and direct execution.
The virtual operating system believes that it owns the hardware itself. The virtual
operating system’s privileged instructions are translated by the hypervisor before
sending them to the CPU whereas user level instructions are directly executed.

1.3.2 Para Virtualization

Pare-virtualization was introduced by Xen projects team. It is a virtualization
technique in which the kernel of the guest operating system is modified to make it
aware of the hypervisor. In this technique the privileged instructions are replaced
Chapter 1. INTRODUCTION 4

by hypercalls that directly communicate with the hypervisor. This technique is

efficient and easy to implement [20].

1.3.3 Hardware Assisted Virtualization

Hardware assisted virtualization is also called accelerated virtualization [4].
This technique allows users to use unmodified operating system by using the
special features provided by computer hardware. Both Intel and AMD started
supporting hardware virtualization (VT-X / AMD-V) since 2006. In this tech-
nique, the virtual machine monitor runs on a root mode privilege level below the
ring 0 [53]. That is, hardware virtualization creates an additional privilege level
below ring-0 which contains hypervisor and leaves ring-0 for unmodified guest
operating system [52]. The protective ring structure of hardware assisted virtu-
alization is shown in figure 1.3. CPU’s manufactured before 2006 cannot make
advantage of this system.

Figure 1.3: Types of Virtualization

1.4 Virtual Machine Migration

.The virtual machine migration is the process of moving a working/running
virtual machine from one physical host to another physical host without interrupt-
ing the services provided by it. During migration the memory, CPU, network and
disk states are moved to the destination host. The end users using the services
provided by the virtual machine should not detect the notable changes. There
are three types of migrations: cold migration, hot migration and live migration.
In cold migration, the virtual machine is first powered off at the source node
before migrating it to the destination node. CPU state, memory state and existing
network connections in the guest OS are lost during cold migration. In hot
Chapter 1. INTRODUCTION 5

migration, the virtual machine is suspended at the source node before migrating
it to the destination node where it is resumed. Most of the OS state can be
preserved during this migration.

Figure 1.4: Xen Live Migration [9]

In live migration [9] the hosted operating system is migrated along with CPU,
memory and disk state from source to destination while the hosted OS is still
in running state without losing the active network connectivity. The disk state
migration is not necessary in the case of using shared storages like Network At-
tached Storage (NAS) or Storage Area Network (SAN). Among three migration
techniques, live migration is best suited to reduce notable downtime of services
running on the virtual machine [9]. By using live migration, load on physical
machines hosting several virtual machines can be decreased to a large extent [22].
Live migration may increase the total migration time of the virtual machines
running server applications.
In order to perform live migration, hypervisor needs to move the memory
state of the virtual machine from the source to the destination physical machine.
The crucial method used for migrating memory state is known as pre-copy, and
Chapter 1. INTRODUCTION 6

is clearly explained in [8] and [9]. In pre-copy phase the memory state of vir-
tual machine is copied to destination host in iterations. Unmodified or unused
pages are moved in first round and modified pages are moved in next nth round.
Hypervisor maintains dirty bitmap to track modified pages. If pre-determined
bandwidth is reached or when bandwidth range is below 256kb, pre-copy phase
is terminated [4] and stop and copy phase is initiated. In stop and copy phase
the virtual machine is paused on source host and modified pages are copied to
destination host. The virtual host is resumed at the destination host and starts
working [3] as usual. Xen live migration time line is shown in figure 1.4 [9].
Various disk state migration and network state migration schemes proposed by
various researchers are discussed in related work section of chapter 2.

1.5 Problem Statement

Xen [8], Hyper-V and VMware are some of the virtualization suits that sup-
port live migration over LAN. Some researchers are working on live migration of
virtual machines over WAN. This is quite challenging because typically the WAN
has limited bandwidth and high latency when compared to the LAN. Migration
over WAN requires network reconfigurations because of the changes observed in
the IP address pool1 at destination host after migration. If Virtual Private Net-
work (VPN) or any other form of tunnels are not used, the source host and
destination host resides at different sub-network. So, the VMs migrating over
WAN should change their IP address to destination hosts subnet to have uninter-
rupted network connectivity. For example a virtual machine hosting a webserver
is running at site 1. Due to some issues like hardware maintenance or overload,
that virtual machine is required to migrate to other location called site 2 which
is at a different geographical location. After migration the virtual machine which
is at site 2, where a different network prefix is used must acquire a new IP ad-
dress. Additionally, it may also have to reconfigure its default gateway. This
situation leads to loss of connection with clients using services of the webserver
running on virtual machine. We conducted a literature study on different live
migration schemes proposed by various researchers to support live migration of
virtual machine over WAN without losing network connectivity. VM migration
over WAN has many advantages like cloud bursting, load balancing, enabling “fol-
low the sun” IT-strategy [11], consolidation, maintenance, scalability, reliability,
recovery, disaster management, etc. [3].
Shared storage and private storage are two data storage techniques used to
store the disk of the virtual machines. Private data storage, which is also called as
direct-attached storage is the storage allotted to a particular host. It cannot share
its unused space with other hosts, whereas shared storage is the space allocated
1
A set of IP addresses available.
Chapter 1. INTRODUCTION 7

commonly to two or more hosts. If a virtual machine use private storage it requires
the virtual disk to be migrated along with memory, CPU and network states. This
increases the downtime and total migration time of the virtual machine. Frequent
disk migrations also leads to high network overhead due to their size. Therefore
shared storage is utilized to eliminate the need of disk state migration when a
virtual machine is migrated. This will reduce the total migration and downtimes.
However, when migrated over WAN, the performance of virtual machine may still
be effected due to the high latency in the network. This is because the virtual
machine uses network to access disk which is located at another geographical
location. To our best knowledge the disk I/O performance of virtual machine
while migrating over WAN was not fully explored earlier. Our thesis work may
look similar to [2], the difference between both the works is discussed in section
2.3.
The primary purpose of our thesis is to investigate disk performing, during the
live migration of a virtual machine over the WAN using different shared storage
techniques. We will analyse the performance of iSCSI, NFS and DRBD storage
techniques and recommend the best technique for specific scenarios.

1.6 Advantages of Live Migration over WAN

Almost all the advantages of VM live migration are currently limited to LAN
as Migrating over WAN effects the performance due to low latency and network
changes. The main goal of our thesis is to analyse the performance of various disk
solutions available during the live migration of VM over WAN. When a virtual
machine using shared storage is live migrated to a different physical host, end
users interacting with a server running on the migrating virtual machine should
not sense notable changes in the performance of the server. Live migration is
supported by various popular virtualization tools like VMWare and Xen. The
advantages of live migration over WAN will motivate our thesis work in this area.

1. Maintenance: During the time of scheduled maintenance all the virtual

machines running in the physical host are migrated to other physical host
so that the maintenance work doesn’t create interruption to the services
provided by the virtual machines.
2. Scaling and Cloud Bursting: Load balancing and consolidation can
make best use of virtual machine migration over WAN. If the physical host
gets overloaded beyond the capacity of hardware resources it will affect the
performance of other virtual machines. So the virtual machines should be
migrated (cloud busted) to physical hosts at other geographical locations
to attain load balancing.
3. Power Consumption: Virtual machines running on low populated hosts
can be migrated to moderately loaded physical hosts at different location.
Chapter 1. INTRODUCTION 8

This allows the initial host to be shutdown to reduce unnecessary power

wastage.
4. Disaster Recovery and reliability: During the times of disaster the
virtual machine running on a physical host can be saved by migrating it
to another physical host over WAN. When a physical host is corrupted or
destroyed the virtual machine can be recreated or booted at other mirror
location by using the VM’s shared disk and configuration file (in case of
Xen) reducing the service downtimes.
5. Follow-the-Sun: It is a new IT strategy where a VM can be migrated
between different time zones in timely manner. This was designed for the
teams working for a project round the clock. Team A works on a project
during their working hours and the data is migrated to other location where
team B will take care of the work during their work hours and migrate data
to team A later.
Chapter 2
Aim, Objectives, Research Questions &
Related Work

In this chapter we briefly described our aim and objectives along with research
questions. We discussed the related work conducted by various researchers on
network and disk state migrations.

2.1 Aim and Objectives

In this thesis we analyzed the performance of disk state during the live mi-
gration of virtual machine over an emulated WAN. We had used Xen [8], an
open source hypervisor developed at the Cambridge university computer science
lab which supports both hardware assisted virtualization and Paravirtualization.
Xen also supports live migration of virtual machines. To overcome the limitations
of disk migration over WAN, which we reported in chapter 1 we had used DRBD
to replicate disk at both cloud locations through periodic updates. We identified
the performance metrics relevant to our approach. We had also repeated the
experiment with other shared storage solutions like NFS and iSCSi. Finally, we
analyzed the performance of all the disk storage solutions and compared their
performance.
The primary purpose of our thesis is to analyze the performance of virtual ma-
chine’s disk during the live migration of virtual machine over WAN. We analyzed
the performance of network and distributed storage solutions and recommend the
best solution.
We laid out the following objectives to achieve the above aim.
• Conduct a literature review of the various virtual machine migration schemes
over WAN.
• Conduct a literature review of various disc migration algorithms/methods
proposed by various researchers.
• Conduct a study on Distributed Replicated Block Device (DRBD) tech-
nique.
• Perform an experiment on a laboratory test bed.

9
Chapter 2. Aim, Objectives, Research Questions & Related Work 10

• Identify the performance metrics to evaluate the above experiment.

• Repeat the experiment with different network storage systems like NFS and
iSCSi.
• Analyze the results and draft the observations.

2.2 Research Questions

The following are the research questions we are going to answer in our thesis.

RQ1:
What are the strategies proposed by various researchers to support live
virtual machine migration over WAN/Internet?
RQ2:
What are the various disc migration schemes proposed by various researchers?
RQ3:
How efficient is the Distributed Replicated Block Device (DRBD) to mirror
disc data in different locations?
RQ4:
What is the I/O performance when migrating virtual machines, over a WAN
using DRBD?
RQ5:
What is the I/O performance when migrating virtual machines with shared
storage solutions over a WAN?
RQ6:
Among distributed and shared storages which solution performs better while
migrating over WAN?

2.3 Related work

Luo et al. [12] discussed a live migration scheme which migrates the local
disk, CPU and memory state of the Virtual node. They proposed a three-phase
migration algorithm that minimizes the downtime caused by large disc migrations.
The three phase migration algorithm was shown in figure 2.1. This migration
scheme has three phases named Pre-copy, Freeze-and-Copy and Post-Copy, which
is similar to Xen’s hypervisors memory migration process. In the first stage disk
data is copied to destination in n iterations. Block-map is used to track the dirtied
disk during this stage. The dirtied disk data is copied during the next iteration.
This stage is limited to a particular number or iteration and is proactively stopped
when the dirtying rate is faster than the transfer rate. In freeze-and-copy phase
the virtual machine running on the source machine is suspended and dirtied
Chapter 2. Aim, Objectives, Research Questions & Related Work 11

memory pages are migrated to destination along with CPU state. The third
stage is the combination of push and pull phase. Here the virtual machine is
resumed and according to the details in dirty bitmap, the source push the dirty
blocks and the destination machine pulls them. This process reduce the total
migration time of the virtual machine.

Figure 2.1: Three-Phase whole system live migration

In [12] authors developed a mechanism called Incremental Migration (IM)

algorithm which reduce the total migration time when migrating the virtual ma-
chine back to the source machine. When the virtual machine is sent to the physical
machine from which it is migrated earlier, this algorithm checks the block-bitmap
to find out the dirty blocks modified in the virtual disk after earlier migration.
Only this dirtied blocks are migrated to the migrating physical location. This
algorithm is shown in the figure 2.2.
Kleij et al. [2] conducted an experiment on laboratory test-bed to evaluate
the possibility of using Distributed Replicated Block Device (DRBD) [6] for mi-
Chapter 2. Aim, Objectives, Research Questions & Related Work 12

Figure 2.2: Incremental Migration Algorithm

grating disk state over WAN, which created two copies of data disks by mirroring.
DRBD supports asynchronous, semi synchronous and full synchronous replication
modes. All the three replication modes are discussed in chapter 3.2.2. They ex-
pected that using DRBD in asynchronous mode gives better performance. They
constructed a laboratory test bed with 50ms RTT delay to emulate WAN. They
compared the performance of DRBD with high latency Internet Small Computer
System Interface (iSCSI) and concluded DRBD test is 2.5 times faster than re-
mote iSCSI test. The downtime observed during the virtual machine’s migration
is only 1.9 seconds. While coming to read statistics, DRBD’s performance is
better when compared to remote iSCSI. But there is no significant difference be-
tween the performance of DRBD and local iSCSI. However, the authors admit
that there are some inconsistencies in their results, which they cannot account
for. Our study may look similar to this work. In our thesis we include NFS
as a shared storage solution and we also measure the performance for different
latencies. In this authors analyzed the virtual machines disk performance based
Chapter 2. Aim, Objectives, Research Questions & Related Work 13

on migration time, http mean request time and I/O read statistics. They used
DRBD asynchronous replication algorithm to replicate data between nodes. We
analyzed the disk I/O performance using different performance metrics discussed
in section 4.6 by conducting both read and write I/O tests. We used DRBD
synchronous replication model which is more secure than asynchronous protocol
used in [2]. Our experiment results confirm that DRBD outperformed iSCSI and
NFS while the virtual machine is migrated over WAN.
Wood et al. [11] discussed solutions for the limitations faced by virtual
machines migrating over WAN. They experimented with an architecture called
CloudNet that is distributed across datacenters in the United States separated
by 1200km. Their architecture reduced network bandwidth utilization by 50%,
and memory migration and pause time by 30 to 70%. The authors used the
DRBD disk replication system to replicate disk in both source and destination.
Firstly DRBD brings the remote disk to consistent stage using synchronization
demon. When both the disks are synchronized and stable, DRBD’s synchronous
replication protocol is switched and the modified data is placed in TCP buffer
to transmit towards destination disk. When the write state is confirmed on des-
tination the synchronous mode is completed. The memory and CPU modes are
migrated after disk migration stage. Content Based Redundancy (CBR) [11]
block technique is used to save bandwidth. CBR splits disk blocks and memory
pages into fixed size finite blocks and uses Super-Fast Hash Algorithm (SFHA)
to generate hash bases on their content. This hash is used to compare previously
sent blocks. This solution saved 20 GB bandwidth when compared to the pro-
cess of migrating full disk. It also reduced memory transfer time by 65%. They
designed a scheme to retain network state using layer-2 VPN solution. In this
experiment authors analyzed the network bandwidth utilized for virtual machines
migration along with memory migration and pause times. On the other hand, in
our experiment, we focused on the disk I/O performance of DRBD, iSCSI and
NFS when the VM is migrating over WAN along with network utilization. From
our experiments we can say that iSCSI and NFS consumed more than 75% of
network bandwidth than DRBD.
Akoush et al. discussed the parameters [19] which effect the live migration
of virtual machines. Total migration time and downtime are the two important
performance parameters used to analyze virtual machines performance during
migration. Total migration time is the time required to move virtual machines
from one physical host to another physical host. Downtime is a period of time
where virtual machine doesn’t run. Page dirtying rate and network bandwidth
are the factors which affect the total migration time and downtime. The authors
implemented two simulation models, namely average page dirty rate (AVG) and
history based page dirty rate (HIST) which are accurate up to 90 percent of actual
results. They proposed a model called Activity Based Sector Synchronization
(ABSS) [22] which migrates virtual machine efficiently in a timely manner. The
Chapter 2. Aim, Objectives, Research Questions & Related Work 14

ABSS algorithm predicts the sectors that are likely to be altered. This algorithm
helps in minimizing the total migration time and the network bandwidth.
Robert et al. [23] worked on both disk related and network related issues.
They worked on migrating local persistent state by combining block level solution
with pre copy state. When the virtual machine need to migrate, disk state is first
pre-copied to the destination host. The virtual machine will still run during
this stage. After some time this mechanism starts XEN to migrate memory and
CPU states of the virtual machine. All the changes made to the disk on source
side during the migration process are recorded in the form of deltas (Unit which
contain written data, its location and size). These deltas are sent to destination
and applied to the image. This mechanism has downtime of 3 seconds in LAN and
68 seconds in WAN. They combined Dynamic DNS (DDNS) with IP tunneling
to retain old network connections after live migration. When the virtual machine
in source host goes to pause state, a tunnel is created using Linux iproute2. This
tunnel is between the virtual machine’s old IP address and the new IP address at
destination. This mechanism discards / drops the packets on source host’s side
during the final stage of migration. The packets related to the old connection are
forwarded from source host to IP address of the virtual machine at destination
via tunnel. The virtual machine is configured with a new dynamic IP address
to prevent it from using the old tunnel for new connections. Thus, the virtual
machine will now have two IP addresses. The disadvantage of this mechanism is
that the tunnel between source and destination will not be closed until the old
connections end. This requires the source host to run until old connections are
closed.
Kazushi et al. [13], used data duplication technique to propose a fast virtual
machine storage migration. This technique will reduce the volume and time of
data transfer. Suppose there is a situation where a virtual machine should migrate
frequently between site A and site B. Migrating a large disk between sites A and
B will waste a lot of network resources and will take a long time. These frequent
disk migrations can be eliminated by using duplication. When the VM is migrated
for the first time from site A to site B, full disk is copied to site B. When the VM
migrates back to site A, only changes made on disk in site B are replicated to
disk in site A. A new diff image structure, which is a new virtual machine image
format and Dirty Block Tracking (DBT) are developed to track the changes. This
technique successfully reduced the migration time from 10 minutes to 10 seconds.
Travostino F et al. discussed about the advantages and requirements of long-
haul live migration [5]. Their experiment proved that virtual machines can mi-
grate over WAN across long distance geographical locations instead of being lim-
ited to small local datacenters. Their scheme has a dedicated agent called as
Virtual Machine Traffic Controller (VMTC) that created dynamic tunnels be-
tween clients and the virtual hosts. The VMTC is responsible for migration of
the virtual machine and it maintains connectivity with the destination host where
Chapter 2. Aim, Objectives, Research Questions & Related Work 15

the virtual machine should be migrated. The VMTC communicates with Authen-
tication, Authorization and Accounting (AAA) module to get authentication in
the form of a token to setup an on-demand, end-to-end path. It is the responsibil-
ity of the VMTC to migrate the virtual machine and reconfigure the IP tunnels
so that it can seamlessly communicate with its external clients. When a virtual
host migrated from one host to other host the tunnel is broken and a new tunnel
is created between the client and virtual host. They migrated their virtual ma-
chine between Amsterdam and San Diego using a 1Gbps non-switched link with
Round Trip Time (RTT) of 198ms. The application downtime observed during
the migration is in the range of 0.8 to 1.6 seconds. Their experiment results show
that, when compared to LAN migration, the downtime of migration over WAN is
just 5-10 times higher, but the round trip time is more than 1000 times.This may
be due to the dynamically established link (light path) of 1Gbps using layer-1 and
layer-2 circuits without routing protocols. Anyhow, layer-3 protocols are used for
communication between VM and clients.
Eric et al. [14] used Mobile IPv6 (MIPv6), a protocol for managing mobility
of mobile nodes across wireless networks, to support live migration of virtual
machine over WAN. The virtual machines behave as mobile nodes by enabling
MIPv6 [24] in the TCP/IP protocol stack. This scheme enables a virtual machine
to retain its IP address after migrating from the source host. It eliminates the
need of DNS updates [23] on the virtual machine. The disadvantage in this scheme
is that the virtual machine’s kernel must be modified to support mobility (unless
that was already preconfigured).
Solomon et al. presented a live migration approach using Proxy Mobile IPv6
(PMIPv6) protocol [15]. This approach is similar to MIPv6 [14] but it does not
require installation of mobile software on a virtual machine [25]. However, it
requires specific infrastructure deployed in the network. In this experiment the
source host and destination host act as Mobile Access Gateways (MAGs) and are
connected to the Local Mobility Anchor (LMA). This choice in their experiment
was just for convenience, to keep the experiment testbed simple. The LMA and
MAG are entities independent infrastructure elements. The LMA and MAGs are
equivalent of a home agent and foreign agent in Mobile IP. The LMA arranges that
packets for the mobile node and forwarded them to the MAG where the node is
currently located. The MAG is responsible for emulating the mobile node’s home
network and for forwarding packets sent by the mobile node over the tunnel to
LMA. LMA acts as a central management node tracking the virtual machines.
When a virtual machine is booted on source machine, suppose MAG1 it registers
with LMA via Proxy Binding Unit (PBU) and Proxy Binding Acknowledgement
(PBA). Upon successful registration a tunnel is created between MAG1 and LMA.
Now the virtual machine can be connected to outer world via LMA. When the VM
is migrated to other location, for example toMAG2, the tunnel between MAG1
and LMA is broken and the VM is deregistered via PBU and PBA messages. Now
Chapter 2. Aim, Objectives, Research Questions & Related Work 16

the tunnel is created between MAG2 and LMA using the same previous process
that happened for MAG1. In this solution VM will retain its ip address and
network connections after the migration. Their experiment results showed that,
this approach migrated virtual machine with minimum downtime when compared
to MIPv6 approach. In this experiment authors used iSCSI shared storage to store
VM’s disk to avoid disk state migration.
Forsman et al. [16] worked on automatic seamless or live migration of virtual
machines in a data center. They developed an algorithm that migrates virtual
machine based on the CPU work load. They used two strategies called push
and pull. When the physical host’s workload crosses higher threshold level, the
hotspot is detected and the push phase initiates virtual machine migration to
other underutilized physical host to balance the load. When a physical host is
underutilized and is below the lower threshold level the pull phase is initiated
to balance the system by requesting virtual machines from other physical nodes.
They also used a concept called variable threshold, which varies the threshold
level of the load periodically, which resulted in a more balanced system. They
successfully simulated their technique using OmNet++ simulation software.
Chapter 3
XEN Hypervisor and Storage Solutions

In this chapter we described about the Xen hypervisor and various stor-
age solutions we used to analyse the disk performance during the live migration
of virtual machine. Xen, Kernel Virtual Machine (KVM), VMware ESXi and
Hyper-V are some of the widely used type-1 hypervisors. Among these hyper-
visors VMware ESXi and Hyper-V are expensive and proprietary software with
restrictive licence scheme. So, we restricted ourselves to open source hypervisors
like Xen and KVM. KVM requires the physical host to support hardware assisted
virtualization (HVM) but unfortunately our testbed doesn’t have hardware sup-
porting HVM. So, we used Xen hypervisor and created para-virtualized virtual
machines. We briefly described Xen hypervisor along with its features in section
3.1.
We used internet Small Computer Interface (iSCSI), Network File Storage
(NFS) and Distributer Replicated Block Device (DRBD) storage solutions to store
the disk of the virtual machines. These are the three widely used storage solutions
supported by various Hypervisors to store the disk state of virtual machines. Each
storage solution has its own advantages and disadvantages. A short comparison
of these three storage solutions is shown in table 3.1.
Among these three solutions DRBD and iSCSI are block storage solutions and
NFS is a file storage solution. In this thesis we used Xen hypervisor to migrate a
virtual machine while conducting disk I/O tests. Our virtual host has its disk at
shared or replicated storage solutions. So, the Xen migrates only the memory and
CPU states leaving the disk state. The performance of VM’s disk I/O operations
depends on underlying storage solutions. So, we assume that our results are valid
for other available hypervisors.

3.1 Xen Hypervisor

Xen, [8] is an open source hypervisor developed at the Cambridge university
computer science lab which supports both hardware assisted virtualization and
Paravirtualization. It also supports live migration of virtual machines. Xen is

17
Chapter 3. XEN Hypervisor and Storage Solutions 18

one of the five1 type-1 or bare metal hypervisor that are available as open source
[42]. Xen allows multiple operating systems to run in parallel to host operating
system. Xen is used for different open source and commercial application such
as desktop virtualization, server virtualization, Infrastructure as a service (IaaS),
security, hardware and embedded appliances. Today Xen hypervisor is powering
large clouds in production [20]. The Xen hypervisor is responsible for handling
interrupts, scheduling CPU and managing memory for the virtual machines.

Figure 3.1: Xen Architecture

Dom0 or the Domain-0 is the domain in which Xen starts during the boot.
From the Xen architecture which is shown in figure 3.1 we can see that Dom0
is the privileged control domain which has direct access to the underlying hard-
ware. The Dom0 has the toolstack which is a user management interface to the
Xen hypervisor. Xen toolstack can create, manage and destroy virtual machines
or domUs which are unprivileged domains [20]. Xen supports hardware virtual-
ization and Paravirtualization. In hardware virtualization, unmodified operating
systems can be used for the virtual machines whereas Paravirtualization requires
modification to the operating system’s kernel running inside virtual machines.
Doing so will increase the performance of paravirtualized hosts. The host operat-
ing system should be Xen Paravirtualization enabled to create virtual machines.
The Linux kernels before 2.6.37 version are not Paravirtualization enabled. Their
kernels should be recompiled to enable Paravirtualization. All the Linux kernels
released after 2.6.37 version are by default Xen Paravirtualization enabled.
1
The other open source hypervisors are KVM, OpenVZ, VirtualBox and Lguest.
Chapter 3. XEN Hypervisor and Storage Solutions 19

Xen allows virtual machines to migrate between hosts while the guest operat-
ing system is running. This feature is called live migration. In Xen, the demons
running in the Dom0 of source and destination hosts takes the responsibility of
migration. The memory and CPU states of the virtual machine are migrated
from source machine to destination machine by the control domain. The Xen
Hypervisor copies memory pages in series of rounds using Dynamic Rate-limiting
and Rapid Page Dirtying [8] techniques to reduce the service downtime. Dy-
namic Rate-Limiting algorithm adapts bandwidth limit for each pre-copy round
and is used to decide when the pre-copy stage should end and stop-and-copy
phase should start. Rapid Page Dirtying algorithm is used to detect the rapidly
dirtied2 pages and skip them from copying them in pre-copy stage. Xen uses a
microkernel design which consists of small footprint and interface that is around
1MB of size making it more secure and robust than the other available hypervi-
sors. Xen hypervisor is capable to run main device driver for a system inside a
virtual machine. The virtual machine that contains main device driver and device
driver can be rebooted leaving the rest of the system unaffected [20]. This feature
of Xen is called driver isolation. Driver isolation is a safe execution environment
which protects the virtual machines from any buggy drivers [55]. Xen is operating
system agnostic which means different operating systems like NetBSD and Open
Solaris can be hosted.
Basically Xen supports two types of virtual Block Devices named ‘Phy’ and
‘file’. Phy is the physical block device which is available in the host environment
where as file is the disk image which is available in the form of a file in the host
computer. The loop block device is create from the available image file and the
block device is handled to the domU. Shared storage solutions like iSCSI use Phy
and NFS use file.

3.2 Distributed Replicated Block Device (DRBD)

DRBD [43] stands for Distributed Replicated Block Device. It is a software-
based tool which replicates data from block devices between servers and provides
virtual/cluster storage solution. The block devices may be hard disks, logical
volumes or disk partitions. In DRBD the data is replicated as soon as the ap-
plication writes to or modifies the data on the disk. The top level applications
are not aware of replication. DRBD uses synchronous and asynchronous tech-
niques to mirror the block device. DRBD is controlled by a cluster software
called “heartbeat”. If an active node crashes heartbeat initiates the failover pro-
cesses. Each DRBD peer acts either in primary or secondary failover role. The
user space applications in the secondary node will not have write access to the
DRBD disk resource but they can read data. Write access to data is granted only
2
Rapidly modified memory pages which are generally copied in stop-and-copy phase.
Chapter 3. XEN Hypervisor and Storage Solutions 20

in primary mode. DRBD’s Cluster management software is generally responsible

for the role assignment.

Figure 3.2: DRBD Position in I/O stack

DRBD’s functions are purely implemented using Linux kernel module. Fea-
tures like flexibility and versatility makes DRBD suitable for providing duplica-
tion solution to many applications. DRBD package has two main components
namely, the driver code and the user tool. The driver code runs in kernel space
and the user tool is used to control the driver code and cluster management pro-
grams from user space. Drbdadm, drbdsetup, drbdmeta are some of the user tools
used to communicate with kernel module for managing and configuring the re-
sources of DRBD. DRBD remained an out-of-tree kernel module for many years
[43] but was then integrated into the Linux kernel from version 2.6.33 release.
We used DRBD v8.4.3 in our experiments. DRBD installation and configuration
details are described in the appendix.
DRBD has two operation modes namely single primary mode and dual pri-
mary mode. In single primary mode only one of the two disks in the cluster
acts as primary disk. All the changes done to the data on this primary disk are
replicated to the other peer disk which is at a different physical location. The
secondary disk can be used as backup in case of data loss on the primary disk.
The dual primary mode was first introduced in DRBD v8.0 and disabled by de-
fault. In dual primary mode both the disks in the cluster act as primary disks.
The dual primary mode can be integrated with Xen hypervisor to eliminate disk
migration during the live migration [2]. The changes we made in “/etc/drbd.conf ”
Chapter 3. XEN Hypervisor and Storage Solutions 21

configuration file enabled dual primary mode and integrated DRBD with XEN.
These configuration changes are described in the appendix.

Figure 3.3: Block diagram of DRBD algorithm

3.2.1 DRBD disk replication algorithm

In this section we explained the DRBD disk replication algorithm [6] by using
a block diagram shown in figure 3.3. DRBD uses five steps to replicate write event
on both the disks.

1. When a write request is issued by an application running on primary node,

the write request is submitted to the local I/O system and is placed in the
TCP buffer to be sent to the peer node.
2. The write event is send to the peer node in the form of DRBD data packet
over TCP and is written on the disk of the peer node.
3. When the write request is written on peer node acknowledge (ACK) is
created and send to source node.
4. The ACK packet will reach source node.
5. Local write confirmation is received by source node.

Order of events from 1 to 4 is always the same. But the write event on local
disk can take place anytime independently of events 2 to 4.
Chapter 3. XEN Hypervisor and Storage Solutions 22

3.2.2 DRBD Replication Modes

DRBD supports three replication modes namely Protocol A, Protocol B and
Protocol C [43]. They were also called as Asynchronous, Memory Synchronous
and Synchronous modes.

3.2.2.1 Protocol A

Figure 3.4: Asynchronous protocol

Protocol A is also called as Asynchronous replication protocol. In this

protocol the operations written on the primary disk are considered complete after
the data is written on the local disk and data packets are placed in the local TCP
buffer. When the local disk crashes, failover occurs on local disk but the data
is consistent on the secondary disk. The data modified before crash may be lost
but the secondary disk will be in consistent state. This secondary disk can be
used instead of crashed primary disk or the data from the secondary disk can be
synchronised with a new primary disk.

3.2.2.2 Protocol B

This protocol is also called as Memory Synchronous protocol or SemiSyn-

chronous protocol. In this protocol the operations written on the primary node
are considered complete when the data is written on the local disk and data pack-
ets have reached to the location of secondary node. With Protocol B, written data
is generally not lost during the forced failover but the data maybe lost in case of
failure occuring on both the locations at same time.
Chapter 3. XEN Hypervisor and Storage Solutions 23

Figure 3.5: Memory or semi Synchronous protocol

Figure 3.6: Synchronization protocol

3.2.2.3 Protocol C

This protocol is also called synchronous replication protocol. Write operations

are considered completed on primary node when write operations are confirmed
from both remote and local disks. This assures that no data loss occurs in the
nodes. However data loss may occur if both nodes are destroyed irreversibly at
same time or may be lost due to power failure at both the locations. Protocol C
is the most commonly used in DRBD setups [43].

3.2.3 Multiple Replication Transports

Multiple low-level transports like TCP over IPv4, TCP over IPv6 and Su-
perSocket [54] are supported by DRBD. This was made available from DRBD
v8.2.7. TCP over IPv4 is DRBD’s default and canonical implementation. Any
system that is IPV4 enabled can use this implementation. DRBD can also use
TCP over IPv6 as its transport protocol and there should be no difference in the
performance. The only difference between TCP over IPv4 and TCP over IPv6
Chapter 3. XEN Hypervisor and Storage Solutions 24

is their address schemes. In SuperSocket, TCP/IP portion stack is replaced by

super sockets with a RDMA capable, single monolithic and highly efficient socket
implementation. The disadvantages with this transport is, it is limited to the
hardware which is produced from a single vendor named Dolphin interconnect
Solutions.

3.2.4 Three Way Replication

Figure 3.7: DRBD Three Way Replication

In three way replication an additional disk at another node is added to DRBD.

This additional disk is added on top of the existing primary and secondary node
design. The data is replicated to the third disk which can be used for immediate
backup and disaster recovery purpose. Three way replication is shown in the
figure 3.7. In three way replication different DRBD protocols can be used. For
example protocol A can be used to for backup and at the same time protocol C
can be used for replicating production data from primary to secondary disk.

3.2.5 Efficient Synchronization

To make DRBD operational, Disks at both the nodes should be first synchro-
nized so that the data will be consistent. Typically data from the primary node
will be synchronized with secondary node. There is a difference between disk
synchronization/ Re-synchronization and disk replication. In disk replication the
write event issued by an application on the primary node to make changes to the
data on the disk is replicated to the secondary node. The disk synchronization is
decoupled from the writes and the entire disk is affected. In DRBD, if disk repli-
cation was interrupted due to node (may be primary or secondary) or network
failures one of the peer disk will become inconsistent. The data in inconsistent
disk cannot be used. So the disk synchronization should be employed to make
Chapter 3. XEN Hypervisor and Storage Solutions 25

both the disks consistent. Synchronization is really effective as DRBD doesn’t

synchronize modified blocks in the original written order, but rather in linear or-
der. This is because, several successive write operations written on the consistent
disk are synchronized in blocks with the peer disk. Synchronization is also fast
because the inconsistence disk is partly updated and it requires synching only the
new data blocks.
When one of the primary or secondary node is affected by failure that disk
become inconsistent. The inconsistent disk cannot be used by DRBD until it
is synchronised by the peer nodes disk. If the secondary node failed during
the operation it enters into inconsistent state. But the application running on
the primary node can operate without any disturbance. When secondary node
is restarted DRBD initiates disk synchronization so that the changes made on
primary disk are synchronized to secondary disk. If the primary node fails the
applications cannot use it. It should be synchronized from the secondary disk.
In DRBD the disk synchronization should be initiated during the deployment
time. If it is done before or during migrating virtual machine, the Tsync time
required and the bandwidth utilized for it will affect the performance of the VM.
When the synchronized peer node is removed and new node is added then the
initial synchronization should be initiated during the deployment of new node.
In DRBD the initial synchronization consume more network bandwidth in the
initial stage but improves the VM performance during migration.
Tsync is the expected synchronization time. Synchronization time depends on
the volume of data modified by the user application when the secondary node is
down and the rate of synchronization. D is the volume of data to be synchronized,
on which we may not have any influence over (this is the volume of data that was
changed by user application while the replication link was broken). R is the rate
of synchronization, which is configurable (by administrator) and depends on the
type of network being used or available network bandwidth.
D
Tsync = (3.1)
R

Formula 3.1. Synchronization time

3.2.6 XEN with DRBD

When DRBD is integrated with Xen hypervisor it acts as a virtual block stor-
age device. Xen uses DRBD block device to make the entire content of domU’s
virtual block disks available at two different locations by replicating them. When
the virtual machine is migrated to the other location it use the disks at that
location to read and write I/O operations. In this way DRBD provides redun-
dancy virtualized operating systems managed by the Xen hypervisor [43]. DRBD
supports both x64 and x86 operating systems.
Chapter 3. XEN Hypervisor and Storage Solutions 26

Double primary mode should be activated on both the locations to enable

the DRBD disks to accept write events from the migrated virtual machine. The
DRBD configuration file which is at /etc/drbd.conf location should be modified to
enable dual primary mode at both locations. Xen virtual disk details in the virtual
machine configuration file found at the directory /etc/xen/ should be modified to
support DRBD. All the modifications done in the configuration files of Xen and
DRBD are described in the appendix. Let us suppose there is a virtual machine
which should migrate between locations A and B. Before the virtual machine
boots DRBD scans the disks at both locations and synchronize them if they are
not up-to-date. The system uses the last updated disk to synchronize the other
disk. When the virtual machine boots up at location A it uses the local disk
to store and access its data (i.e., this disk becomes the target for read or write
I/O events). The DRBD demon replicates the changes made on local disk to the
disk at location B using DRBD’s replication protocols (for example protocol C).
During this stage, the disk at location A acts as primary disk and disk at location
B acts as secondary disk.

Figure 3.8: DRBD disk status during vm live migration.

When an emergency situation like maintenance or load balancing arises it

requires virtual machine to be migrated from location A to location B. During
the migration, DRBD disks enters dual primary mode. During the initial stage
of migration the virtual machine enters pre-copy stage where pages are copied to
destination in iterations. During this time the changes made on disks at location
A are replicated to disks at location B. After some time the virtual machine
is suspended at location A and resumed at location B. As soon as the virtual
machine is resumed at location B and the entire resources are pulled the status
of disks at location A changes to secondary and the disks at location B acts as
primary disks. All the changes made on disks at location B are replicated to disks
at location A.
Chapter 3. XEN Hypervisor and Storage Solutions 27

iSCSI NFS DRBD

Category Block-storage File-storage Block-storage
Storage type Network based Network based Replication based
Virtualization Compatible Compatible Compatible
compatibility
Configuration Medium Easy Difficult
complexity
Maximum hosts 256 256 3
Maximum volume 64 TB 64 TB 1PB (1024 TB)
Volume elasticity Yes Yes Yes

Table 3.1: Comparison of storage solutions

3.3 iSCSI Block Storage

Internet Small Computer System Interface (iSCSI) is the most widely used
network based block-access storage technique which uses IP technology [44]. The
iSCSI [46] uses SCSI, a popular block transport command protocol over the tradi-
tional IP network to transmit high performance “block-based” storage data over
the local Area Network (LAN) and Wide Area Network (WAN). This technol-
ogy is well understood, affordable, easy to implement, not limited by distance,
scalable and secure. The iSCSI allows user for remote mirroring, remote backup
and similar applications. Incorporating IP networking into a storage network can
remove barriers to implementing and managing networked storage.

3.3.1 iSCSI Architecture

The iSCSI has two important components namely iSCSI target and iSCSI
initiator. The iSCSI target is the endpoint, where storage disk is located. It
provides I/O data to the iSCSI initiator when requested. The iSCSI initiator is
an endpoint which functions as a client by starting iSCSI sessions. Applications
on iSCSI initiators are enabled to perform traditional “block-level” transfers over
a common network. The iSCSI is built on two well-known protocols, SCSI and
Ethernet, which are the dominant standards for networking and storage. Initia-
tors use IP protocols to transmit and receive data from the storage disk which is
located at iSCSI target. The iSCSI initiator uses SCSI protocol to encapsulate
the I/O data generated by the application. These packets are again encapsulated
using TCP/IP protocol and sent to the iSCSI target over the network [47]. I/O
data is decapsulated back by the iSCSI target and written to the disk. IPsec
and Challenge-Handshake Authentication Protocol (CHAP) are used to provide
authentication between initiator and target.
When a connection is established between initiator and target, the disk block
Chapter 3. XEN Hypervisor and Storage Solutions 28

Figure 3.9: Communication between initiator and target

is exported from the target and it is seen as a local disk by the operating system
of the iSCSI initiator. The applications running on the initiator are not aware of
the iSCSI target. Here the initiator runs the local file system which reads from
and write the data blocks to the iSCSI target when required. This is displayed
in the figure 3.9. In Xen virtualization iSCSI is used to store the disks of virtual
machine so that the disks are available even after migrating virtual machine to
another physical host. This requires the two physical hosts to connect with the
iSCSI target. When the virtual machine is running on physical host A, it access
the disk which is at other location (iSCSI target) using iSCSI initiator drivers
running on physical host A. When it is migrated to the physical host B (which
is configured to connect the iSCSI target), it will resume using the disk from the
iSCSI target. The process to configure iSCSI to enable live migration using Xen
hypervisor is described in the appendix.
The physical node hosting a virtual machine needs an iSCSI initiator con-
nected to the network in order to access storage from the target. An iSCSI driver
can act as initiator with a standard network card or with a TCP offload engine
(TOE) network card which reduces the CPU utilization. The storage device must
also implement the iSCSI protocol stack on the target side. Unlike NFS which is
a file storage solution, block storage solutions like iSCSI doesn’t have file locking.
In iSCSI the target can’t connected to multiple initiators at same time unless
the target uses a cluster aware file systems like GFS and OCFS2. The multi-host
feature which is disabled by default should be enabled. In iSCSI each initiator has
a unique iSCSI Qualifier Name (IQN) id which is used to connect to the target.
Chapter 3. XEN Hypervisor and Storage Solutions 29

The iSCSI target generally grants access control based on the IQN. The virtual
machine can’t be created from different location using the same disk from iSCSI
target at a same time.

Figure 3.10: iSCSI Block-Access Protocol

3.4 Network File Storage (NFS)

Network File System also termed as NFS [48] was developed by Sun Microsys-
tems in 1984. It is a distributed file system which allows clients to access user
data over the IP network as if the data is on the local disk. NFS is designed to
be easily portable to other operating systems and machine architectures, unlike
many other remote file systems that are implemented under UNIX. NFS is fast,
well-developed and it can handle non homogeneous systems or operating systems.
It is a file-access protocol where the local namespace of the NFS server machine
is shared with NFS clients [44]. In other words, NFS shares the files from the
server to the clients and these files can be for example images, music, documents
etc. Remote Procedure Call (RPC) based protocol is used by the clients to
access files and meta-data from the server. In the NFS storage solution the file
system is located on the server and the I/O operations are transmitted over an
IP network. The structure of NFS protocol is shown in figure 3.11. There are
four versions of NFS available namely NFSv1, NFSv2, NFSv3 and NFSv4. In the
earlier two versions NFS clients used RPCs over UDP to communicate with NFS
server machines. The latter two versions added features such as TCP support
and asynchronous writes thereby increasing the reliability and performance. The
Chapter 3. XEN Hypervisor and Storage Solutions 30

Figure 3.11: NFS File-access protocol

NFSv4 is the latest protocol in which different protocols like nfs3 ,mountd, nsm
and nlm [56] are integrated into a single protocol. Mountd protocol is responsible
for mounting exported system so that it can be accessed by nfs. Network Status
Monitor (nsm) protocol is used to monitor the machines state i.e., client or server.
Network Lock Manager (nlm) protocol manages a lock system which avoids the
clients to modify the data at same time.
We had used NFSv4 to create a NFS server-client setup in our laboratory
test-bed using three computers. We had one NFS server to share storage with
the two NFS clients. We created disk images for virtual machine and placed them
in a shared directory. If a virtual machine running on client 1 make changes to
the disk image the changes are written to the NFS server over the IP network.
The changes made on NFS server are made available to the NFS client 2. When
the virtual machine is migrated from client 1 to client 2 it uses the disk available
at client 2 and works as usual. NFS and Xen configurations to support live
migration are described in the appendix.

3
Nfs is the base protocol which allows to create, read, write and search files.
Chapter 4
Research Methodology and Test-bed

In this chapter we describe the research methodology we followed to achieve

our objectives and the three laboratory test beds we constructed to conduct our
experiments. We also discuss the various tools used in this experiment and the
performance metrics we used to analyze the performance of various disk storage
solutions. We concluded this chapter by describing the delay characteristics of
WAN.

4.1 Research Methodology

In this thesis we followed applied research method [17]. It is used to solve
practical problems faced by a group or by an individual. In this method research is
carried on the basis of the previous work carried by earlier researchers. In general
applied research is conducted to solve the research questions either theoretically or
practically/experimental methods. It can be used for solving scientific problems
and developing new technologies. To achieve our research objectives we divide
our research approach into three stages. In stage 1 we conducted literature review
to gain in-depth knowledge in this field. We had constructed a laboratory test
bed to conducted experiments in stage 2 and analyzed the performance of the
three storage solutions in stage 3. The three stages are explained below.

4.1.1 Stage 1
In this phase we conducted a thorough literature study to gain proper tech-
nical knowledge in the area of virtualization. We had conducted this study by
reading various articles, journals, books and web pages. We searched several
databases like IEEE explore, Inspec, Scopus, etc. to identify honest and quality
related articles.

31
Chapter 4. Research Methodology and Test-bed 32

Figure 4.1: Experiment setup

4.1.2 Stage 2
In this phase we had conducted experiments to evaluate the disk performance
of virtual machine during the live migration over WAN or internet on laboratory
test-bed using various disk sharing solutions. There are three ways to achieve the
aim of our proposal i.e. mathematical modelling, real-time implementation and
simulation [18]. Mathematical modelling is very complex and not fully accurate
due to the simplifications done in modelling process while simulation is a simpli-
fied model of real time environment created to simulate real conditions virtually.
Simulations are used when the real time environment is unavailable. Different
simulators programs may give varying results. For example OPNET and Net-
work Simulator (NS-2), the popular network simulators gives different outputs
even if they were given same input parameters. So, users should be careful while
selecting the simulation software. To analyze the disk performance of a virtual
machine we required limited resources, for example 4 computers and a switch.
This made us to choose laboratory testbed to conduct our experiments. In cases
like ours, the laboratory experiment brings us closer to real-time environment
and gives advantage over simulation and mathematical analysis. Figure 4.1 is the
simplified experimental setup we used to collect the disk performance data.

4.1.3 Stage 3
In this stage we analyzed the performance of the three disk storage solutions
with the help of the observed performance metrics.
Chapter 4. Research Methodology and Test-bed 33

4.2 Experiment Test-beds

The basic design of our laboratory test-bed is shown in the figure 4.1. We used
two physical nodes at site 1 and site 2. Each node hosts multiple virtual machines.
We used Xen hypervisor to create and migrate virtual machines. Xen hypervisor
was described in chapter 3. We initially planned to migrate virtual machine along
with network state over WAN using OAI’s PMIPv6 implementation over Linux,
but couldn’t do it due to some bugs in the PMIPv6’s authentication modules.
From out literature study described in related work section in chapter 2, many
researchers concluded that disk migrations will increase the migration time and
downtime of the virtual machine. Therefore, we used two different shared storage
solutions, iSCSI and NFS, to store virtual machine’s disk state there by elimi-
nating disk state migration. However, we also used DRBD [6], a disk replication
approach that mirror the disks at both the sites. We did this to analyze the virtual
machine’s disk performance while using network-based and replicated-based stor-
age solutions. We introduced various network delays using Linux based network
emulator NetEm [28] to emulate a WAN in our test-bed.
In our experiment we simulated a scenario in which VM 3 at site 1 (shown
in Figure 4.1) is live migrated to physical host at site 2 and used Iometer [26],
a disk IO analyzing tool to analyze the disk performance of virtual machines
during the live migration. We identified the performance metrics [19] and used
them to estimate the performance of virtual machines during live migration. This
was done for each of the storage solutions mentioned above, i.e., iSCSI, NFS and
DRBD.

4.3 Hardware configuration

The hardware configuration of our test-bed is shown below.

• Site 1 and Site 2 runs on HP dx5150 Small Form Factor with AMD Athlon(tm)
64 Processor 3200+, 1GB of RAM, 40GB Samsung SP0411C HDD and a
gigabit Ethernet adapter. These machines have Ubuntu 12.04 LTS Desktop
64 bit operating system.
• NFS and iSCSI servers runs on HP dx5150 Small Form Factor with AMD
Athlon(tm) 64 Processor 3200+, 2GB of RAM, 80GB Seagate HDD and a
gigabit Ethernet adapter. This machine has Ubuntu 13.10 64 bit operating
system.
• Virtual machine was allotted with 256MB of RAM, 4GB storage and 1
virtual CPU. This virtual machine has Ubuntu 12.04 LTS desktop 32-bit
operating system
Chapter 4. Research Methodology and Test-bed 34

• Client run on Lenovo Y480 notebook with 8GB of RAM, Intel core i7 CPU
and gigabit Ethernet NIC. This machine has Windows 8.1 64 bit operating
system.
• 16.0 X 10/100 D-Link 1016 DES unmanaged switch is used to connect Site
1, Site2, Storage server and client.

4.4 Benchmarking and other tools

We used various tools or software like Iometer, NetEm, Wireshark, etc. in
our experiments to collect, process and analyze our test results. In this section
we gave a brief description of these software or tools.

4.4.1 Iometer
Iometer [26] formerly known as “Galileo” is a very popular I/O subsystem
benchmarking tool developed by Intel Corporation. It was first released on 17th
February 1998. It can be used as both workload generator (Dynamo) and mea-
surement tool (Iometer) [26] on a single or networked systems.
The uses of Iometer are described below:

• It measures the performance of disk and network controllers.

• Measures the bandwidth and latency capabilities of buses.
• Measures the Network Throughput to attached drivers.
• It measures the performance of shared bus.
• It measures the system level network performance.
• It measures the system level hard drive performance.

Iometer is a collection of two modules, Iometer and Dynamo. Iometer is a

graphical interface that initializes the workload generator (Dynamo). Dynamo
generated asynchronous read and write I/O operations. Iometer is used to con-
figure test, set parameters, start test, collecting results into output file and stop
tests. Iometer runs on both the windows and Linux machines [30]. Intel dis-
continued from Iometer development in 2001 and handed over it to Open Source
Development Lab under Intel Open Source License. There are many other disk
IO benchmarking tools like Iorate, Iozone, Jetstress, Orion tool, etc. IOmeter
and Iorate are the two widely used disk I/O benchmarking tools. They both ana-
lyze the disk performance using the performance metrics like total IO operations,
MBps, IOps and I/O response time. Iometer has good graphical user interface
when compared to Iorate which uses command line interface and Iometer is simple
to configure and use.
Chapter 4. Research Methodology and Test-bed 35

4.4.2 NetEm
Network emulator, in short NetEm [28] is a Linux kernel based program
that we used to emulate the properties of WAN over LAN. NetEm introduces
packet delay, packet loss, packet duplication and packet re-ordering. NetEm is
enables in all Linux distributions. It has two components, Command-line utility
and Kernel module [29]. The command line utility uses netlink socket interface
to communicate with kernel module. The NetEm command used to introduce
network delay is described in appendix.

4.4.3 Wireshark
Wireshark is an open source network packet analyzer which is used for cap-
ture, analyses of network packets. Wireshark is available on Linux, OS X, Win-
dows and many other operating systems. Some important purposes of wireshark
[31] are troubleshooting, examining network problems, learning network protocol
internals, etc.

4.5 Experiment Scenarios

We had three scenarios in our experiments. We implemented our laboratory
test-bed which we described in the above section using three storage solutions i.e.
NFS, iSCSI and DRBD. We emulated WAN using three types of RRT delays i.e.,
Inter-Europe (30ms), Trans-Atlantic (90ms) and Trans-Pacific (160ms) [49].

4.5.1 Scenario 1
In this scenarios we conducted IO tests to analyze the disk performance of
virtual machine during the live migration over WAN using iSCSI shared storage.
In this scenario we have four computers in which two computers act as physical
hosts hosting virtual machines1 . These two physical hosts also act as iSCSI
initiators (iSCSI clients). One computer act as iSCSI target (server) and one
computer is used for collecting IO results using Iometer. This scenario is shown
in figure 4.2. Here we used NetEm to emulate WAN by introducing network delays
on the interfaces of the two physical hosts and iSCSI target. For example, we
introduced 15ms delay on the three interfaces so that the network delay between
the two physical hosts is 30ms and the delay between physical host and iSCSI
target is also 30ms.
1
The disk size of our virtual machines is 4GB
Chapter 4. Research Methodology and Test-bed 36

Figure 4.2: Scenario 1 and 2

4.5.2 Scenario 2
In this scenarios we conducted IO tests to analyze the disk performance of
virtual machine during the live migration over WAN using NFS shared storage.
In this scenario we have four computers in which two computers act as physical
hosts hosting virtual machines. These two physical hosts also act as NFS clients.
One computer act as NFS server and one computer is used for collecting IO
results using Iometer. We followed the same procedure described in Scenario 1
to introduce network delay.

Figure 4.3: Scenario 3

Chapter 4. Research Methodology and Test-bed 37

4.5.3 Scenario 3
In this scenarios we conducted IO tests to analyze the disk performance of
virtual machine during the live migration over WAN using DRBD storage. We
had three computers in this scenario, in which two computers act as physical
nodes hosting virtual machines. Here when the two physical hosts are booted,
the DRBD checks the status of the disks at both the sites. If the two disks are
up-to-date (synchronized) the connection is established between the two hosts. If
any of the two disks is not up-to-date, it is synchronized from the last updated
disk and the connection is established. The rate of synchronization is described in
chapter 3.2.5. When the connection is established between the two sites DRBD
asynchronous protocol C is executed on these two physical hosts to create a cluster
with the two sites. If a virtual machine makes changes to disk on site 1 the changes
are replicated to the disk on site 2. One computer is used for collecting IO results
using Iometer. We used NetEm to introduce delay on both hosts. We introduced
equal amount of delay on the interfaces of both sites to achieve required delay.
For example we introduced 45ms delay on both sites for 90ms delay.

4.6 Performance Metrics

In this section we discussed the performance metrics we used to analyze the
performance of virtual machine’s disk drives. Throughput and response time are
some of the performance metrics used to analyze the disk I/O performance of
virtual machines. Throughput is the rate at which the data is fetched from the
disk drive. It can be measured in two ways i.e., number of inputs and outputs
(I/O) accesses per second and volume of data read or written from the disk per
second in megabytes (MB). Response time is the time between the initiation and
completion of the I/O operation. The I/O system should be reliable with few or
no errors as too many errors may threaten the integrity of data stored on disk.
In this thesis we used the following performance metrics using Iometer
and Wireshark to analyze the disk performance of the virtual machine.

4.6.1 IOps performance

IOps performance or total IOps metric is defined as the average number of
total input and output operations (both read and write) operated per second.
It reflects the overall capacity of the disk in handling the input and output op-
erations. The disk storage solution with high total IOps is superior to other
solutions. We measured this metric using Iometer.
Chapter 4. Research Methodology and Test-bed 38

4.6.2 I/O Throughput

I/O throughput or total MBps metric is defined as the average number of
Megabytes read and written per second. It is the total rate of data in bytes
received and sent by the disk. The disk storage solution with the highest total
MBps is superior to the other solutions. We measured this metric using Iometer.

4.6.3 Maximum I/O Response Time

Maximum I/O response time is defined as the Maximum time between initia-
tion and completion of an I/O operations. This is measured in milliseconds (ms).
This is a crucial metric during the live migration of virtual machine. Maximum
response time is directly related to the downtime of the virtual machine. The
solution with less downtime is better than other solutions. We measured this
metric using Iometer.

4.6.4 Network Load

Network Load is the bandwidth consumed by various disk storage solutions
for the operation on the virtual machine. This metric will be helpful to analyze
the bandwidth consumption of various disk storage solutions. As we are migrating
our virtual machine over low latency WAN, the solution consuming less network
for operation is superior to other alternatives. For DRBD we used Wireshark on
interface of site 1 to capture and analyze the disk related network packets. We
used wireshark on the interface of the storage server for iSCSI and NFS storage
solutions.

4.7 Delay Characteristics of WAN

WAN is an interconnection of different networks which are separated by geo-
graphical locations. This network of networks may have millions of devices which
are interconnected using different physical layer technologies like wireless, cop-
per wire, optical fiber, etc. The data transmitted from source device reaches
the destination device which is at other physical location passing through var-
ious intermediate devices called boarder gateway routers with some time delay.
This time delay for the data to reach destination device from the source device is
called as End-to-End2 (E2E) delay. The overall E2E day over a WAN depends on
various factors like distance between two end devices, number of interconnecting
devices, the type of underlying physical layer technology used, etc.
2
E2E delay can be calculated using Round Trip Delay and One-Way Delay [59].
Chapter 4. Research Methodology and Test-bed 39

According to [58], the E2E delay components are divided into four categories.
They are processing delays, transmission delays, propagation delays and queuing
delays. The processing delay is added by the intermediate devices which are in
the path between the two end points. It is the time taken by the intermediate
devices to process the data packet and retransmit it to next node or end point.
The processing delay is independent of network load but it depends on the node
hardware, processing or computing efficiency and the protocol stack used. The
processing delay time is a stochastic random variable as it is different for each
individual packet. Transmission delay is the time taken to transmit a packet
over the communication link and it depends on the link speed of the hardware.
Propagation delay is the time taken for the packet to travel from one end to
another end over a communication channel. It depends on the type of channel
used between the endpoints. For example the propagation time taken on optical
fiber is different to the propagation time taken on satellite link. While coming to
the queuing delay, it is stochastic in nature and depends on the packet’s waiting
time in the buffer of the network devices before being processed or serviced.
From [58], most of the End to End delay distributions observed over WAN
are heavy-tailed distributions with gamma shape (84%). While the other ob-
served delay distributions are, gamma-like with Gaussian or triangle lob distri-
butions (6.3%), two gamma-like shaped distributions (2.8%), distribution with
many peeks (5%) and other distribution classes (1.5%). The network delays with
non-normal or heavy tail distributions degrade the network performance due to
their high probability spread in the values far from mean delay. Since, the heavy
tail distribution spread their probability curve to a higher delay time’s, a higher
waiting time is probable, this may increases the waiting times drastically at some
point of time. The impact of heavy-tailed service time destitution on the sys-
tem performance are discussed in [60]. Due to heavy-tailed distributed delays in
networks, the system takes more time to reach steady state. Since, the transient
nature of heavy tail distributions are high the waiting time distributions can’t
be accurately estimated. So the E2E heavy-tail delay distributions are difficult
to emulate. So, for simplicity most of the WAN emulators use Normal or Pois-
son distributions. These Normal / Poisson distributions decrease the probability
values exponentially on either side of mean, which makes it easy to emulate the
E2E delay characteristics of WAN. In our thesis we emulated the properties of
WAN using NetEm tool which uses normal distribution to introduce network de-
lay on the interface of the host. We introduced normally distributed intra-Europe
(30ms), trans-Atlantic (90ms) and trans-Pacific (160ms) delays (example shown
in appendix) to analyze the performance of virtual machine’s disk while migrating
over WAN. The network latencies we used in our thesis are observed from the IP
latency statistics published by Verizon Enterprise Solutions [49].
Chapter 5
Experiment Results

In this chapter we discuss the performance of iSCSI, NFS and DRBD storage
solutions under the influence different network delays. We had emulated a WAN
by introducing various network delays using NetEm. We had conducted a total
of twelve experiments for each of the three disk storage solutions by varying four
network delays. We ran every experiment for 40 iterations and calculated confi-
dence intervals for the mean values to deal with the stochastic variations caused
by I/O interrupts, kernel process delays, IP timers and measurement errors.
We used the selected performance metrics discussed in chapter 4 to analyze the
performance of iSCSI, NFS and DRBD by varying the network delays. First we
conducted experiment without introducing any network delay. This is a baseline
scenario that represents a campus network. Then we introduced 30ms RRT net-
work delay to emulate an inter-European WAN. Next we introduced 90ms RTT
network delay to emulate Trans-Atlantic WAN. Finally we conducted experiments
by introducing 160ms RTT network delay to emulate a trans-Pacific WAN.The
chosen values are the average latency guarantees, expressed as round-trip times
(RTTs), specified in the SLA for networks operated by Verizon Enterprise Solu-
tions [49].
We calculated 95 % CI [50] for the sample mean of the performance metrics
observed during different network delays. We compared the performance of the
three storage solutions with respect to performance metric and network delays.
The confidence interval is calculated using the formula shown in (1).
s
x̄ ± t(n−1)(1−α/2) √ (5.1)
n

Formula 5.1 Confidence Interval

In the above equation x is the sample mean calculated from the n (in our
case 40) iterations of the experiment and s is the standard deviation, which is the
square root of the variance. From equation (1) t(n−1)(1−α/2) , is the value of the
t-variate with n-1 degree of freedom and α denotes the level of significance.The
equation to calculate relative or marginal error is shown in equation (2). It is
the precision in the estimated mean value. It is calculated by multiplying the

40
Chapter 5. Experiment Results 41

confidence intervals half-length with 100 and dividing with mean x.

100 ∗ t( n − 1)(1 − α2 ) √Sn

(5.2)
x

Formula 5.2 Relative error

Iometer benchmarking tool was used to analyze the disk I/O performance of
various disk storage solutions during the live migration of VM over WAN. We
had conducted our experiments with 80-20% random chance of read-write and
transfer request size of 4K.This is to mimic a webserver, where 80% of users read
and 20% write. Here the I/O operations are randomly selected with 80% chance
of being read operation and the remaining 20% are write operations. Each I/O
operation reads or writes 4K bytes of data. We analyzed the performance using
Input Output per second (IOPS) performance, I/O Throughput during migration
(MBps), Maximum Response Time (MRT) and Network Load.

5.1 Input Output per Second Performance

We conducted four experiments to analyze the performance of iSCSI, NFS
and DRBD storage by varying network delays. We ran each experiment for 40
iterations. In each iteration we migrated virtual machine from source host to
destination host. We ran Iometer in virtual machine and recorded IOPS during
the migration. We calculated 95% confidence interval of the sample mean using
the above equation.

5.1.1 No delay (0.02 ms)

When the virtual machine was migrating over a campus network NFS handled
more number of IOps when compared to other storage solutions. NFS handles
98.82 I/O operations where as DRBD and iSCSI handles 85.34 and 78.69 I/O
operations respectively. NFS handles 25.59% more I/O operations when com-
pared to iSCSI and 15.79% more I/O operations when compared to DRBD. When
DRBD and iSCSI are compared, DRBD handles 8.45% more I/O operations than
iSCSI.
With 95% confidence we can say that the IOps of iSCSI lies in between 77.21
to 80.18 with standard deviation value of 6.07. The IOps of NFS lies in the range
of 95.65 to 101.99 with standard deviation of 12.19. The IOps of DRBD lies in
the range of 84.18 to 86.51 with standard deviation of 4.76. The relative error of
all the three methods lie below 5 percent. When observing all the three storage
solutions NFS outperformed others by handling more I/O operations. When
comparing block based storage solutions, DRBD out performed iSCSI.
Chapter 5. Experiment Results 42

Campus Network iSCSI NFS DRBD

Mean 78.69 98.82 85.35
Lower 95% Limit 77.20 95.65 84.18
Upper 95% Limit 80.18 101.99 86.51
Standard Deviation 6.08 12.19 4.76
Relative Error (%) 1.89 % 3.21 % 1.37 %

Table 5.1: IOps performance of different solutions on campus network

5.1.2 Inter-Europe (30ms Delay)

When a virtual machine was migrated over a network with Inter-Europe la-
tencies, DRBD outperformed the other two solutions. DRBD handles 53.27 I/O
operations whereas iSCSI and NFS handles 22.20 and 50.85 I/O operations re-
spectively. The performance DRBD is 139.95% higher than the performance of
iSCSI and only 4.75% higher than NFS. When file and block storages solutions are
compared, the performance difference is only 4.75%. There is no significant dif-
ference between block storage and file storage solutions. When comparing block
storage solutions DRBD out performed iSCSI by 139.95%.

Inter-Europe Network iSCSI NFS DRBD

Mean 22.21 50.85 53.28
Lower 95% Limit 21.68 49.26 52.38
Upper 95% Limit 22.73 52.45 54.18
Standard Deviation 2.13 6.51 3.47
Relative Error (%) 2.35 3.14 1.69

Table 5.2: IOps performance of different solutions on Inter-Europe Network

With 95% confidence we can say that I/Os handles by iSCSI lies in the range
of 21.68 to 22.72 with standard deviation of 2.12 and relative error of 2.34%. The
I/O handled by NFS lies in the range of 49.25 to 52.44 with standard deviation
of 6.50 and relative error of 3.13%. I/O’s handled by DRBD lies in the range of
52.37 to 54.17 with the standard deviation of 3.46 and the relative error of 1.69%.

5.1.3 Trans-Atlantic (90ms Delay)

When a virtual machine was migrated over a network with Trans-Atlantic
latency, NFS handled more IO operations per second when compared to iSCSI
and DRBD. NFS handled 37.43 IO operations whereas iSCSI and DRBD handled
15.39 and 26.24 I/O operations. The performance of NFS is 143.20% higher than
iSCSI and 42.64% higher than DRBD. When block and file storage solutions are
compared, file storage solutions outperformed the other solution.
Chapter 5. Experiment Results 43

Trans-Atlantic Network iSCSI NFS DRBD

Mean 15.39 37.43 26.24
Lower 95% Limit 14.78 36.42 25.84
Upper 95% Limit 16.012 38.4 26.65
Standard Deviation 2.37 4.15 1.56
Relative Error (%) 4.01 2.72 1.54

Table 5.3: IOps performance of different solutions on Trans-Atlantic Network

With 95% confidence we can say that the I/O operations of iSCSI operations
lies between 14.77 and 16.01 with standard deviation of 2.37 and relative error of
4.01%. The I/O operations handled by NFS lied in the range of 36.41 to 38.45
with standard deviation of 4.14 and relative error of 2.71%. The I/O operations
handled by DRBD are in range of 25.83 and 26.64 with standard deviation of
1.55 and relative error of 1.54%. When comparing block storage solutions DRBD
outperformed iSCSI. The I/O operations handles be DRBD are 70.50 % more
than iSCSI.

5.1.4 Trans-Pacific (160ms Delay)

When the virtual machine was migrated over a network with Trans-Pacific
latency, NFS outperformed DRBD and iSCSI storage solutions. NFS handled
24.47 I/O operations whereas iSCSI and DRBD handled 10.27 and 15.70 I/O
operations when migrating on trans-pacific network. NFS handles 138% more
I/O operations than iSCSI and 55.85% more I/O operations than DRBD.

Trans-Pacific Network iSCSI NFS DRBD

Mean 10.27 24.47 15.71
Lower 95% Limit 9.95 23.7 15.38
Upper 95% Limit 10.59 25.18 16.03
Standard Deviation 1.34 2.88 1.25
Relative Error (%) 3.16 2.89 2.06

Table 5.4: IOps performance of different solutions on Trans-Pacific Network

With 95% confidence we can say that the I/O operations of iSCSI lies in
the range of 9.94 to 10.59 with standard deviation of 1.33 and relative error of
3.16%. The I/O operations handled by NFS lies in the range of 23.76 to 25.17
with standard deviation of 2.88. The I/O operations handled by DRBD lies in
the range of 15.38 to 16.03 with standard deviation of 1.24 and relative error of
2.06%. When file and block storages solutions are compared, NFS has better
performance than the two block storage solutions. When block storage solutions
Chapter 5. Experiment Results 44

are compared, DRBD’s performance is 52.87% more than iSCSI.

In the below graph we can see the performance of the three storage solutions.
NFS outperformed iSCSI and DRBD under all the network delays except in inter-
Europe where DRBD outperformed NFS with a small margin. NFS shared storage
solution had taken advantage of its efficient caching mechanism where frequently
read data is stored in the local cache [32]. We had mimicked a webserver with 80%
read and 20% write requests. In NFS all the frequently read I/O operations are
stored in the local memory thereby improving the number of read operations. This
will reduce the network usage for read operations and allowing write operations
to use more network resources.

Figure 5.1: Graph: IOps Vs Network Delay

5.2 I/O Throughput during Migration

In this section we analyze the total megabytes of data read and written ev-
ery second from the virtual machine’s disk when migrated over a network. We
migrated virtual machine over emulated WAN running Iometer using iSCSI, NFS
and DRBD storage solutions. We conducted four experiments with varying net-
work RRT delays of 0.02ms, 30ms, 90ms and 160ms. We ran each experiment
for 40 iterations and calculated 95% confidence interval of the sample mean. The
solution showing highest MBps value is superior to the other systems.
Chapter 5. Experiment Results 45

5.2.1 Campus Network (0.02 ms)

When the virtual machine was migrated over campus network, NFS outper-
formed other two solutions. Iometer recorded 0.386 MB of data read and written
per second from NFS disk, whereas 0.311 MB and 0.334 MB of data is read or
written per second from iSCSI and DRBD disks. NFS outperformed iSCSI by
24.11% where as it outperformed DRBD by 15.56%. With 95% confidence interval
value of iSCSI lies in between 0.305 to 0.316 with standard deviation of 0.02 and
relative error of 1.7%. With 95% confidence, we can tell that the sample mean
of MBps from NFS lies in the range of 0.373 and 0.398 with standard deviation
of 0.04 and relative error of 3.21%. With 95% confidence the sample mean value
of MBps from DRBD lies in the range of 0.329 to 0.339 with standard deviation
of 0.01 and relative error of 1.46%. When block and file storage solutions are
compared, NFS outperformed block storage solutions and among block storage
solutions, DRBD out performed iSCSI with a small margin of 7.39%.

Campus Network iSCSI NFS DRBD

Mean 0.311 0.386 0.335
Lower 95% Limit 0.306 0.374 0.329
Upper 95% Limit 0.317 0.398 0.339
Standard Deviation 0.021 0.048 0.0188
Relative Error (%) 1.757 3.211 1.465

Table 5.5: I/O throughput for different solutions on Campus Network

5.2.2 Inter-Europe (30ms Delay)

Inter-Europe Network iSCSI NFS DRBD

Mean 0.086 0.198 0.208
Lower 95% Limit 0.084 0.191 0.205
Upper 95% Limit 0.088 0.205 0.212
Standard Deviation 0.008 0.026 0.014
Relative Error (%) 2.504 3.457 1.693

Table 5.6: I/O throughput for different solutions on Inter-Europe Network

When a virtual machine was migrated over Inter-Europe network using differ-
ent storage solutions, DRBD outperformed iSCSI and NFS. The average volume
of data read and written per second on DRBD disk is 0.208 MB. When coming
to iSCSI and NFS, 0.085 and 0.197 MB of data is read or written per second
on the disks. The performance of iSCSI is 144.70% less than the performance of
Chapter 5. Experiment Results 46

DRBD whereas the performance of NFS is only 5.58% lower. With 95% confi-
dence we can say that the performance of iSCSI lies in the range of 0.083 to 0.087
with standard deviation of 0.008 and relative error of 2.5%. The 95% confidence
interval values of NFS lies in between 0.190 to 0.204 with standard deviation of
0.026 and relative error of 3.45%. With 95 % confidence we can say that the
performance of DRBD lies in between 0.2045 to 0.211 with standard deviation
of 0.013 and relative error of 1.69. When block and file storage solutions are
compared, DRBD slightly outperformed NFS. When two block storage solutions
are compared, DRBD outperformed iSCSI by 142.28%.

5.2.3 Trans-Atlantic (90ms Delay)

When a virtual machine was migrated over WAN with Trans-Atlantic network
delay, NFS outperformed iSCSI and DRBD. Here 0.146 MB of data is read and
written per second on the NFS disk whereas 0.060 and 0.102 MB of data is read
and written per second on iSCSI and DRBD disks. The performance of NFS
is 143.33% greater than the performance of iSCSI, whereas it is 43.13% greater
than the performance of DRBD. The 95% confidence values along with standard
deviation and relative error of MBps read and written on the disks of iSCSI,
NFS and DRBD are shown in the table 5.7. The relative error of all the three
solutions are below 5%. When block and file storage solutions are compared, NFS
outperformed both iSCSI and DRBD. When compared both the block storage
solutions DRBD outperformed iSCSI by 70%.

Trans-Atlantic Network iSCSI NFS DRBD

Mean 0.061 0.147 0.103
Lower 95% Limit 0.058 0.142 0.101
Upper 95% Limit 0.063 0.151 0.104
Standard Deviation 0.009 0.016 0.006
Relative Error (%) 4.011 2.982 1.545

Table 5.7: I/O throughput for different solutions on Trand-Atlantic Network

5.2.4 Trans-Pacific (160ms Delay)

When a virtual machine was migrated over WAN with Trans-Pacific network
delay NFS outperformed iSCSI and DRBD. Here 0.095 MB of data is read and
written per second on the NFS disk whereas 0.04 and 0.061 MB of data is read
and written per second on iSCSI and DRBD disks. The performance of NFS is
137.5% greater than the performance of iSCSI, whereas it is 55.73% greater than
the performance of DRBD.
Chapter 5. Experiment Results 47

Trans-Pacific Network iSCSI NFS DRBD

Mean 0.041 0.095 0.061
Lower 95% Limit 0.039 0.092 0.060
Upper 95% Limit 0.042 0.098 0.063
Standard Deviation 0.006 0.011 0.005
Relative Error (%) 3.569 3.099 2.062

Table 5.8: I/O throughput for different solutions on Trans-Pacific network.

The 95% confidence values along with standard deviation and relative error
of MBPS read and written on the disks of iSCSI, NFS and DRBD are shown in
the table 5.8. The relative errors of iSCSI, NFS and DRBD are 3.56%, 3.09%
and 2.06% respectively. When block and file storage solutions are compared, NFS
outperformed both iSCSI and DRBD. When block storage solutions are compared
DRBD outperformed iSCSI by 52.5%.
From the below graph we can see that NFS outperformed the other storage
solutions when a virtual machine is migrated over campus, trans-Atlantic and
trans-Pacific network. In inter-Europe network DRBD outperformed NFS with a
small margin. NFS network storage solution made best use of its efficient caching
technique to handle more megabytes of data when compared to iSCSI and DRBD.

Figure 5.2: Graph: MBps Vs Network Delay

Chapter 5. Experiment Results 48

5.3 Maximum Response Time (MRT)

In this section we analyzed the disk I/O performance of virtual machine
during the live migration over emulated WAN using maximum response time
metric. MRT is the maximum time taken from the initiation to the completion
of an I/O operation. It is measured in milliseconds. The disk solution with
small maximum response time is considered as better one. The value of MRT is
directly proportional to the downtime of the virtual machine. We had conducted
a total of 12 experiments using three disk solutions and four network delays.
We ran each experiment for 40 iterations for redundancy. We calculated 95%
confidence interval for the sample mean value of MRT obtained for 40 iterations.
We calculated standard deviation and relative error for all the experiments.

5.3.1 No delay (0.02 ms)

When a virtual machine was migrated over campus network iSCSI outper-
formed NFS and DRBD. In iSCSI shared disk, the maximum response time be-
tween initiation and completion of an I/O operation is 472.13 milliseconds. NFS
and DRBD disk systems has maximum response time of 1969.35 milliseconds and
606.33 milliseconds. The performance of iSCSI is 317.12% greater than the perfor-
mance of NFS shared storage and 28.42% greater than the performance of DRBD.
We calculated the 95% confidence interval of the sample mean value of MRT from
40 iterations along with standard deviation and relative error. The relative error
values of iSCSI, NFS and DRBD are 2.19%, 8.52% and 2.92% respectively. When
block and file storage solutions are compared iSCSI outperformed NFS.

Campus Network iSCSI NFS DRBD

Mean (ms) 472.13 1969.35 606.33
Lower 95% Limit 461.79 1801.56 588.57
Upper 95% Limit 482.48 2137.15 624.09
Standard Deviation 39.76 645.12 68.27
Relative Error(%) 2.19 8.52 2.93

Table 5.9: MRT of different solutions on Campus network

5.3.2 Inter-Europe (30ms Delay)

When a virtual machine was migrated over Inter-Europe network DRBD
outperformed iSCSI and NFS. In DRBD, the maximum response time between
initiation and completion of an I/O operation is 668.02 milliseconds. NFS and
iSCSI storage systems recorded a maximum response time of 2262.71 milliseconds
Chapter 5. Experiment Results 49

and 2101.92 milliseconds. The performance of DRBD is 238.71% greater than the
performance of NFS shared storage and 214.64% greater than the performance
of iSCSI. The 95% confidence interval of the sample mean MRT value from 40
iterations along with standard deviation and relative error are shown in table
5.10. The relative error values of iSCSI, NFS and DRBD are 4.15%, 4.27% and
2.83% respectively. When block and file storage solutions are compared DRBD
outperformed NFS.

Inter-Europe Network iSCSI NFS DRBD

Mean (ms) 2101.92 2262.72 668.03
Lower 95% Limit 2014.51 2166.05 649.07
Upper 95% Limit 2189.33 2359.39 686.98
Standard Deviation 336.07 371.66 72.89
Relative Error(%) 4.16 4.27 2.84

Table 5.10: MRT of different solutions on inter-Europe network

5.3.3 Trans-Atlantic (90ms Delay)

When a virtual machine was migrated over Trans-Atlantic network, DRBD
outperformed iSCSI and NFS. In DRBD, the MRT between initiation and com-
pletion of an I/O operation is 1576.29 milliseconds. The iSCSI and NFS storage
solutions had a maximum response time of 2374.45 milliseconds and 2457.83 mil-
liseconds respectively. The performance of DRBD is 50.63% greater than the
performance of iSCSI and 55.93% greater than the performance of NFS. The
95% confidence interval values of the sample mean MRT calculated from 40 iter-
ations along with standard deviation and relative error are shown in table 5.11.
The relative error values of iSCSI, NFS and DRBD are 3.48%, 3.18% and 4.70%
respectively. When block and file storage solutions are compared, DRBD outper-
formed NFS.

Trans-Atlantic Network iSCSI NFS DRBD

Mean (ms) 2374.46 2457.84 1576.29
Lower 95% Limit 2291.78 2379.67 1502.11
Upper 95% Limit 2457.13 2536.01 1650.48
Standard Deviation 317.86 300.52 285.22
Relative Error (%) 3.48 3.18 4.71

Table 5.11: MRT of different solutions on trans-Atlantic network

Chapter 5. Experiment Results 50

5.3.4 Trans-Pacific (160ms Delay)

When a virtual machine was migrated over Trans-Pacific network, DRBD
outperformed iSCSI and NFS. In DRBD, the MRT between initiation and com-
pletion of an I/O operation is 2431.30 milliseconds. The iSCSI and NFS storage
solutions has a maximum response time of 3272.01 milliseconds and 3500.16 mil-
liseconds. The performance of DRBD is 34.57% greater than the performance
of iSCSI and 43.96% greater than the performance of NFS. The 95% confidence
interval values of the sample mean MRT calculated from 40 iterations along with
standard deviation and relative error are shown in table 5.12. The relative error
values of iSCSI, NFS and DRBD are 2.89%, 3.14% and 3.97% respectively. When
block and file storage solutions are compared, DRBD outperformed NFS.

Trans-Pacific Network iSCSI NFS DRBD

Mean (ms) 3272.02 3500.17 2431.31
Lower 95% Limit 3177.15 3390.16 2334.54
Upper 95% Limit 3366.89 3610.17 2528.07
Standard Deviation 364.74 422.93 372.03
Relative Error (%) 2.89 3.14 3.98 %

Table 5.12: MRT of different solutions on trans-Pacific network

Figure 5.3: Graph: MRT vs Network delay

The maximum response time of various storage solutions with respective to

varying network delay are displayed in the below graph. When the virtual ma-
Chapter 5. Experiment Results 51

chine is migrating over campus network, the iSCSI storage has least maximum
response time, while DRBD outperformed other storage solutions when virtual
machine is migrated over inter-Europe, trans-Atlantic and trans-Pacific networks.
Among the three storage solutions iSCSI and NFS are shared storage solutions
where the disk is located at another location. The I/O operations generated from
the virtual machine are encapsulated and forwarded to server using IP packets.
But DRBD is a disk replication solution where the disk resides at the client and
the I/O changes mode on the disk are replicated to clustered disk using IP pack-
ets. When a VM is migrated over WAN, the MRT value observed in iSCSI and
NFS storages system is higher when compared to DRBD because iSCSI and NFS
have a storage server which is located at a remote location and the time taken by
the I/O packets to reach the server adds up to the observed I/O response time.
When coming to DRBD, there is no third device between the two clustered disks
to add additional travel and process times. This will also reduce the network con-
sumption when compared to iSCSI and NFS. So the DRBD outperforms iSCSI
and NFS storage solutions when migrating over WAN.

5.4 Network Load

Network Load is the bandwidth consumed by various disk storage solutions
for the I/O operation of the virtual machine over WAN. Initially we tried to
analyze the network overhead utilized, but unfortunately there is no provision or
option to detect the actual volume of data generated by Iometer during the I/O
tests. We tried to calculate manually by multiplying IOps, 4k (work load of each
I/O) and 60 (test time in seconds) which gave possible data accessed by Iometer.
In the above calculated value 20% is written and 80% is read. All the 20% writes
will use network to reach server or peer disk. The problem is with 80% reads. In
DRBD these 80% reads doesn’t use network as the disk is available locally. In
NFS and iSCSI not all read I/O’s are fetched from server as some of the frequent
reads are readily available in local cache. So, it’s difficult to calculate network
overhead. This made us to analyze network load instead of network overhead.
We used Wireshark to capture and analyze the network bandwidth utilized by
iSCSI, NFS and DRBD storage solutions. We ran twenty I/O tests on iSCSI, NFS
and DRBS storage solutions. The iSCSI, NFS and DRBD storage solution uti-
lized 65.81, 75.47 and 37.77 Mega Bytes of network bandwidth during the entire
experiment run time. DRBD consumed less network resources when compared to
iSCSI and NFS. The 37.77 MB data in DRBD includes only the replication data
but not the initial disk synchronization data. If both the disks are up-to-date
there will be no synchronization phase. If one of the disk is new or not up-to-date
DRBD initiates synchronization phase. The synchronization time depends on the
volume of data required to synchronize and the rate of synchronization. It can be
Chapter 5. Experiment Results 52

estimated using the formula described in chapter 3.2.5. In this experiment both
the DRBD nodes are up-to-date which eliminated the initial disk synchronization
phase. The iSCSI consumed 74.23% more network resources than DRBD whereas
NFS consumed 99.81% more network resources. DRBD uses less network band-
width than other two solutions because, in DRBD only the changes made by the
write I/O operations are replicated to peer disk over the network. Whereas in
iSCSI and NFS both write I/O operations and read1 I/O operations uses network
to reach the storage server. So, using DRBD along with Xen when migrating
virtual machine over low latency WAN reduces the network bandwidth.

Figure 5.4: Graph: Network Bandwidth vs Storage Solutions

1
Actually not all read operations. This is because some read I/O‘s can access cached data.
Chapter 6
Conclusion and Future work

6.1 Conclusion
In this thesis we had analyzed the disk performance of virtual machines during
live migration over networks with different varying network latencies and using
different storage solutions. We had used iSCSI, NFS and DRBD storage solutions
to store the disk of the virtual machine and analyzed disk I/O performance of
a simulated webserver with 80% reads and 20% writes using Iometer disk ana-
lyzer tool. We had conducted our experiments by varying three network delays
to emulate WAN. When the disk performance of virtual machine using different
storage solutions is compared with varying network delays, the experiments con-
ducted with NFS storage solution handles more I/O operations and fetched more
megabytes of data when compared to iSCSI and DRBD storage solutions. The
performance of DRBD is slightly inferior to the performance of NFS and clearly
out performed iSCSI storage solutions. In NFS, it’s efficient caching mechanism
caches the most frequently read operations and reduces the need of accessing the
disk at another location over a network. This caching technique could be the
main reason for the better performance of NFS storage solution when the VM
hosts a server handling more read operations.
In case of maximum response time (MRT) metric, DRBD outperformed iSCSI
and NFS storage solutions. This performance metric is the total time taken
between the initiation and completion of an I/O operation. The iSCSI and NFS
are shared storage solutions where the virtual machine’s disk is stored at a remote
location and read and write operations are carried to the disk over IP protocol
using the underlying WAN. The time taken by the read and write operations to
reach the remote disk will add up to the I/O response time and thereby increasing
it. On the other hand, DRBD it is a disk replication technique where the virtual
machines disk is stored in the local disk and only disk changes are replicated
to other peer disk. In DRBD the I/O operations are read and written to the
local disk. The network overhead consumed by DRBD is smaller than that used
by iSCSI or DRBD as only disk modifications are replicated to other peer node.
Among all our performance metrics MRT is crucial because if it is high, the

53
Chapter 6. Conclusion and Future work 54

client accessing the services from the virtual machine may detect the performance
difference. This may lead to poor Quality of Experience (QoE). Consequently, we
recommend DRBD storage solution for storing the virtual machine’s disk during
live migration over WAN.
From our experiment results we can say that even though NFS handles more
I/O operations and throughput, from the IO tests with 80% read 20% write
(which is a typical workload for a webserver) it had recorded more MRT and
network bandwidths effecting the VM’s performance during the live migration.
Suppose if the VM hosts a server which need to handle more write operations
than a webserver (for example in the case of a gaming server) the performance
of NFS shared storage may be further decreased due to high MRT and network
bandwidth. From our results we can conclude that DRBD is best suitable for
storing disk of Virtual machines during the live migration of VM over WAN and
iSCSI could be the next best option.

6.2 Future Work

In this thesis we analyzed the disk I/O performance of a virtual machine
which mimics a webserver application with 80% read operations and 20% write
operations. Our analysis holds good for server applications which handles more
read and less write operations. Observations from this analysis may fail while
using server applications which handles more write operations than read opera-
tions. For example the disk performance of the web server is different from the
performance of gaming server as the later requires more write operations. As a
possible future work we think it could be a good idea to analyze the disk I/O
performance of a virtual machine with different block size / work load and read-
/write I/O patterns like 50% read & 50% write, 80% write & 20% read, 100%
read, 100% write, etc. We used Iometer to analyze the disk I/O and it can be
replaced by Iorate [27] and validate our experiments. Due to time considerations
we couldn’t investigate the time taken for DRBDs initial disk synchronization
and the network overhead of iSCSI, NFS and DRBD when the virtual host is
down. These network overheads and synchronization times can be investigated
in future.
Appendix
 Appendix

In this appendix we describe the procedure to install various tools and soft-
ware we used in our thesis. This appendix is divided in three sections. In section
one we described the procedure to install and configure NFS, iSCSI and DRBD.
The procedure to install and configure Xen hypervisor to support various storage
solutions is described in section two. Finally other tools like Iometer and NetEm
were discussed in section three. All the shell commands described in this ap-
pendix requires root privileges and we assume that all the devices are connected
to internet.

I. Section A
In this thesis we analysed the performance of iSCSI, NFS and DRBD storage
systems. Among these three solutions iSCSI and NFS are shared storage systems
whereas DRBD is replicated storage system.

1. Network File System (NFS)

NFS is a file based storage solution which is used to share files with
clients. In our experiment the scenario with NFS storage solution has three
computers. One computer acts as server hosting image files of the virtual
machine and two systems acts as client which access the shared image files
locally. All the three computers should be connected to the network. The
below table contains the IP address (examples) which are used to configure
NFS.

Machine Type IP Address

Server 192.168.137.7
Client 1 192.168.137.2
Client 2 192.168.137.3

Table 1: NFS address Details

56
Appendix 57

a. NFS Server:
The system should have root access to install and configure NFS
on Linux. We have to install the NFS software using apt-get install
command.

apt-get install nfs-kernel-server portmap

Next we have to choose or create a directories to share with clients.

We created a directory named vm1 and changed its owner ship to
nobody and nogroup.

mkdir /vm1
chow nobody:nogroup /vm1

Next we have to export the above directory to the other computers

by add the below lines at the bottom of the file /etc/exports

/vm1 192.168.137.0/24 (rw,sync)

In the above command rw indicates that the client can read and
write to the files in the shared directory and the command sync indi-
cates that the changes made by the client are sent synchronously to the
server. In synchronous mode, the write requests issued from the client
side are considered complete after the changes are confirmed on the
server side. In asynchronous mode the write event issued on server side
are considered complete after they reaches the server. When compared
to asynchronous mode, synchronous mode is safe and reliable.
We need to export the settings using the below command to make
them effective.

$ exportfs -a

We can view the shared directories using the below command.

$ exportfs -v

Now we have to install NFS on client machine and mount the

shared storage.
b. NFS Client:
First we have to install NFS client software in client computers
using apt-get install.

$ apt-get install nfs-common portmap

Appendix 58

Now we have to create directories on both the clients to mount

shared storage from the NFS server.

$ mkdir p /mnt/nfs/vm1

Now the shared directory should be mounted.

$ mkdir p /mnt/nfs/vm1

We can check the mounted directories using df h or mount com-

mands.

$ mount $ df -h

We can test the connectivity by creating a file from one of the client
and view it on other client and server. The virtual machine’s image
places in the server can be accesses by the virtual machine running at
client1 or client2.

2. Internet Small Computer System Interface (iSCSI)

Internet small computer system interface (iSCSI) is a block based file
storage system which is used to share storage over network. In our exper-
iment the scenario with iSCSI storage solution has three computers. One
computer acts as iSCSI target which has storage disk and the other two
computers act as iSCSI initiators. The virtual machines run on iSCSI ini-
tiator and the virtual machines disk is located at iSCSI target. We had
followed how-to-forge guide [34] to setup iSCSI storage.

Machine Type IP Address

iSCSI target 192.168.137.7
iSCSI initiator 1 192.168.137.2
iSCSI initiator 2 192.168.137.3

Table 2: iSCSI address Details

a. Creating Logical Volume Manager (LVM) partition on target

The Logical Volume Manager (LVM) program should be installed
on iSCSI target to create or manage logical volume. The LVM can be
installed during the installation of Ubuntu or after installing Ubuntu.
After installing Ubuntu LVM can be installed using the below shell
command.
Appendix 59

$apt-get install lvm2

After installing the lvm, the physical volume should be created to

create the volume group. This physical volume and volume group can
be created by using the below shell command.

$pvcreate /dev/<sda name>

for e.g. $pvcreate /dev/sda2
$vgcreate <vg name> <pv path>
for e.g. $vgcreate vg0 /dev/sda2

Next logical volume should be created to store the disks of virtual

machine.

$ lvcreate -n <volume name> -L <size in G or M> <volume group

name>

To create a logical volume of 50 gigabytes with volume name shared

in volume group vg0, the below shell command should be used.

$lvcreate -n shared -L 50G vg0

b. Installing and configuring iSCSI target

The iSCSI target provides shared storage for the connected initia-
tors. The virtual machines running on initiator will home their disks
on iSCSI target. We installed iSCSI target program on Ubuntu 13.04
operating system. The iSCSI target can be installed by entering the
below shell command in Ubuntu terminal.

$aptitude install iscsitarget iscsitarget-modules-‘uname -r‘

When the installation is finished, the iscsitarget file located at

/etc/default/ should be opened and the below command should be
added at the end of the file. Adding this line will enable iSCSI target
when the operating system boot.

ISCSITARGET ENABLE=true

Next, open the file “/etc/iet/ietd.conf ’ ’ and add the below lines in the
file by removing all the existing content.
Appendix 60

Target iqn.2014-04.com.example:shared.lun1
IncomingUser
OutgoingUser
Lun 0 Path=/dev/vg0/shared,Type=fileio
Alias LUN1

The target name should be unique and named according to the

name standard defined by iSCSI. Target iqn. <yyyy-mm>.<tld.doma-
in.some.host>[:<identifier>] is the iSCSI defined naming standard.
Username and password can be added for incoming users to provide
security. The full path of the shared storage should be specified in line
lun.
The ip address of the initiators should be add in the file “/etc/i-
et/initiators.allow” so that only those hosts will be connected to the
target. The entire network pool can be selected by adding the network
address to initiators.allow.

iqn.2014-04.com.example:shared.lun1 192.168.137.0/24

After saving the modified files the iSCSI target service should be
restarted.

$/etc/init.d/iscsitarget start

c. Installing and configuring iSCSI initiator

We are having two iSCSI initiators. So, the iSCSI initiator pro-
gram should be installed on two hosts.

$aptitude install open-iscsi

After installing initiator program on two hosts the “/etc/iscsi/isc-

sid.conf ” file should be opened and the below line should be added.
This modification will allow initiator to automatically connect to the
target.

$node.startup = automatic

Next, the iSCSI initiator service should be restarted.

/etc/init.d/open-iscsi restart

This iSCSI target can be discovered by using the below command.

Appendix 61

iscsiadm -m discovery -t st -p 192.168.137.7

The iSCSI target can be login to the discovered target by using

the below command.

iscsiadm -m node login

3. Distributed Replicated Block Device (DRBD)

Distributed replicated block device (DRBD) is a disk replication storage
solution where the changes made on one disk are replicated to other disk.
In our experiment the scenario with DRBD storage solution has only two
hosts. Before installing DRBD on the computers, Xen hypervisor should
be installed and the LVM based virtual machine should be created on one
of the host. The logical volumes created on first host should be created on
second host by using the procedure shown in section A.2.A. After that drbd
should be installed on both the hosts using the below shell command.

$apt-get install drbd8-utils

We have two logical disks for the virtual machine. In drbd the disk is
called as resource. Let us suppose they are drbdvm-disk and drbdvm-swap.
In our experiment we used DRBD protocol C (synchronous) to replicate
disks. We followed the procedure in [36] and [35] to configure the DRBD.
The drbd configuration file “/etc/drbd.conf should be edited to establish a
connection between the two hosts.
include ”/etc/drbd.d/global common.conf”;
include ”/etc/drbd.d/*.res”;

# VM disk image
resource drbdvm-disk {
protocol C;
net {
allow-two-primaries yes; #Enables dual primary mode
}
syncer {
rate 500M;
verify-alg sha1;
}
on site1-HP-dx5150-SFF-PE680AV {
address 192.168.137.146:7789; #Host1
Appendix 62

device /dev/drbd1;
disk /dev/vg0/drbdvm-disk; #disk location on host1
meta-disk /dev/vg0/meta[0];
}
on site2-HP-dx5150-SFF-PE680AV {
address 192.168.137.122:7789; # Host2
device /dev/drbd1;
disk /dev/vg0/drbdvm-disk; #Disk location on host2
meta-disk /dev/vg0/meta[0];
}
}

# VM swap image
resource drbdvm-swap {
protocol c;
net {
allow-two-primaries yes; #Enables dual primary mode
}
syncer {
rate 500M;
verify-alg sha1;
}

on site1-HP-dx5150-SFF-PE680AV {
address 192.168.137.146:7790; #Host1
device /dev/drbd2;
disk /dev/vg0/drbdvm-swap; #Disk location on host1
meta-disk /dev/vg0/meta[1];
}
on site2-HP-dx5150-SFF-PE680AV {
address 192.168.137.122:7790; #Host2
device /dev/drbd2;
disk /dev/vg0/drbdvm-swap; #Disk location on host2
meta-disk /dev/vg0/meta[1];
}
}

More details of the above configuration will be found in [34]. A new

logical disk named Meta should be created to store DRBD’s Meta data. It
can be either internal or external. Internal Meta disk should be used with
new disk. So we used external Meta disk. We created a Meta disk of 1GB
side on both hosts by using lvcreate command. After creating meta disks
Appendix 63

the DRBD resources should be create on both the locations by using the
below commands.

$drbdadm create-md test-disk

$drbdadm create-md test-swap

Next the DRBD should be started on both the locations.

$/etc/init.d/drbd start

The DRBD disk status can be seen by using the below commands. In
this stage disks on the both locations will be in inconsistent state.

$/etc/init.d/drbd status
$cat /proc/drbd

Next disks in both locations should be attached be using the below shell
command on both the nodes.

$drbdadm up all

Next the disks on node one should be made primary followed by second
node. This will synchronize both disks. This will take time depending on
network bandwidth and disk size. In LAN environment synchronization
may take upto 5 minutes.

$drbdadm – –overwrite-data-of-peer primary all

The synchronization time left and disk status can be seen by using the
below shell command.

$cat /proc/drbd
Appendix 64

II. Section B
In this section we described the procedure we followed to install and configure
XEN Hypervisor.

1. Xen Hypervisor installation

We followed the Xen proposed document [37] to install xen hypervisor.
We installed 64 bit Xen hypervisor on Ubuntu 12/04 LTS 64 bit operating
system. The g5 bit hypervisor even works on 32 bit host operating system
and allows both 64 bit and 32 bit guest operating systems. The below shell
command is used to install xen hypervisor.

$apt-get install xen-hypervisor-amd64

After installing Xen hypervisor modify the grub to make Xen default
while booting.

$sed -i ‘s/GRUB DEFAULT=.* +/GRUB DEFAULT=”Xen

4.1-amd64”/’ /etc/default/grub
$update-grub

Now make xm as default tool stack.

$sudo sed -i ’s/TOOLSTACK=.* + /TOOLSTACK=”xm”/’

/etc/default/xen

Then reboot the system so that the system boots with Xen hypervisor.
After rebooting you can verify xen by typing xm list which displayes dom0
as result.

$reboot
$xm list

2. Network Configuration
We used Linux Bridge to emulate the network link between physical
host and virtual machine. The Ubuntu 12.04 LTS desktop version will have
preinstalled bridge-utils package or it can be installed using the below shell
command.
Appendix 65

$apt-get install bridge-utils

Open the file “/etc/network/interfaces” and make the below changes to

it. We assume that you have dynamic host configuration protocol (DHCP)
network.
auto lo
iface lo inet loopback

auto xenbr0
iface xenbr0 inet dhcp # You can replace dhcp with manual
bridge ports eth0

auto eth0
iface eth0 inet manual

Restart the networking service to make the changes effective.

$/etc/init.d/networking restart
Or
$service networking restart

Note: You can also use synaptic package manager to install Xen hyper-
visor and network bridge [39].

3. Configuring Xen to allow VM migration

Open the file “/etc/xen/xend-config.sxp” and add the below lines at the
bottom of the file. This will enable virtual machine to migrate from one
host to another host over network [40].
xend-relocation-hosts-allow
xend-relocation-port 8002
xend-relocation-address
xend-relocation-server yes
Restart the xend demon to enable the changes.

$service xend restart

4. Creating virtual machines

Xen supports both hardware-assisted virtual machine (HVM) and par-
avirtualized virtual machine (PVM). We were restricted to PVM due to
Appendix 66

hardware limitations. We used xen-tools [38] package to create virtual ma-

chine automatically by using debootstrap. The below shell command is
used to install xen-tools.

$apt-get install xen-tools

In our experiments we used virtual machine with 32 bit Ubuntu 12.04

LTS operating system as it is the latest fully developed Ubuntu distribution.
We used 32 operating system as Iometer tool runs only on 32-bit machines.
The downloaded version of xen-tool package doesn’t include Ubuntu 12.04
LTS (precise). So, we downloaded xen-tools v4.3.1-1 and updated the cur-
rent version using dpkg tool.

$ wget http://ftp.se.debian.org/debian/pool/main/x/xen-tools/xen-
tools 4.3.1-1 all.deb
$dpkg i xen-tools 4.3.1-1 all.deb

After updating xen-tools, the virtual machine can be created by using

different storage solutions.

a. Installing VM on iSCSI shared storage

The virtual machine can be created by storing its disk on iSCSI
shared storage by using the below shell command.

xen-create-image –dhcp –memory 256M –hostname vmtest –lvm vg0

–bridge xenbr0 –pygrub –dist precise passwd arch i386

In the above shell command vmtest is the name of the virtual

machine, vg0 is the name of the iSCSI shared volume group. Arch i386
downloads 32 bit version of operating system. For more information on
xen-tools rood man pages [38]. After creating virtual machine open the
virtual machine’s configuration file “/etc/xen/vmtest.cfg” and remove
the character M after 256M next to memory. SSH the configuration
file to other host where the vm will be migrated. After modifying the
configuration file looks like below text.
bootloader = ’/usr/lib/xen-default/bin/pygrub’
vcpus = ’1’
memory = ’256’
root = ’/dev/xvda2 ro’
disk = [ ’phy:/dev/vgxen/vmtest-disk,xvda2,w’,
’phy:/dev/vgxen/vmtest-swap,xvda1,w’,]
Appendix 67

name = ’vmtest’
dhcp = ’dhcp’
vif = [ ’mac=00:16:3E:C5:24:63,bridge=xenbr0’ ]
on poweroff = ’destroy’
on reboot = ’restart’
on crash = ’restart’

b. Installing VM on DRBD shared storage

The process of creating virtual machine by storing its disk on
DRBD storage is similar to the above procedure. The only differ-
ence is, the volume group of shared storage is replaced by local volume
group. After creating virtual machine its configuration file should be
modified as shown below.
bootloader = ’/usr/lib/xen-default/bin/pygrub’
vcpus = ’1’
memory = ’256’
root = ’/dev/xvda2 ro’
disk = [ ’drbd:drbdvm-disk,xvda2,w’,
’drbd:drbdvm-swap,xvda1,w’,]
name = ’drbdvm’
dhcp = ’dhcp’
vif = [ ’mac=00:16:3E:A4:85:06,bridge=xenbr0’ ]
on poweroff = ’destroy’
on reboot = ’restart’
on crash = ’restart’
We can observe that the disk location is changed from phy to drbd.
Finally the configuration file should be send to peer node.
c. Installing VM on NFS shared storage
NFS is file share storage system which mounts files from the server
to connected clients. The virtual machines images are stored in the
server and shared to clients. The virtual machine in the clients use
the shared image files to boot. There are two ways to create virtual
machine using NFS. One is through creating VM using xen-tools and
the other is by converting the LVM disk into image file [41] and placing
them in shared directory. We used the later procedure to create virtual
machine. The below commands can be used to convert the LVM disks
to image files.

dd if=/dev/vg0/vmtest-disk of=/mnt/nfs/img/nfsvm-disk.img
dd if=/dev/vg0/vmtest-swap of=/mnt/nfs/img/nfsvm-swap.img
Appendix 68

The virtual machines configuration file should be modifies to boot

the virtual machine using the disk images stored in shared directory.
The modified configuration file look like
bootloader = ’/usr/lib/xen-4.3/bin/pygrub’
vcpus = ’1’
memory = ’256’
root = ’/dev/xvda2 ro’
disk = [ ’file:/mnt/nfs/img/nfsvm-disk.img,xvda2,w’,
’file:/mnt/nfs/img/nfsvm-swap.img,xvda1,w’, ]
name = ’nfsvm’
dhcp = ’dhcp’
vif = [ ’mac=00:16:3E:D1:16:1A,bridge=xenbr0’ ]
#NFS options
nfs server = ’192.168.137.112’
nfs root = ’/mnt/nfs/img/’
root = ’/dev/nfs’
on poweroff = ’destroy’
on reboot = ’restart’
on crash = ’restart’

III. Section C
In this section we described the procedure to download and configure miscel-
laneous tools like Iometer and NetEm.

1. Installing and Configuring Iometer

Iometer is a popular I/O benchmarking tool used to analyse the disk
performance. It has two components namely Iometer and dynamo. Iometer
is the GUI which displays the test results generated by dynamo. GUI is
not developed for Linux. So, a connection should be established between
dynamo running in the Linux computer and a windows computer running
Iometer.exe. The Iometer program should be downloaded from the internet
[30] using wget.

$wget http://prdownloads.sourceforge.net/iometer/iometer-
2006 07 27.linux.i386-bin.tgz

The downloaded compressed file should be extracted.

$gunzip iometer-2006 07 27.linux.i386-bin.tgz

$tar -xvf iometer-2006 07 27.linux.i386-bin.tar
Appendix 69

Go to src directory to access dynamo file

$cd iometer-2006 07 27.linux.i386-bin

$cd src

Use the below command to connect the dynamo to GUI running in

windows

$ ./dynamo -i windows ip -m Linux ip

2. Configuring NetEm
NetEm is a Linux based tool used to emulate the properties of WAN over
LAN by introducing network delay. Since the network delay is not uniform
in WAN, the delay variation can be normally distributed using NetEm. The
below shell command is used to introduce the mean trans-Atlantic delay of
90ms with a variation upto 10ms which is normally distributed. Users
can also model their own distributions my making changes to the iproute2
compilation code which is located at “usr/lib/tc”.

$tc qdisc change dev xenbr0 root netem delay 90ms 10ms distribution
normal
References
 References

1. "Understanding Full Virtualization, Paravirtualization, and Hardware Assist", White

Paper, VMware. Palo Alto, CA, USA, 2007.
2. Michael van Kleij and Attilla de Groot, "Virtual Machine WAN Migration with
DRBD", OS3 report, September 2011. [Online] Available:
https://www.os3.nl/_media/2008-2009/students/attilla_de_groot/virt_migration.pdf
[Accessed: 12/02/2014].
3. Thomas Burger, "The Advantages of Using Virtualization Technology in the
Enterprise", Intel Developer Zone, May. 2012. [Online]. Available:
http://software.intel.com/en-us/articles/the-advantages-of-using-virtualization-
technology-in-the-enterprise. [Accessed: 02/12/2013]
4. "Hardware-assisted virtualization", [online]. Available:
http://en.wikipedia.org/wiki/Hardware-assisted_virtualization. [Accessed: 01/12/2013]
5. Travostino F, Daspit P, Gommans L, Jog C, de Laat C, Mambretti J, Monga I, van
Oudenaarde B, Raghunath S, Wang P.Y, "Seamless live migration of virtual machines
over the MAN/WAN", Future Generation Computer Systems, v 22, n 8, p 901-7, Oct.
2006.
6. Philipp Reisner, Lars Ellenberg, "DRBD v8: Replicated Storage with Shared Disk
Semantics", August 2007, [online]. Available:
ww.drbd.org/fileadmin/drbd/publications/drbd8.pdf [Accessed: 02/12/2013]
7. Andrew J. Younge, Robert Henschel, James T. Brown, Gregor von Laszewski, Judy
Qiu, Geoffrey C. Fox, "Analysis of Virtualization Technologies for High Performance
Computing Environments", IEEE 4th International Conference on Cloud Computing,
2011.
8. Ian Pratt, Keir Fraser, Steven Hand, Christian Limpach, Andrew Warfield, "Xen 3.0
and the Art of Virtualization", 2005. [Online]. Available:
https://www.kernel.org/doc/ols/2005/ols2005v2-pages-73-86.pdf [Accessed:
04/12/2013]
9. Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen, Eric Jul, Christian
Limpach, Ian Pratt, Andrew Warfield, "Live migration of virtual machines",
Proceedings of the 2nd conference on Symposium on Networked Systems Design &
Implementation - Volume 2, Pages 273-286.
10. Mayank Mishra, Anwesha Das, Purushottam Kulkarni, and Anirudha Sahoo, "Dynamic
resource management using virtual machine migrations", IEEE Communications
Magazine, v 50, n 9, 34-40, Sept. 2012.
11. Timothy Wood, Prashant Shenoy, K.K. Ramakrishnan, Jacobus Van der Merwe,
"CloudNet: Dynamic Pooling of Cloud Resources by Live WAN Migration of Virtual
Machines", SIGPLAN Notices, v 46, n 7, p 121-32, July 2011.

71
12. Yingwei Luo, Binbin Zhang; Xiaolin Wang; Zhenlin Wang; Yifeng Sun; Haogang
Chen, "Live and incremental whole-system migration of virtual machines using block-
bitmap", 2008 IEEE International Conference on Cluster Computing (CLUSTER), p
99-106, 2008.
13. Kazushi Takahashi, Koichi Sasada, Takahiro Hirofuchi, "A Fast Virtual Machine
Storage Migration Technique Using Data Deduplication", The Third International
Conference on Cloud Computing, GRIDs and Virtualization, 2012.
14. Eric Harney, Sebastien Goasguen, Jim Martin, Mike Murphy, Mike Westall, "The
efficacy of live virtual machine migrations over the internet", Proceedings of the 2nd
international workshop on Virtualization technology in distributed computing, 2007.
15. S. Kassahun, A. Demessie, and D. Ilie, "A PMIPv6 Approach to Maintain Network
Connectivity During VM Live Migration Over the Internet", in Proceedings of IEEE
CloudNet, Luxembourg, October 2014.
16. Mattias Forsman, Andreas Glad, "Automated Live Migration of Virtual Machines",
M.S.Thesis, School of Computing, Blekinge Institute of Technology, Karlskrona,
Sweden, 2013.
17. "Applied Research", [online]. Available:
http://en.wikipedia.org/wiki/Applied_research, [Accessed: 07/12/2013].
18. Guanhua Yan, Yuehui Jin, Ilulu Ma, Shiduan Cheng, Jian Ma, "AN EFFICIENT
METHOD TO SIMULALATE COMMUNICATION NETWORKS", International
Conference on Communications, June 2000, pp. 192-194.
19. Sherif Akoush, Ripduman Sohan, Andrew Rice, Andrew W. Moore and Andy Hopper,
“Predicting the Performance of Virtual Machine Migration”, IEEE International
Symposium on Modeling, Analysis & Simulation of Computer and Telecommunication
Systems (MASCOTS), pp. 37-46, Aug 2010.
20. “Xen Project Software Overview”, [Online]. Available:
http://www.wiki.xen.org/wiki/Xen_Project_Software_Overview, [Accessed:
07/12/2013].
21. Timothy Wood, Prashant Shenoy, Arun Venkataramani, and Mazin Yousif, "Black-box
and gray-box strategies for virtual machine migration", 4th USENIX proceedings,
conference on Networked systems design & implementation, April 2007, pp. 17-17.
22. Sherif Akoush, Ripduman Sohan, Bogdan Roman, Andrew Rice and Andy Hopper
, "Activity Based Sector Synchronisation: Efficient Transfer of Disk-State for WAN
Live Migration", IEEE 19th International Symposium on Modeling, Analysis &
Simulation of Computer and Telecommunication Systems (MASCOTS), Singapore,
July 2011, pp 22-31.
23. Robert Bradford, Evangelos Kotsovinos, Anja Feldmann and Harald Schioberg, "Live
wide-area migration of virtual machines including local persistent state", Proceedings
of the 3rd international conference on Virtual execution environments, June 2006, pp.
169-179.

72
24. D. Johnson, C. Perkins, and J. Arkko, “Mobility Support in IPv6,” RFC 3775, Jun.
2004.
25. L. Dunbar and B. Sarikaya, “Virtual Machine Mobility Protocol Using Distributed
Proxy Mobile IPv6,” IETF Internet draft, Feb. 2013, [Online]. Available:
http://tools.ietf.org/html/draft-sarikaya-nvo3-vmm-dmm-pmip-00/ [Accessed:
02/01/2014]
26. Wayne Allen, Ming Zhang, Vedran Degoricija and Daniel Scheibli, “Iometer user
guide”, Iometer project, September 2004.
27. Vince Westin, “User guide: iorate Open Systems I/O Driver and Measurement Tool”,
EMC corporation, 2005.
28. Linux foundation, “NetEm”, November 2009, [Online].
http://www.linuxfoundation.org/collaborate/workgroups/networking/netem,
[Accessed: 12/04/2014]
29. Pradyumna Dash, "Bandwidth Throttling with NetEM Network Emulation", June 2012,
[Online]. http://www.opensourceforu.com/2012/06/bandwidth-throttling-netem-
network-emulation/, [Accessed: 12/04/2014]
30. Greg Porter, Howto:Iometer, November, 2010, [Online].
http://greg.porter.name/wiki/HowTo:iometer [Accessed: 03/12/2013]
31. Ulf Lamping, Richard Sharpe and Ed Warnicke, "Wireshark User's Guide", [Online].
https://www.wireshark.org/download/docs/user-guide-a4.pdf, [Accessed: 30/06/2014].
32. George Crump, "What is NFS Caching?", June 2011, [Online] http://www.storage-
switzerland.com/Articles/Entries/2011/6/22_What_is_NFS_Caching.html [Accessed:
19/07/2014]
33. Prabhakar Chaganti, “Xen virtualization: A practical handbook”, 1st edition, Packt
Publishing, Birmingham, UK, December 2007.
34. Falko, "Xen Live Migration Of An LVM-Based Virtual Machine With iSCSI On
Debian Lenny", April 2009, [Online]. http://www.howtoforge.com/xen-live-migration-
of-an-lvm-based-virtual-machine-with-iscsi-on-debian-lenny/ [Accessed: 06/02/2014]
35. Keith Herron, "Live Migration and Synchronous Replicated Storage with Xen, DRBD
and LVM", April, 2010. [Online]. http://backdrift.org/live-migration-and-synchronous-
replicated-storage-with-xen-drbd-and-lvm/ [Accessed: 12/01/2014].
36. Xen Cluster: How to, February, 201, [Online] http://www.asplund.nu/xencluster/xen-
cluster-howto.html, [Accessed: 13/04/2014].
37. Ubuntu Xen Community, "XenProposed", September, 2012, [Online].
https://help.ubuntu.com/community/XenProposed, [Accessed: 23/12/2013].
38. Ian Jackson, "xen-tools – a straightforward VM provisioning/installation tool", August
2012, [Online]. http://blog.xen.org/index.php/2012/08/31/xen-tools-a-straightforward-
vm-provisioninginstallation-tool/ [Accessed: 21/01/2014]
39. Mehmet Batuhan Özcan, Gabriel Iro, “Paravirtualization Implementation In Ubuntu
With Xen Hypervisor”, B.Sc thesis, Dept. of Communication, Blekinge Institute of
Technology, Sweden, 2014.

73
40. Paul Virijevich, "Live migration of Xen domains", July, 2005, [Online]
http://archive09.linux.com/feature/55773/ [Accessed: 20/03/2014]
41. Falko, "Xen: How to Convert An Image-Based Guest To An LVM-Based Guest", April
2009, [Online], http://www.howtoforge.com/xen-how-to-convert-an-image-based-
guest-to-an-lvm-based-guest/, [Accessed: 20/04/2014]
42. Rodney Gedda, "5 open source virtualisation technologies to watch: Third-party
management tools growing around OS hypervisors", January 2011, [Online].
http://www.cio.com.au/article/373314/5_open_source_virtualisation_technologies_wa
tch/ [Accessed: 20/07/2014]
43. P. Reisner, F. Haas and L. Ellenberg, “The DRBD User’s Guide,” LINBIT Solutions
GmbH, 2011.
44. Peter Radkov, Li Yin, Pawan Goyal, Prasenjit Sarkar and Prashant Shenoy, "A
Performance Comparison of NFS and iSCSI for IP-Networked Storage", 3rd USENIX
Proceedings, Conference on File and Storage Technologies of USENIX Association,
2004, pp 101-114.
45. Ibrahim Yakti and Walid A. Salameh, "iSCSI (Internet Small Computer System
Interface) for your Startup", International Journal of Information and Communication
Technology Research, Vol 2, March 2012, pp. 294-303.
46. J. Satran, K. Meth, C. Sapuntzakis, M. Chadalapaka and E. Zeidner, " Internet Small
Computer Systems Interface (iSCSI)", RFC 3720, April 2004. [Online]
http://www.ietf.org/rfc/rfc3720.txt/ [Accessed: 23/07/2014]
47. White paper, "iSCSI Basics: A Practical Introduction", Mosaictec Solutions, 2006.
[Online] http://www.mosaictec.com/pdf-docs/whitepapers/iSCSI_Guide.pdf/
[Accessed: 09/06/2014]
48. S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler and D.
Noveck, "Network File System (NFS) version 4 Protocol", RFC number 3530, April
2003. [Online] https://www.ietf.org/rfc/rfc3530.txt/ [Accessed: 23/06/2014]
49. Verizon Enterprise Solutions, "IP Latency Statistics", [Online].
http://www.verizonenterprise.com/about/network/latency/ [Accessed: 16/04/2014]
50. R. Jain, "The Art of Computer Systems Performance Analysis- Techniques for
Experimental Design, Measurement, Simulation, and Modeling," Wiley- Interscience,
New York, NY, April 1991.
51. David Marshall, “Top 10 benefits of server virtualization”, Virtualization report,
[Online]. http://www.infoworld.com/article/2621446/server-virtualization/top-10-
benefits-of-server-virtualization.html [Accessed: 16/09/2014]
52. Mohamed Fawzi, "Virtualization and Protection Rings (Welcome to Ring -1) Part I",
[Online], http://fawzi.wordpress.com/2009/05/24/virtualization-and-protection-rings-
welcome-to-ring-1-part-i/ [Accessed: 12/09/2014]
53. "Protection ring", [Online] Available: http://en.wikipedia.org/wiki/Protection_ring
[Accessed: 11/09/2014]

74
54. Gregory Steulet, "Dolphin Express Interconnect", [online],
http://www.trivadis.com/uploads/tx_cabagdownloadarea/DolphinInterconnect_U5.pdf
[Accessed:16/09/2014]
55. Jose Renato Santos, G. Janakiraman, Yoshio Turner, Ian Pratt, "Bridging the gap
between software and hardware techniques for I/O virtualization", USENIX 2008
Annual Technical Conference on Annual Technical Conference, p.29-42, June 22-27,
2008.
56. Frédéric Raynal, "NFS - Network File System", [online] Available:
http://www.linuxfocus.org/English/November2000/article164.meta.shtml [Accessed:
18/09/2014].
57. Paxson V, Floyd, Sally, "Wide area traffic: the failure of Poisson modeling",
IEEE/ACM Transactions on Networking, vol.3, no.3, pp.226-244, Jun 1995.
58. C. J. Bovy, H. T. Mertodimedjo, G. Hooghiemstra, H. Uijterwaal and P. van Mieghem,
“Analysis of end-to-end delay measurements in Internet”, in Proc. Passive and Active
Measurement Workshop (PAM 2002), Fort Collins, CO, Mar. 2002.
59. Wang, Junfeng and Zhou, Mingtian and Li, Yuxia: “Survey on the End-to-End Internet
Delay Measurements”, in Mammeri, Zoubir and Lorenz, Pascal (Edt): “High Speed
Networks and Multimedia Communications”, Lecture Notes in Computer Science,
Vol.3079, Springer Berlin Heidelberg, 2004, pp. 155-166.
60. Ward Whitt "The Impact of a Heavy-Tailed Service-Time Distribution Upon the
M/GI/s Waiting-Time Distribution", Queueing Systems, vol. 36, November 2000, pp.
71-87.

75