Live Migration of Virtual Machines

Christopher Clark, Keir Fraser, Steven Hand,

1
Introduction
Live OS migration
◦ Migrating an entire OS and all of its appli
cations as an unit
◦ Memory state can be transferred in a consis
tent and efficient fashion
◦ Allow a separation of concerns between the
users and operator
◦ Minimize the downtime and total migration t
ime
◦ Pre-copy approach

2
Related Work
Vmotion
Process Migration
◦ Residual dependency
 an ongoing need for a host to maintain dat
a structures or provide functionality for
a process even after the process migrates
away from the host

3
Design(1) - Migrating Memory
Minimize both downtime and total migra
tion time
◦ Downtime – the period during which the ser
vice is unavailable
◦ Total Migration Time – the duration betwee
n when migration is initiated and when the
original VM can be discarded

4
Design(2) - Migrating Memory
Three phases of memory transfer
◦ Push phase
 Source VM continues running
 Pages are pushed across the network to destination
◦ Stop and copy phase
 The source VM stopped, pages are copied across to the
destination VM
◦ Pull phase
 New VM executes and find faults
 “Pull” pages from the source
Pre- copy
◦ A bounded iterative push phase with a very short
stop and copy phase

5
Design(3) – Network & Dis
k
Network
◦ Generate an unsolicited ARP reply fro
m the migrated host, advertising the
IP has moved to a new location
◦ A small number of in-flight packets m
aybe lost
Disk
◦ Network-attached Storage(NAS)

6
 Design(4) – Logical Steps
Stage 0: Stage 3:
Pre-Migration Stop and copy

Stage 2:
Stage 1: Stage 4: Stage 5:
Iterative
Reservation Commitment Activation
Pre-copy

VM running Downtime VM running

7
Design(5) – Logical Steps
Stage 0: Pre-Migration
◦ Preselect target host
Stage 1: Reservation
◦ Confirm the resource are available on d
estination host
Stage 2: Iterative Pre- copy
◦ First iteration, all pages are transfer
red from source to destination
◦ Subsequent iteration, copy dirty pages
during the previous transfer phase
8
Design(6) – Logical Steps
Stage 3: Stop and copy
◦ Stop the running OS at source host
◦ Redirect the network flow to destination host
◦ CPU state and remaining memory pages are trans
ferred
Stage 4: Commitment
◦ Destination host indicates to source it has su
ccessfully received a consistent OS image
◦ Source Host acknowledge and now can be discard
Stage 5: Activation
◦ VM starts on Destination host

9
Writable Working Set (WWS)
WWS
◦ A certain set of pages will be writte
n often
◦ Should be transferred via stop and co
py phase
◦ Use Xen’s shadow page tables to trac
k(?)

10
11
Implementation Issues(1)
Dynamic Rate - Limiting
◦ Administrator decides a minimum(m) an
d a maximum(M) bandwidth limit
◦ Transfer speed (v)
◦ Subsequent round calculate the dirty
ing rate (r)
 r = dirty pages / duration of previous rou
nd

12
Implementation Issues(2)
Dynamic Rate - Limiting
◦ The first round v = m
◦ Next round v = v*r
◦ Pre-copy will be terminated when v >
M or remain pages less than 256KB

13
Implementation Issues(3)
Rapid Page Dirtying
◦ The page dirtying is often physically clustered
◦ “Peek” those pages dirtied in the previous round
Stunning Rogue Process
◦ Some process may produce dirty memory at fast speed
◦ Ex. A test program which writes one word in every pa
ge was able to dirty memory at a rate of 320Gbit/sec
◦ Forking a monitoring thread within the OS kernel whe
n migrating begins
◦ Monitor the WWS of individual processes
◦ If the process dirty memory too fast, then “stun”
it

14
15
Implementation Issues(4)
Freeing Page Cache Pages
◦ OS can tell some or all of the free p
ages
◦ Do not transfer these pages while the
first iteration
◦ Reduce transferred time

16
Implementation Issues(5)
Two method for initiating and man
aging state transfer
◦ Managed migration
 A migration daemon running in the manageme
nt VM
◦ Self migration
 Implemented within the migratee OS
 A small stub required on the destination m
achine

(?)
17
Evaluation(1)
Dell PE-2650 server-class machine
Dual Xeon 2Ghz CPUs
2GB memory
Broadcom TG3 network interface
Gigabit Ethernet
Netapp F840 NAS
XenLinux 2.4.27

18
Evaluation(2)- Simple Web Server
Continuously serving a single 512
KB file to a set of 100 clients

19
Evaluation(3)- SPECweb99
SPECweb99 – a application-level
benchmark for evaluating web serve
r

20
Evaluation(4)
Quake 3 server – an online game
server with 6 player
◦ Downtime: 50ms
Diabolical Workload
◦ Running a 512MB host and use a simple
program that writes constantly to a 2
56MB region of memory
◦ Downtime : 3.5sec
◦ Rare in real world

21
Conclusion
Minimal impact on running service
s
Small downtime with realistic ser
ver

22
Virtual Machine Files

23
File format(1)
.XML File
◦ Save VM Configuration details
◦ Named with the GUID

24
File format(2)
.BIN files
◦ This file contains the memory of a vi
rtual machine or snapshot that is in
a saved state(running programs, data
for those programs, word processing d
ocuments you are viewing, etc.)
.VSV files
◦ This file contains the saved state fr
om the devices associated with the vi
rtual machine.
25
File format(3)
.Vhd files
◦ These are the virtual hard disk files
for the virtual machine(save things s
uch as files, folders, file system an
d disk partitions)
.avhd files
◦ These are the differencing disk files
used for virtual machine snapshots

26