How to:

Migrate VMs
from to

A step-by-step Guide
 How to: Migrate VMs from to

Table of Contents

Abstract 3

Introduction 4

Supported Hypervisors 6

Used Tools 7

Migrating VMs from Hyper-V to KVM 8

Building virt-v2v 12

www.storpool.com 2
 How to: Migrate VMs from to

Abstract
In a series of papers, we have outlined a detailed methodology for migrating virtual machines
from some of the most popular traditional hypervisors to the New Age It stack, powered by KVM,
with minimal downtime, using open-source tools.

This Step-by-step Guide is targeted at senior technical staff, who are building & managing IT
infrastructure and the applications running on top of this infrastructure.

The presented techniques are successfully used to migrate CentOS and Windows Server virtual
machines from VMWare ESXi, Hyper-V and Citrix XenServer to KVM, but it can be used with other
guest operating systems and other types of hypervisors.

www.storpool.com 3
 How to: Migrate VMs from to

Introduction
For the last few years, companies have witnessed the rising popularity of open-source (OSS)
technologies driven by large- and hyper-scale deployments, the cloud-native movement and the
pursuit to eliminate vendor lock-in. These are some of the fundamental drivers to an economy,
heavily driven by digital transformation where all businesses have to be agile, efficient and
re-invented by and with technology.

The requirements of the business today require any company to heavily invest in New-Age,
modern IT, which is resilient, automated, self-service, scalable. This is best achieved by intelligent
soſtware, thus we witness the rise of soſtware-defined technologies and datacenters.

This process inevitably means that the old, traditional ways of designing and building IT systems
are being replaced with latest-generation designs, tools, and systems, which deliver what the
businesses of tomorrow will need.

On the hardware/IT infrastructure side, this usually means adopting SDDC (Soſtware-Defined
Datacenter) designs. I.e. using COTS (Commodity of the Shelf servers), standard Ethernet
networks and SDS / SDN (Soſtware-Defined Storage / Soſtware-Defined Networking).

On the soſtware side, this means adopting self-service, API-first stacks - typically Linux + KVM +
containers with different orchestrators - Kubernetes, OpenStack for example. It also means
migrating applications (or the VMs in which they run) from legacy hardware and soſtware stacks
(think VMware, Hyper-V, Oracle, etc.) to the modern IT stacks. A process that is not always simple
or straight-forward.

There are many tools developed to facilitate the migration of virtual machines from one
virtualization platform to another. Migrating a VM from one type of hypervisor to another requires
converting the Virtual machine metadata, disk image format, and VM image content or OS
morphing. The latter includes updating guest OS configuration settings and installing required
drivers for the new target hypervisor.

One of the challenges in the migration process is reducing the downtime. Most tools we know
implements more or less this generic workflow:

1. Stop the VM at the source hypervisor

2. Convert VM metadata and define a new VM at the target hypervisor
3. Copy the disk images from the source to the target hypervisor
4. Convert the VM disk image format
5. OS morphing or converting disk image content - update settings, install drivers for the
new emulated hardware, etc.
6. Start the VM at the target hypervisor

www.storpool.com 4
 How to: Migrate VMs from to

Some tasks could be combined in a single step and the order could be slightly different, but in
general, this workflow imposes a significant downtime. The majority of this downtime is accounted
for copying the disk images, especially if this involves a migration of large disk images to remote
storage.

The proposed method can reduce the downtime in many cases to less than a minute, even when
large disk images have to be copied over low-bandwidth links. This can be achieved by altering
the workflow as follows:

1. Convert VM metadata and define a new VM at the target hypervisor

2. Take a snapshot of the VM disks on the source hypervisor
3. Copy the snapshot to the target hypervisor
4. Convert the VM disk image format
5. Take a second snapshot, if needed, and copy the latest changes to the target hypervisor
6. Apply the changes to the target image.
7. Stop the VM at the source hypervisor
8. Copy the data changed since the last snapshot to the target hypervisor
9. Apply the changes to the target image.
10. OS morphing
11. Start the VM at the target hypervisor

While the overall number of steps has been increased, the downtime is reduced by limiting the
time when the VM is non-operational to only 3 short steps, while the most time-consuming steps
- copying large amounts of data are executed outside of this critical time frame.
The major challenge with this approach is the ability to take snapshots created in one image
format on one storage and apply them on a different image format on different storage. In this
paper, we’ll show how this can be done with a set of open-source tools.

www.storpool.com 5
 How to: Migrate VMs from to

Supported Hypervisors
The method described in this paper supports migration from VMware ESXi, MS Hyper-V, and
XenServer to KVM. Other types of hypervisors can be supported with small modifications, as far
as source images are stored in the supported image file formats - vmdk (on VMFS6), vhd and
vhdx. Check StorPool’s website for the Step-by-step guides on how to migrate VMs from VMware
to KVM and from XenServer to KVM.

www.storpool.com 6
 How to: Migrate VMs from to

Used Tools
In this paper we use several tools:
• qemu-img - part of qemu project, used to convert base vmdk files to raw images.
• vhdx - open source tool used to read vhdx images and snapshots created by MS Hyper-V
and convert them to raw images.
• virt-v2v - part of libguestfs project, this tool is used to morph the OS for use with KVM. It
supports major Linux distributions and Windows as guest OS. The used version is compiled
from source and includes a patch to install virtio drivers in WIndows guests.
• Support scripts to copy images from the source hypervisor

In this paper, we’re using a dedicated Linux VM (called here conversion VM) to do the actual
conversion. The conversion VM has an SSH access to the source hypervisors or the storage
where it can download images from and has direct access to the target storage, where the target
images are created as raw image files or the image content is stored in block devices. All tools
listed above are installed on the conversion VM.

Tools can be obtained from:

• qemu-img: use the package supplied by the Linux distro
• sesparse, vhdx, vhd, various support scripts, patch for virt-v2v:
https://github.com/storpool/any2kvm
• virt-v2v: http://download.libguestfs.org/ (see “Building virt-v2v” later in this paper for
details)

www.storpool.com 7
 How to: Migrate VMs from to

Hyper-V to KVM

Preparation
• Install OpenSSH server on the Hyper-V hypervisor from
https://github.com/PowerShell/Win32-OpenSSH/releases . Follow the instructions at
https://hostadvice.com/how-to/how-to-install-an-openssh-server-client-on-a-windows-201
6-server/
• create dir a C:\ProgramData\ssh
• aſter creating host keys run
C:\Program Files\OpenSSH-Win64> .\FixHostFilePermissions.ps1
• edit C:\ProgramData\ssh\ssd_config and add: PermitRootLogin yes
• Set PowerShell as default shell for OpenSSH:
PS C:\> New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force
• Add ssh public keys for password-less authentication to
C:\Users\Administrator\.ssh\authorized_keys

Step-by-step process

• Get the VM details - CPU, RAM size, number of interfaces, etc.

• Create a target VM on the target hypervisor with the same settings and empty disks of the
same size. The actual steps depend on the target orchestration.
• Stop the target VM
• If shared block storage is used for the target VM (e.g. iSCSI), attach the target block device
to the conversion VM.

[user@libguestfs-2 hyper-v]$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 150G 0 disk
└─sda1 8:1 0 150G 0 part /
sdb 8:16 0 10G 0 disk

• Make a snapshot of the source VM.

[user@libguestfs-2 hyper-v]$ ssh Administrator@s19 Get-VM -VM WinSrv2019-1

Name State CPUUsage(%) MemoryAssigned(M) Uptime Status Version

---- ----- ----------- ----------------- ------ ------ -------
WinSrv2019-1 Running 0 4096 01:59:32.1370000 Operating normally 8.0

[user@libguestfs-2 hyper-v]$ ssh Administrator@s19 'Checkpoint-VM WinSrv2019-1'

www.storpool.com 8
 How to: Migrate VMs from to

• Get the disk image filename and path. This is the filename of the top-most file in the
snapshot hierarchy, that is currently being used by the VM as read-write:
[user@libguestfs-2 hyper-v]$ ssh Administrator@s19 '(Get-VM -VM WinSrv2019-1).HardDrives.Path'
C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_268F4E46-B05F-4811-9B5A-7B2A5C8C9798.avhdx

• Run the copy-hv-to-raw.py script that will copy the entire image hierarchy and convert it
to a flat raw image or store it on a block device. This script takes as a parameter the path of
the currently used top-most image file. With the default parameters, the script can be run
while the VM is still running and will not copy or convert the top-most image file because it is
in use and not in a consistent state. The path to the source mage is given in windows style
with backslashes.

[user@libguestfs-2 hyper-v]$ ./copy-hv-to-raw.py s19 \

'C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_268F4E46-B05F-4811-9B5A-7B2A5C8C9798.avhdx' /dev/sdb
Parent = C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_DE61D07E-C5D3-4727-AF7E-B0B8EBA71536.avhdx
Parent = C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1.vhdx
Parent =
Skipping the top image C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual
Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_268F4E46-B05F-4811-9B5A-7B2A5C8C9798.avhdx.
Chain to convert, starting from root:
C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1.vhdx
C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_DE61D07E-C5D3-4727-AF7E-B0B8EBA71536.avhdx
Downloading a9fe144be3c40a80b11f8a14c9580bbd (C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual
Machines\WinSrv2019-1\Virtual Hard Disks\WinSrv2019-1.vhdx)
WinSrv2019-1.vhdx 100% 8836MB 6.6MB/s 22:23
Converting /var/tmp/hv-convert/a9fe144be3c40a80b11f8a14c9580bbd

syncing

Conversion Done!
Downloading ee141755bfb4498e5507e71d3a9b71a6 (C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual
Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_DE61D07E-C5D3-4727-AF7E-B0B8EBA71536.avhdx)
WinSrv2019-1_DE61D07E-C5D3-4727-AF7E-B0B8EBA71536.avhdx 100% 889MB 23.4MB/s 00:38
Converting /var/tmp/hv-convert/ee141755bfb4498e5507e71d3a9b71a6

syncing

Conversion Done!
To continue with the conversion from the current state, next time run this tool with
`--start-at C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_DE61D07E-C5D3-4727-AF7E-B0B8EBA71536.avhdx`

www.storpool.com 9
 How to: Migrate VMs from to

The script caches the downloaded image files.

• (Optional) Make a second snapshot and copy it to the target image. This will reduce the
time needed to transfer the last snapshot, while the VM is powered off. Repeat the previous
3 steps. Note the disk image path is changed on every snapshot.
• Shutdown the VM at the source hypervisor
• Run the final transfer. Note the -f flag. Use the top-most file in the snapshot hierarchy.

[user@libguestfs-2 hyper-v]$ ./copy-hv-to-raw.py -f --start-at \

'C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_DE61D07E-C5D3-4727-AF7E-B0B8EBA71536.avhdx' s19 \
'C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_268F4E46-B05F-4811-9B5A-7B2A5C8C9798.avhdx' /dev/sdb

Parent = C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard

Disks\WinSrv2019-1_DE61D07E-C5D3-4727-AF7E-B0B8EBA71536.avhdx
Reached the image C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual
Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_DE61D07E-C5D3-4727-AF7E-B0B8EBA71536.avhdx, Skipping all the rest.
Chain to convert, starting from root:
C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_268F4E46-B05F-4811-9B5A-7B2A5C8C9798.avhdx
Downloading 5b85cfe0df40da7e13b3b182a46c9c13 (C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual
Machines\WinSrv2019-1\Virtual Hard
Disks\WinSrv2019-1_268F4E46-B05F-4811-9B5A-7B2A5C8C9798.avhdx)
WinSrv2019-1_268F4E46-B05F-4811-9B5A-7B2A5C8C9798.avhdx 100% 325MB 27.0MB/s 00:12
Converting /var/tmp/hv-convert/5b85cfe0df40da7e13b3b182a46c9c13

syncing

Conversion Done!

• Morph the OS. Use the compiled virt-v2v tool. Use a minimal libvirt domain xml
configuration file listed below for the target VM. This file is needed only for this step. The
actual domain xml of the target VM is created by the orchestration system and will be
different.

[user@libguestfs-2 hyper-v]$ cat domain.xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>tmp-v2v</name>
<devices>
<disk type='file' device='disk'>
<source file='/dev/sdb'/>
<target dev='sda' bus='scsi'/>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'>
<driver queues='1'/>
</controller>
</devices>
</domain>

[user@libguestfs-2 hyper-v]$ RUN=/home/user/libguestfs-1.40.2/run

[user@libguestfs-2 hyper-v]$ $RUN virt-v2v -i libvirtxml domain.xml --in-place
[ 0,0] Opening the source -i libvirtxml domain.xml
[ 0,0] Opening the source VM
[ 13,1] Inspecting the source VM
[ 20,5] Checking for sufficient free disk space in the guest

www.storpool.com 10
 How to: Migrate VMs from to

[ 20,5] Converting Windows Server 2019 Standard Evaluation to run on KVM

virt-v2v: warning: /usr/local/share/virt-tools/pnp_wait.exe is missing.
Firstboot scripts may conflict with PnP.
virt-v2v: warning: there is no QXL driver for this version of Windows (10.0
x86_64). virt-v2v looks for this driver in /usr/local/share/virtio-win

The guest will be configured to use a basic VGA display driver.

virt-v2v: This guest has virtio drivers installed.
[ 29,4] Mapping filesystem data to avoid copying unused and blank areas
virt-v2v: warning: fstrim on guest filesystem /dev/sda2 failed. Usually
you can ignore this message. To find out more read "Trimming" in
virt-v2v(1).

Original message: fstrim: fstrim: /sysroot/: the discard operation is not

supported
[ 30,3] Closing the source VM
[ 30,4] Finishing off
[user@libguestfs-2 hyper-v]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 150G 0 disk
└─sda1 8:1 0 150G 0 part /
sdb 8:16 0 10G 0 disk
├─sdb1 8:17 0 499M 0 part
├─sdb2 8:18 0 99M 0 part
├─sdb3 8:19 0 16M 0 part
└─sdb4 8:20 0 9,4G 0 part

• Start the VM at the target hypervisor

Notes:
1. The actual location of vhdx files may be different, depending on the storage options and
Hyper-V settings.
2. In this example, we’re using a raw block device for the target virtual disk. If file storage is
used a raw image file will be created and the filename shall be passed to the
copy-hv-to-raw.py tool.
3. If cqow2 output image format is needed, then the processing shall be done in raw format,
and as the last step, the image shall be converted from raw to qcow2 with qemu-img .
4. In this example, we’re converting Windows Server 2019 VM. The output of the morphing
will be different for different guest OS. See below for preparing the virt-v2v .
5. If any of the conversion steps aſter the VM has been stopped at the source hypervisor fails
or if the converted VM is not able to start at the target hypervisor, the VM can be re-started
on the source HV and the procedure can be repeated.
6. The procedure of converting virtual machines and virtual network definition from Hyper-V
to the target KVM-based cloud is not covered in this paper. Only the transfer, conversion and
morphing of the disk images is covered here.

www.storpool.com 11
 How to: Migrate VMs from to

Building virt-v2v
An important part of the conversion process is morphing the guest OS - installing virtio drivers
required to run the OS on KVM and updating the OS configuration. This is done using the virt-v2v
tool, part of libguestfs project (http://libguestfs.org/). virt-v2v is available as a package for most
Linux distributions, but the packaged version has some limitations. For example, the package
available in the CentOS repo doesn’t permit in-place image processing. This is a key feature for
the method described here to reduce the downtime, by avoiding mass copying of large amounts
of data. Also, the bundled package is not capable of installing virtio drivers for windows, because
it doesn’t register the certificate that is used to sign the drivers.
For these reasons, virt-v2v needs to be compiled from source aſter applying some changes.
Following is the step-by-step procedure for building virt-v2v for CentOS 7.

• Get the source tarball of the latest stable libguestfs from http://download.libguestfs.org/
• Get the patch for virt-v2v, link is available in section ‘Used Tools’.
• Get virtio-win-*iso from
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-qemu-ga/
• Use a windows workstation or VM to extract a signed driver file form the iso image, for
example amd64\w10\vioscsi.sys and extract RedHat certificate from it:
Select the .sys file, right-click -> Properties -> Digital Signatures -> Select ‘Red Hat, Inc.’ ->
Details -> View Certificate -> Details -> Copy to File …; Select ‘DER encoded binary X.509
(.CER)’
• Get the created .cer file in the conversion VM and save it as cert1.cer . It will be used
a few steps later.
• Verify cert1.cer file is correct, and the certificate is valid (note the ‘Not Aſter’ date) :

[user@libguestfs-2 ~]$ openssl x509 -inform DER -in cert1.cer -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
5d:10:cb:18:eb:3a:79:00:87:83:ab:74:77:f9:d3:19
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=Symantec Corporation, OU=Symantec Trust Network, CN=Symantec Class 3
SHA256 Code Signing CA - G2
Validity
Not Before: Nov 27 00:00:00 2018 GMT
Not After : Jan 25 23:59:59 2022 GMT
Subject: C=US, ST=North Carolina, L=Raleigh, O=Red Hat, Inc., CN=Red Hat, Inc.
Subject Public Key Info:
...

• Untar the libguestfs source:

tar zxvf libguestfs-1.40.2.tar.gz

www.storpool.com 12
 How to: Migrate VMs from to

• Install the dependencies:

sudo yum install yum-utils

sudo yum-builddep libguestfs
sudo ln -s supermin5 /usr/bin/supermin

• Apply the patch for virt-v2v:

cd libguestfs-1.40.2/
patch -p 1 -i ../v2v-patch

• Configure and build libguestfs:

./configure
make
make quickcheck

• Install windows support. This is needed only if windows VM will be morphed:

sudo yum install libguestfs-winsupport
cp /usr/lib64/guestfs/supermin.d/zz-winsupport.tar.gz appliance/supermin.d/
sudo mkdir -p /usr/local/share/virtio-win
sudo mount -o loop,ro virtio-win-0.1.173.iso /usr/local/share/virtio-win
sudo cp /usr/local/share/virtio-win/guest-agent/qemu-ga-x86_64.msi /opt/windows-convert/files/
sudo mkdir -p /opt/windows-convert/files/
sudo cp cert1.cer /opt/windows-convert/files/cert1.cer
sudo cp /usr/local/share/virtio-win/guest-agent/qemu-ga-x86_64.msi /opt/windows-convert/files/

We hope this paper gives you a detailed methodology and easy way to migrate your virtual
machines. If you have any comments or questions, our technical team will be happy to talk to you!
Drop us an email at [email protected].

www.storpool.com 13
 Get in Touch

+1 415 670 9320

+44 (0) 20 7097 8536

[email protected]

[email protected]

Get Started Book a Demo

www.storpool.com