Proc. of Int. Conf.

on Emerging Trends in Engineering and Technology

Evaluating Performance of Virtual Machines on

Hypervisor (Type-2)
Prakash P and Biju R Mohan
Department of Information Technology National Institute of Technology, Karnataka Surathkal, India
[email protected], [email protected]

Abstract—Virtualization is becoming increasingly popular, both for servers as well

as desktop systems. It is a core technology of the IT applications such as green IT
and cloud computing. Operating system which directly communicates with
system hardware is host operating system and Operating system which runs on
hypervisor type-2 is guest (virtual) operating system. Performance of the guest
(virtual) operating system relies equally on both host operating system and
hypervisor. Furthermore, Hypervisor is the core component of virtual machine (VM)
system and its effectiveness greatly impacts the performance of whole system. This
paper provides the performance of different guest (virtual) operating on same host
operating system and also it discuss performance of two Virtualization hypervisors
(type-2) available for x86 architecture- VMware Workstation and Virtual Box using
benchmark applications. From performance evaluation the VMware Workstation has
the best performance and also Windows 8 has the best performance as virtual
operating system on both VMware and VirtualBox

Keywords—Virtualization, Hypervisor type-2, Virtual Machine, VMware, VirtualBox

I. I NTRODUCTION
In IT, things are changing day by day. Often users face situations where they want run applications which
are not compatible with legacy machines. Web designers want to view their work on various operating
systems using their native web browsers. To overcome this situation and save resources there is need of
running multiple operating systems on the same machine at a time. In such scenario, it is interesting
to evaluate the performance of virtual machines running on different hypervisors type-2 available in the
market.
Virtualization provides many benefits [1][2] such as re- duced electricity consumption, improved
utilization, perfor- mance isolation, increased availability, fault tolerance, ease of management, system
security, and flexibility. Those benefits are translated into greater benefits in Cloud computing. A user can
request for multiple virtual machines (VMs) and expect their provision in a few minutes. The user can
release the VMs anytime.
A. Hosted
Hosted is a type-2 hypervisor[3][4]. In this architecture the base operating system is first installed. A
software layer called hypervisor or virtual machine monitor is installed on the top of the host operating
system and allows the users to run various guest operating systems within their own application window as
shown in Fig. 1.

DOI: 03.AETS.2013.3.85
© Association of Computer Electronics and Electrical Engineers, 2013
B. VMware Workstation
A hosted x86 virtualization monitor which can run a guest operating system unmodified with some
performance loss. The x86 architecture offers four levels of privilege known as Ring

Fig. 1: Hosted virtual machine monitors install on the top of an underlying host operating system

0, 1, 2 and 3 to operating systems and applications to manage access to the computer hardware. While user
level applications typically run in Ring 3, the operating system needs to have direct access to the memory
and hardware and must execute its privileged instructions in Ring 0.
Virtualizing[5] the x86 architecture requires placing a vir- tualization layer under the operating system
(which expects to be in the most privileged Ring 0) to create and manage the virtual machines that deliver
shared resources. Some sensi- tive instructions cant effectively be virtualized as they have different
semantics when they are not executed in Ring 0. The difficulty in trapping and translating these
sensitive and privileged instruction requests at runtime was the challenge that originally made x86
architecture virtualization look im- possible. VMware resolved the challenge by developing binary translation
techniques that allow the VMM to run in Ring 0 for isolation and performance, while moving the operating
system to a user level ring with greater privilege than applications in Ring 3 but less privilege than the
virtual machine monitor in Ring 0. It does not support Hyper Threading3 and requires a host operating
system, which means an extra layer and additional overhead.
C. VirtualBox
Oracle VM VirtualBox[6][7] is an x86 cross platform open source virtualization software package,
now developed by Oracle Corporation as part of its family of Virtualization products. VirtualBox is a so-
called ”hosted” hypervisor. To a very large degree, VirtualBox is functionally identical on all of the host
platforms, and the same file and image formats are used. This allows you to run virtual machines created on
one host on another host with a different host operating system; for example, you can create a virtual
machine on Windows and then run it under Linux.
VirtualBox primarily uses software virtualization to run virtual machines. This is the default behavior for
any virtual machines (with the exception of 64-bit guest operating sys- tems) created within the VirtualBox
environment. VirtualBox does, however, provide the option to enable hardware virtual- ization on a per
virtual machine basis when running on AMD- V and Intel-VT capable CPUs. On more recent CPU designs,
VirtualBox is also able to make use of nesting paging tables to improve virtual machine performance.
Support for USB and RDP is missing.
D. Operating System
Windows is most widely used operating system with a lot of features. operating systems on desktop
and portable computers. Every new version of Windows brings many new features and enhancements.
window 7 was launched in 2009. windows 7 adds improved performance on multi-core proces- sors,
improved boot performance, Direct Access, and kernel improvements. The main benefit from a new kernel
part of Windows 7 called MinWin is that it can be built, booted and tested separately from the rest of the
system.
Windows 8 is a totally new version of Windows that, in addition to the traditional desktop, also
includes a new- style interface for use with touch screens - whether that’s on a touch screen laptop,
all-in-one PC or tablet. Windows has only supported x86-based Intel and AMD PCs but that is all
572
changing with Windows 8, which will support devices running on ARM architecture. Performance wise the
windows
8 takes less boot time, dynamic and innovative desktop and supports USB 3.0 Linux operating system is free
and open source software for development and distribution. Linux was originally developed as a free
operating system for Intel x86-based personal computers.
Ubuntu 12.04 LTS (Precise Pangolin) is the current Ubuntu Long Term Support (LTS) release, made
available on schedule on 26 April 2012. Ubuntu is supported on X86 and ARM hardware, and nearly 70 %
of the PCs shipped by the major PC companies are now certified to work with Ubuntu.
We have chosen windows 7 and Windows 8 operating system[8] as guest operating systems since its market
share are more. These operating systems are most widely used in academics.
The rest of the paper is structured as follows. Section II discusses Related Work. Section III Benchmark
Applications, In section IV Experimental setup and Methodology. Analysis and Results are discussed in
section V. In section VI Conclu- sion.

II. RELATED W ORK

In [9], impact of the host operating system have been discussed in detail. They studied how different host
operating systems influence virtual machine performance. Windows XP, Windows Vista and Windows 7 are
used as host operating systems using Windows Vista as a virtual operating system. Results show that the
virtual operating system has the best performance when Windows 7 is used as the host operating system.
In [10], survey shows discuss the pro and cons of the VMware and VirtualBox. They discussed selection of
virtual environment for education purpose. Survey shows that both VirtualBox and VMware are viable
virtualization solution for educational purpose.
A paper[11] describes the basic knowledge about VMware and VirtualBox. Discuss the test conducted to
measure and analyse the performance of two virtual machine monitors VMware and VirtualBox using
LINPACK and IPERF, and pro- vide a quantitative and qualitative comparison of both virtual machine
monitors.. Result depicts the significant comparison on those operating system on Virtual Machine
Monitor.

III. BENCHMARK APPLICATIONS

Benchmark applications are specially developed for mea- suring the overall performance of computer
systems or for measuring performance of certain components. Components with the greatest impact on the
performance of the computer system are: CPU, memory, graphics subsystem and hard disk. Most computer
users use these applications for mutual com- puter systems performance comparison. We use these bench-
mark applications on the same hardware but on different host operating systems so performance
measurement results will show which host operating system has best performance on this hardware. In
addition to this, with two different programs we measure the time required to perform some complex oper-
ations which demand a lot of hardware resources. Benchmark applications and programs used are
described below.
A. Maxon Cinbench 11.5
Cinebench [12] is a real-world cross platform benchmark application that evaluates computer systems
performance ca- pabilities. It uses users common tasks within Cinema 4D to measure a systems
performance. The test procedure consists of two main components: the CPU performance test and the
graphics subsystem performance test.
CPU performance test scenario uses all of computer sys- tems processing power to render a photorealistic
3D scene, which uses various different algorithms to stress all available processor cores. The graphics
subsystem performance test uses a complex 3D scene depicting a car chase and measures the performance
of the graphics subsystem in an OpenGL mode. The results are shown with the number of points whereby
the higher the number implies the better performance.
B. PCMark7
PCMark2002 [13] has the ability to benchmark the CPU, memory, graphics and hard disk. In our
performance measurements we are using memory and hard disk test suites.
The results are shown with number of points and the higher the points mean better performance.
C. Passmark Performance test
PerformanceTest [14] objectively benchmarks computer systems by using a variety of different speed tests.
573
Standard test suites are:
• CPU tests - mathematical operations, compression, encryption, SSE and 3DNow! instructions,
• 2D graphics tests - drawing lines, bitmaps, fonts, text, and GUI (Graphical User Interface)
elements,
• Disk tests - reading, writing and seeking within disk files,
• Memory tests - allocating and accessing memory speed and efficiency,
D. Super PI
Super PI [15] is a benchmark program that overloads memory and the CPU by calculating number PI to a
specified number of digits after the decimal point up to a maximum of 32 million. We measured times
needed to calculate 32 millions digits. For calculation it uses the Gauss-Legendre algorithm that tests the
decimal floating point and mathematics performance of the CPU. The result is given in seconds whereby the
shorter the time the better the performance.
E. GeekBench 2.4
Geekbench [16] is a processor and memory benchmark that’s been designed for 64-bit operating systems
and multi- core processors. The application provides a comprehensive set of benchmarks to measure
processor and memory performance. Geekbench is refreshingly easy to use for a benchmarking
application and runs testing very quickly.
F. Video Encoding
Video encoding is performed with program Total Video Converter 3.50, available at [17]. The initial file
format is AVI (Audio Video Interleave) and the size of a video file is 702
MB. The destination file format is MPEG-2. Video encoding overloads memory and the CPU and the result
is given in seconds, so that the shorter the encoding time the better the performance.
G. Compression Time
WinRAR 4.0 [18] is an application that uses a highly sophisticated and original compression algorithm for
data compression or decompression. For performance measurement we use a built-in speed compression
test (the result is given in KB/s, the higher the better). Furthermore, we measure the time needed to
compress a 400 MB AVI file to a RAR format (the result is given in seconds whereby the shorter the
time the better the performance).

IV. EXPERIM E NTAL SETUP AND METHODOLOGY

Hypervisor type-2 is installed on the host operating system. A virtual machine is an operating system that is
managed by an underlying control program. A host operating system, in a virtual environment, hosts virtual
machines. The host operating system allocates physical resources, such as processor and memory
resources, to virtual operating systems running on the host operating system. The guest operating request
for the hardware via host operating system.
Performance evaluation considers hardware components which influence the performance. Below Table
depicts the Setup environments[19]. In today's market, the Windows operating system has name and the
Ubuntu Operating System which is an open source.
Several tests are conducted to compare the performance of virtual machine(windows) running on host
operating system. All test are conducted under the same controlled conditions. Steps carried out during the
test are depicted below.
A. Test 1
1) Install the host Operating system
2) Install VMware Workstation on host operating system
3) Install a Guest(virtual) Operating System(Windows7)
4) Install latest device drivers and operating system updates for guest operating system.
5) Install the benchmark application,
6) Reboot the system three times
7) Run the VMware Workstation
8) Run the benchmark application
9) Uninstall the benchmark application and Reboot the system
574
 TABLEI: SYSTEM S PECI FICATION

Host Operating Ubuntu 12.04 (LTS) 64-bit

System
Guest Operating Microsoft Windows 7 Professional 64 bit and
System Microsoft Windows 7 Professional 64 bit
System Specification of Host OS
Memory 4GB
CPU Core(TM) i5 CPU @ 3.20 GHz L2 cache 2MB

Graphics Card Intel graphic media accelerated HD

Hard Disk Hitachi SATA 500 GB, 5400 rpm

Virtual Environment
Memory 2GB
vCPUs with 2 cores per each processor, total 4 cores
Hard Disk 60 GB

Beside the operating system, hypervisor type-2 and benchmark application there was not another
application installed on the computer system. To get accuracy every test was carried out for 3 times and
arithmetic mean was calculated. In Test 2, VirtualBox was installed in step 2 of Test 1 and (3-7) repeated.
Carried out same test for windows 8 as Guest operating system. Rebooting three times(step-6) is required to
re-initialize the drivers and to refresh and clear any stuck temporary data in memory.
Performance evaluation is done by comparing virtual ma- chine performance measurement results for every
benchmark application on same host operating system. Results are evaluated by means of the following
metrics:
• Number of points (obtained in benchmark applica- tions)
• Time (required to complete complex operations)

V. RESULT AND ANALYSIS

In our performance measurements, we tested performance of computer system components with benchmark
applications installed on a virtual operating system. We focused more on performance of the following
hardware components: CPU, memory, graphics subsystem and hard disk drive. By compar- ing the same
performance measurements for different guest operating systems which are Windows 7 and Windows 8, we
can determine which virtual operating system provides the best performance for a virtual machine. And also
we measured the performance of VMware Workstation and VirtualBox. Results provides the best option to
select the better Hypervisor with better operating System.

Fig. 2: VMware Workstation and VirtulBox on Windows 7

The Fig. 2, 3, 4, 5 depicts the results of VMware Workstation and VirtualBox on Windows 7 and
Windows 8.
1) Maxon Cinbench 11.5: the performance of vCPUs in VMware shows that 6.8 % better than
575
VirtualBox using Windows 7 as guest OS where as in Windows 8, VMware shows that 14 % better than
VirtualBox
2) PCMark7: the results show that VMware workstation is 52.31% better than Virtualbox using
Windows 7 as guest OS where as in Windows 8, VMware workstation is 41% better than Virtualbox.
3) PassMark Performance Test: the results shows that VMware Woskstation is 47.83% better in contrast
to VirtualBox using Windows 7 as guest OS where as in Windows 8, VMware workstation is 59% better than

Fig. 3: VMware Workstation and VirtulBox on Windows 7

Fig. 4: VMware Workstation and VirtulBox on Windows 8

Fig. 5: VMware Workstation and VirtulBox on Windows 8

Virtualbox.
4) GeekBench: The performance of VMware Workstation 5.7% better than VirtualBox using Windows
7 as guest OS where as in Windows 8, VMware workstation is 8.1% better than Virtualbox.
5) Super PI: The shortest times needed to calculate 32 millions digits of number PI are obtained in
Vmware Workstation on both Windows 7 and Windows 8.
6) Video encoding and Compression: Time needed for Video encoding and data compression is 20% and
22% respec- tively less than VirtualBox Windows 7 as guest OS where as in Windows 8, VMware
workstation 25.5% and 43.2% respectively less than VirtualBox.
As per the above mentioned results, We concluded that VMware Workstation is better when we use

576
Windows 8 as Guest Operating System.
The following figures 6, 7, 8, 9 depicts the Windows 7 and Windows 8 performance on VMware
Workstation and VirtualBox.

Fig. 6: Win 7 vs Win 8 on VMware Workstation

Fig. 7: Win 7 vs Win 8 on VMware Workstation

Fig. 8: Win 7 vs Win 8 on VirtualBox

Fig. 9: Win 7 vs Win 8 on VirtualBox

577
1) Maxon Cinbench 11.5: the performance of vCPUs in Windows 7 shows that 3 % better than
Windows 8 using VMware Workstation as hypervisor where as in VirtualBox Windows7 shows
that 10% better than Windows 8.
2) PCMark7: the results show that Windows 7 is 3% better than Windows 8 using VMware workstation
as hypervisor where as in where as in VirtualBox Windows 8 shows that 22% better than Windows 7.
3) PassMark Performance Test: the results show that Windows 7 is 3% better than Windows 8 using
VMware workstation as hypervisor where as in where as in VirtualBox Windows 7 shows that 31% better
than Windows 8.
4) GeekBench: The performance of Windows 7 is same as windows 8 in VMware Workstation where as
in VirtualBox Windows 7 shows that 3% better than Windows 8.
5) Super PI: The shortest times needed to calculate 32 millions digits of number PI are obtained in
Windows 8 on both VMware Workstation and VirtualBox.
6) Video encoding and Compression: Time needed for Video encoding and data compression is 5.5% and
5% respec- tively less than Windows 7 for Windows 8 using VMware Workstion where as in VirtualBox,
Windows 8 2.7% and 16% respectively less than Windows 7.
As per the above mentioned results, We concluded that VMware Workstation with Windows 8 is better
in terms of Video Encoding, Data compression, Memory, Calculating million digits. Windows 7 is better
when we consider graphic on both Hypervisors.

VI. CONCLUSION
The experimental results give an insight that how the performance of the Guest Operating System on
Host oper- ating system on both the Virtualized environments. We run the Benchmark applications in the
Virtulized enviroment and examine their performance. These help the customers to choose which hypervisor
is good for their Operating System.
Thus we Observer that VMware Workstation gives better performance as compared to VirtualBox with
respect Hardware Components CPU, Memory, Graphic and Disk Access. We explore that Windows 8 gives
better performance as compared to Windows 7 with respect Calculating PI value, Video Encoding, Data
compression and overall performance given by PCMark7 it incluedes the CPU, memory, graphics, and hard
disk. From performance evaluation we concluded that the guest operating system has the best performance
when VMware Workstation is used as the Hypervisor with Windows 8 operating system.

REFERENCES
[1] W. Huang, J. Liu, B. Abali, and D. K. Panda. A case for high performance computing with virtual
machines. In Proc. of 20th ACM Intl Conf. on Supercomputing (ICS), pages 125134. ACM, Jun 2006.
[2] M. F. Mergen, V. Uhlig, O. Krieger, and J. Xenidis. Virtualization for high-performance computing. ACM SIGOPS
Operating Systems Review 40(2):811, Apr 2006
[3] Devi Prasad Bhukya1, S Ramachandram 1, Reeta Sony A.L 2. Evaluating Performance of Sequential Programs in
Virtual Machine Environments using Design of Experiment IEEE 2010
[4] ”Hypervisor Type-2” http://www.ibm.com/developerworks/cloud/library/cl- hypervisorcompare/
[5] VMware Inc. Understanding Full Virtualization, Paravirtualization and Hardware Assist. White paper.2007
[6] ”Oracle VM VirtualBox”, http://download.virtualbox.org/virtualbox/UserManual.pdf,
[7] Goran Martinovi, Josip Balen, Snjeana Rimac-Drlje, ”Impact of the Host Operating Systems on Virtual Machine
Performance”, MIPRO 2010, May 24-28, 2010, Opatija, Croatia.
[8] Windows Operating Systems, http://www.netmarketshare.com/operating-system-market-share.aspx? qprid=8&
qpcustomd=0
[9] Goran Martinovi, Josip Balen, Snjeana Rimac-Drlje, ”Impact of the Host Operating Systems on Virtual Machine
Performance”, MIPRO 2010, May 24-28, 2010, Opatija, Croatia.
[10] Peng Li, ” Seleceting and using Virtualization solutions - Our experience with VMwaer and VirtualBox ”, CCSC
2009
[11] Vasudevan M.S, Biju R Mohan and Deepak.K.Damodaran, ”Performance Measuring and Comparison
of VirtualBox and VMware”, International Conference on Information and Computer Networks, 2012
[12] Maxon,”Cinebench 11.5”,availableat http://www.maxon.net/products/cinebench/ overview .html, accessed: May
2013.
[13] ”PCMARK7 and Latest Version”, available at http://www.futuremark.com/benchmarks/pcmark, accessed: May
2013.
[14]”PassMark Performance Test”, available at http://www.passmark.com/products/pt.htm, accessed: May 2013.
[15] ”Super PI”, available at http://www.superpi.net/Download/, accessed: May 2013
578
[16] ”GeekBench 2.4”, available at http://www.primatelabs.com/geekbench/download/windows/, accessed: May 2013
[17] ”Total Video Convertor”, available at http://www.softpedia.com /get/Multimedia/Video
/Encoders-Converter- DIVX-Related/Total-Video-Converter.shtml, accessed: May 2013
[18] ”WinRAR 4.20”, available at http://www.softpedia.com/progDownload/WinRAR-Download- 2461.html, accessed:
May 2013
[19] http://www.comx-computers.co.za/AY031AV-HASH-10490310- specifications-52261.htm

579