CSCI3150 Introduction to

Operating Systems

Lecture 16: Virtualization

Hong Xu
https://github.com/henryhxu/CSCI3150
https://github.com/henryhxu/CSCI3150
Acknowledgement

 Based on slides from Brad Cambell@U. Virginia for CS6456

CSCI3150 Intro to OS 2
What is virtualization?

 Virtualization is the ability to run multiple operating systems on a single physical syst
em and share the underlying hardware resources 1

 Allows one computer to provide the appearance of many computers.

 Goals:
 Provide flexibility for users

 Amortize hardware costs

 Isolate completely separate users

1 VMWare white paper, Virtualization Overview

3
Requirements for Virtualizable Architectures

 “First, the VMM provides an environment for programs which is essentially identical
with the original machine;

 second, programs run in this environment show at worst only minor decreases in spe
ed;

 and last, the VMM is in complete control of system resources.”

 VMM: virtual machine monitor (from VMware); a.k.a. hypervisor

4
VMM Platform Types

 Hosted Architecture

 Install as application on existing x86 “host” OS, e.g. Windows, Linux, OS X

 Small context-switching driver

 Leverage host I/O stack and resource management

 Examples: VMware Player/Workstation/Server, Microsoft Virtual PC/Server, Parallels

Desktop

 Bare-Metal Architecture

 “Hypervisor” installs directly on hardware

 Acknowledged as preferred architecture for high-end servers

 Examples: VMware ESX Server, Xen, Microsoft Viridian (2008)

6
Virtualization: rejuvenation

 1960’s: first track of virtualization

 Time and resource sharing on expensive mainframes

 IBM VM/370

 Late 1970s and early 1980s: became unpopular

 Cheap hardware and multiprocessing OS

 Late 1990s: became popular again

 Wide variety of OS and hardware configurations

 VMWare

 Since 2000: hot and important

 Cloud computing

 Docker containers

7
IBM VM/370

 Technology: trap-and-emulate

Normal Application

Privileged Kernel

Trap Emulate

CPU

8
Trap and Emulate Virtualization on x86 architecture

 Challenges
 Correctness: not all privileged instructions produce traps!

 Performance:
 System calls: traps in both enter and exit (10X)

 I/O performance: high CPU overhead

 Virtual memory: no software-controlled TLB

9
Virtualization on x86 architecture

 Solutions:
 Dynamic binary translation & shadow page table

 Para-virtualization (Xen)

 Hardware extension

10
Dynamic binary translation

 Idea: intercept privileged instructions by changing the binary

 Cannot patch the guest kernel directly (would be visible to guests)

 Solution: make a copy, change it, and execute it from there

 Use a cache to improve the performance

11
Binary translation

 Directly execute unprivileged guest application code

 Will run at full speed until it traps, we get an interrupt, etc.
 “Binary translate” all guest kernel code, run it unprivileged
 Since x86 has non-virtualizable instructions, proactively transfer control to the VMM (no need for
traps)
 Safe instructions are emitted without change
 For “unsafe” instructions, emit a controlled emulation sequence
 VMM translation cache for good performance

12
How does VMWare do this?

 Binary – input is x86 “hex”, not source

 Dynamic – interleave translation and execution

 On Demand – translate only what about to execute (lazy)

 System Level – makes no assumptions about guest code

 Subsetting – full x86 to safe subset

 Adaptive – adjust translations based on guest behavior

13
Convert unsafe operations and cache them

Input: BB
55 ff 33 c7 03 ...
Each Translator Invocation
• Consume a basic block (BB)
• Produce a compiled code fragment (CCF)

Store CCF in Translation Cache

translator
• Future reuse
• Capture working set of guest kernel
• Amortize translation costs
• Not “patching in place”
Output: CCF
55 ff 33 c7 03 ...

14
Dynamic binary translation

 Pros:
 Make x86 virtualizable

 Can reduce traps

 Cons:
 Overhead

 Hard to improve system calls, I/O operations

 Hard to handle complex code

15
Shadow page table

Guest
page table

Shadow
page table

16
Shadow page table

 Pros:
 Transparent to guest VMs

 Good performance when working set is stable

 Cons:
 Big overhead of keeping two page tables consistent

 Introducing more issues: hidden fault, double paging …

17
Xen

18
Xen and the art of virtualization

 SOSP’03

 Very high impact (data collected in 2013)

Citation count in Google scholar

6000
8807
5153
5000
4000
3000 30902286
21161796
2000 1219 1222 1229 1413
1093
1000 461
0

19
Para-virtualization

 Full vs. para virtualization

20
Overview of the Xen approach

 Support for unmodified application binaries (but not OS)

 Keep Application Binary Interface (ABI)

 Modify guest OS to be aware of virtualization

 Get around issues of x86 architecture

 Better performance

 Keep hypervisor as small as possible

 Device driver is in Dom0

21
Xen architecture

22
Virtualization on x86 architecture

 Challenges
 Correctness: not all privileged instructions produce traps!

 Performance:
 System calls: traps in both enter and exit (10X)

 I/O performance: high CPU overhead

 Virtual memory: no software-controlled TLB

23
CPU virtualization

 Protection
 Xen in ring0, guest kernel in ring1

 Privileged instructions are replaced with hypercalls

 Exception and system calls

 Guest OS registers handlers validated by Xen

 Allowing direct system calls from app into guest OS

 Page faults: redirected by Xen

24
Memory virtualization

 Xen exists in a 64MB section at the top of every address space

 Guest sees the mapping to real machine address

 Guest kernels are responsible for allocating and managing the hardware page tables.

 After registering the page table to Xen, all subsequent updates must be validated.

25
Porting effort is quite low

26
Evaluation

27
Conclusion

 x86 architecture makes virtualization challenging

 Full virtualization
 unmodified guest OS; good isolation

 Performance issue (especially I/O)

 Para virtualization:
 Better performance (potentially)

 Need to update guest kernel

 Full and para virtualization will keep evolving together

28
Instead: Leverage hardware support

 First generation - processor

 Second generation - memory

 Third generation – I/O device

 In progress

29
Protection Rings

 Actually, x86 has four protection CPU Privilege Level (CPL)

levels, not two (kernel/user).
 X86 rings (CPL)
 Ring 0 – “Kernel mode” (most
privileged)
 Ring 3 – “User mode”
 Ring 1 & 2 – Other
Increasing Privilege Level
 Linux only uses 0 and 3.
 “Kernel vs. user mode” Ring 0
 Pre-VT Xen modified to run the Ring 1
guest kernel to Ring 1: reserve
Ring 0 for hypervisor. Ring 2

Ring 3

[Fischbach] 30
Why aren’t protection rings good enoug
h?

VM
CPL 3
Increasing Privilege Level
guest kernel CPL 1
Ring 0
hypervisor CPL 0 Ring 1
Ring 2

Ring 3

31
A short list of pre-VT problems

Early hypervisors (VMware, Xen) had to emulate various machine behaviors

and generally bend over backwards.
 IA32 page protection does not distinguish CPL 0-2.
 Segment-grained memory protection only.

 Ring aliasing: some IA instructions expose CPL to guest!

 Or fail silently…

 Syscalls don’t work properly and require emulation.

 sysenter always transitions to CPL 0. (D’oh!)
 sysexit faults if the core is not in CPL 0.

 Interrupts don’t work properly and require emulation.

 Interrupt disable/enable reserved to CPL0.

32
First generation: Intel VT-x & AMD SVM

 Eliminating the need of binary translation or modifying OSes

Host mode Guest mode

Ring3 Ring3
VMRUN
Ring2 Ring2
Ring1 VMEXIT Ring1
Ring0 Ring0

33
VT in a Nutshell

 New VM mode bit, Orthogonal to CPL (e.g., kernel/user mode)

 If VM mode is off → host mode

 Machine “looks just like it always did” (“VMX root”)

 If VM bit is on → guest mode

 Machine is running a guest VM: “VMX non-root mode”

 Machine “looks just like it always did” to the guest, BUT:

 Various events trigger gated entry to hypervisor (in VMX root)

 A “virtualization intercept”: exit VM mode to VMM (VM Exit)

 Hypervisor (VMM) can control which events cause intercepts

 Hypervisor can examine/manipulate guest VM state and return to VM (VM Entry)

34
 CPU Virtualization With VT-x

Virtual Machines (VMs)

 Two new VT-x operating modes
 Less-privileged mode
(VMX non-root) for guest OSes Ring 3 Apps Apps
 More-privileged mode
(VMX root) for VMM
Ring 0 OS OS
 Two new transitions
 VM entry to non-root operation VM Exit VM Entry

 VM exit to root operation VMX

VM Monitor (VMM)
Root
• Execution controls determine when exits occur
• Access to privilege state, occurrence of exceptions, etc.
• Flexibility provided to minimize unwanted exits
• VM Control Structure (VMCS) controls VT-x operation
• Also holds guest and host state
35
Second generation: Intel EPT & AMD NPT

 Eliminating the need to shadow page table

36
Containers and isolation

42
Containers: idea

 Run a process with a restricted view of system resources

 Hide other processes

 Limit access to system resources

 Not full virtualization

 Process must use existing kernel and OS

 Benefits
 Consistent environment (runtime, dependencies, etc.)

 Low overhead, rapid launch/terminate

 Performance

43
Pre-container isolation features in Linux

 chroot
 Set the current root directory for processes
 Added to Unix in 1979

 Namespaces
 Provide processes with their own view of resources
 Process IDs, networking sockets, hostnames, etc.

 Akin to virtual address spaces

 Originally introduced in 2002

 Copy-on-Write Filesystem
 Allow a process to view existing filesystem, but any modifications result in copies then
updates
 Akin to virtual memory after fork

44
Key technologies

CSCI3150 Intro to OS 45
Namespace

 “A namespace wraps a global system resource in an abstraction that makes it appear

to the processes within the namespace that they have their own isolated instance of
the global resource.” – Linux man-pages

 Controls what resources a process can see, and what they are called

 Restrict visibility -> solve “inconsistent distribution”

CSCI3150 Intro to OS 46
cgroups in Linux

 Enforce policies to restrict how processes use resources

 Example: A process can only use 1 Mbps of network bandwidth

 Policies are enforced on groups of processes

 Controllers enforce various processes

 Restrict usage -> solves “resource isolation”

47
Controllers enforce restrictions for cgroups

 Different controllers for different system resources

 Each provides a policy for how processes should be restricted

48
Controllers in Linux

 io
 Limit I/O requests either capped per process or proportionally.

 memory
 Enforce memory caps on processes

 pids
 Limit number of new processes in a cgroup

 perf_event
 Allow monitoring performance

 cpu
 Limit CPU usage when CPU is busy

 freezer
 Allow suspending all processes in a cgroup
49
Linux containers = combinations of namespaces & cgroups

50