Logical Domains

Liam Merwick
Sun Microsystems, Inc.
 Contents
• Background
• Components
• Features
• Architecture
• Configuration Examples
• Q&A

Page: 2
 Background

Page: 3
 Consolidation
• Conventional Unix server utilisation is 7 to 15%

• Data center compression

> Goal: Increase utilisation of equipment

10% Utilisation

20%

Time

10%

Page: 4
 Solutions from Sun
Hard Partitions Virtual Machines OS Virtualisation Resource Mgmt.
App
Server Database Identity
Server
File
Server
Web
Server
Mail
Server
Calendar Database Web
Server Server
SunRay Database
Server
App
Server App

OS

Server

Multiple OSes Single OS

Trend to flexibility Trend to isolation
Dynamic System Domains SPARC Logical Domains Solaris Containers Solaris Resource Manager
Solaris Xen (Zones + SRM) (SRM)

VMware ESX Solaris Containers

for Linux Applications
Microsoft Virtual Server
Solaris Trusted Extensions
Page: 5
 Virtualisation for SPARC platforms
• New “sun4v” architecture Operating
System

Solaris X Solaris X
update (genunix)
(genunix)

sun4u code Solaris X (sun4v)

sun4v
US-Z CPU
interface
code
SPARC hypervisor
CPU “Z” SPARC CPU

Platform
Page: 6
 Virtual Machine for SPARC

• Thin software layer between OS and

platform hardware sun4v virtual machine

User User User

App App App
• Hypervisor + sun4v interface
• Virtualises machine HW and isolates OS from Solaris
register-level
• Delivered with platform not OS OpenBoot
• Not itself an OS
Hypervisor

SPARC hardware

stable
interface
“sun4v”

Page: 7
 Logical Domains
• Partitioning capability
> Create virtual LDom 1 LDom 2 LDom 3
machines each with
sub-set of resources Solaris 10 Solaris 10 Solaris
Express App

> Protection & Isolation App

App App
using HW+firmware App

combination
App App
App
Zone Zone 1 Zone 2

Hypervisor
Hardware CPU CPU CPU CPU
Shared CPU,
Memory, IO Mem Mem Mem I/O

Page: 8
 Logical Domains Technology
• Virtualisation and partitioning of machine resources
> Each domain is a full virtual machine, with a dynamically reconfigurable sub-
set of machine resources, and its own independent OS
> Protection & isolation via SPARC hardware and LDoms Hypervisor firmware

LDom A LDom B LDom C LDom D

OS
Environment
of choice
Linux FreeBSD

LDoms
Hypervisor
CPU CPU CPU CPU CPU CPU CPU CPU
Platform Memory Memory Memory Memory
Hardware
I/O I/O

Page: 9
 Coupled with Throughput Computing

• Industry trend to many cores & strands per chip

> Most applications don't scale well
> Even Solaris doesn't scale perfectly

• Ability to dynamically partition system matches

throughput computing model
> Can run multiple services and OS instances on same
box / chip
> More flexibly be able to utilise available compute power

Page: 10
 Components

Page: 11
 Components – Hardware & Software
• Hardware
> UltraSPARC-T1 (Niagara) family of CMT processors
> T1000 Netra CP3060
> T2000 Netra T2000
• Operating System
> Solaris 10 11/06 (U3) + patches
> OpenSolaris (build 57+ recommended)
• Firmware
> 6.4.0
• LDoms Manager Software
> 1.0 (Solaris package - available as Free download)
Page: 12
 Features

Page: 13
 Fundamentals
• Each virtual machine should appear as an entirely
independent machine
> own kernel, patches, tuning parameters
> own user accounts, administrators
> own disks
> own network interfaces, MAC & IP addresses
> Start, stop and reboot independently of each other

• Paravirtualised OS

Page: 14
 Features
• Add and remove cpus while OS is running
• Protection and isolation via SPARC hardware and
LDoms Hypervisor firmware
• Service domains for I/O
> direct I/O access

Page: 15
 Features – UltraSPARC-T1 specific
• Up to 32 LDoms per box
> CPU threads individually assignable to different LDoms
• Memory assignable to with an 8KB granularity
• Two PCI busses independently assignable to the
same or different LDoms.
• Guest Domains can be configured, started and
stopped independently
> Without requiring a power-cycle of machine

Page: 16
 Architecture

Page: 17
 Architecture
• Logical Domains overview
• Hypervisor
• LDoms Manager
• Virtualised I/O
• Dynamic Reconfiguration (DR)

Page: 18
 Hypervisor Support
• Hypervisor software is responsible for maintaining
separation between domains
> Using extensions built into a sun4v CPU
• Also provides Logical Domain Channels (LDCs) so
that domains can communicate with each other
> Mechanism by which domains can be virtually networked
with each other, or provide services to each other

Page: 19
 LDoms Manager
• One Manager per host HV
> Controls Hypervisor and all its LDoms
• Exposes control interface via CLI
• Maps Logical Domains to physical resources
> Constraint engine
> Heuristic binding of LDoms to resources
> Assists with performance optimisation
> Assists in event of failures / blacklisting

Page: 20
 Logical Domains

File Web Mail

Server Server Server Application

OS

Server

Page: 21
 Direct I/O Logical Domain
App App

• Traditional model App

Logical Domain
owns PCI root
> Existing drivers and devices and tree
continue to work Device Driver
/pci@B/qlc@6
Nexus Driver
/pci@B
Privileged

Hyper Hypervisor Virtual Nexus I/F

Privileged

I/O MMU
Hardware I/O
Bridge PCI
Root

PCI-Express

Page: 22
 Virtualised I/O
Logical Service
Domain A Domain
App Device Driver
App /pci@B/qlc@6
App
App
Virtual Device Nexus Driver
Service /pci@B
Virtual Device
Privileged Driver

Virtual Nexus I/F

Hyper Hypervisor Domain Channel
Privileged

I/O MMU
Hardware I/O
PCI
Roo Bridge
t

PCI B
Page: 23
 Virtual (Block) Disk device
Logical Domain Logical Logical Service
1 Domain 2 Domain 3 Domain
App App App vDisk
App App App App App App Server
App App App vDisk
Server
vDisk vDisk vDisk
client client client Device-
Driver driver driver Driver

Hyper- Virtual SAN 2

visor Virtual SAN 1

I/O Bridge

FC-AL
I/F

Page: 24
 Virtual Ethernet device
Logical Logical Logical Service
Domain 1 Domain 2 Domain 3 Domain
App App App V-Ether
App App App App App App
Switch
App App App V-Ether
Switch

V-Ether V-Ether V-Ether

Driver Driver Driver Device-
Driver

Hyper- Virtual LAN 2: 63.24/16

visor Virtual LAN 1: 192.168.0/24

I/O Bridge

Gb
Ether I/F

Page: 25
 Dynamic Reconfiguration (DR)
• Ability to dynamically grow or shrink compute
capacity of an LDom on demand
• No need to re-boot Solaris
• Simply add / remove:
> CPUs
> Memory (future)
> I/O (future)
• Improve utilisation by balancing resources between
LDoms

Page: 26
 LDoms Manager

Logical Logical Logical Service

Domain 1 Domain 2 Domain 3 Domain
App App App LDoms
App App App App App Manager
App App

OS DR OS DR OS DR
Stack Stack Stack

Hyper- HV DR
Stack
visor

Page: 27
 Example: vCPU reconfiguration

Logical Logical Logical

Domain 1 Domain 1 Domain 2
App App App
App App App

V V V V V

Hyper- Hyper-
visor P P visor P P P

Example command line operations:

% ldm remove-vcpu 1 Logical_Domain_1
% ldm add-vcpu 1 Logical_Domain_2
Page: 28
 Configuration
Examples

Page: 29
 Configuring a machine to run LDoms
• Make resources available (remove from control domain)
primary#> ldm set-mau 2 primary
primary#> ldm set-vcpu 8 primary
primary#> ldm set-mem 8g primary

• Configure services
primary#> ldm add-vcc port-range=5000-5100 primary-vcc0 primary
primary#> ldm add-vds primary-vds0 primary
primary#> ldm add-vsw net-dev=e1000g0 primary-vsw0 primary

Page: 30
 Adding another domain (I)
• Add disk device
primary#> ldm add-vdsdev /dev/dsk/c1t0d0s2 vol1@primary-vds0
• Adding resources
primary#> ldm create domain1
primary#> ldm add-mem 2g domain1
primary#> ldm add-vcpu 4 domain1
primary#> ldm add-vdisk vdisk1 vol1@primary-vds0 domain1
primary#> ldm add-vnet vnet1 primary-vsw0 domain1
• Show domain constraints
primary#> ldm list -l domain1

Page: 31
 Adding another domain (II)
• Booting the newly created domain
primary#> ldm bind domain1
primary#> ldm start domain1

• Connect to the console

primary#> telnet localhost 5000

Page: 32
 More information
• BigAdmin Portal
> http://www.sun.com/bigadmin/hubs/ldoms/

• Sun Blueprint
> Beginners Guide to LDoms: Understanding and Deploying
Logical Domains
> http://www.sun.com/blueprints/0207/820-0832.pdf

Page: 33
Logical Domains
Liam Merwick
[email protected]
LDoms Development
SPARC Platform Software

34