LDoms Configuration

Logical Domains
Liam Merwick Sun Microsystems, Inc.
Contents
Background Components Features Architecture Configuration Examples Q&A
Page: 2
Background
Page: 3
Consolidation
Conventional Unix server utilisation is 7 to 15% Data center compression
> Goal: Increase utilisation of equipment
10%
Utilisation
20%
Time
10%
Page: 4
Solutions from Sun

Hard Partitions
App Server Database Identity Server
Virtual Machines OS Virtualisation Resource Mgmt.

File Server Web Server Mail Server Calendar Database Web Server Server SunRay Database Server App Server
App OS Server
Multiple OSes Single OS Trend to flexibility Trend to isolation

Dynamic System Domains SPARC Logical Domains Solaris Xen VMware ESX Microsoft Virtual Server
Page: 5
Solaris Containers (Zones + SRM) Solaris Containers for Linux Applications Solaris Trusted Extensions
Solaris Resource Manager (SRM)
Virtualisation for SPARC platforms

New sun4v architecture
Solaris X update (genunix) sun4u code
US-Z CPU code SPARC hypervisor
Operating System
Solaris X (genunix)
Solaris X (sun4v)
sun4v interface
CPU Z
SPARC CPU
Platform
Page: 6
Virtual Machine for SPARC

Thin software layer between OS and platform hardware Hypervisor + sun4v interface
Virtualises machine HW and isolates OS from register-level Delivered with platform not OS Not itself an OS
sun4v virtual machine User App User App Solaris OpenBoot Hypervisor SPARC hardware User App
stable interface sun4v

Page: 7
Logical Domains
Partitioning capability
> Create virtual
LDom 1 Solaris 10
App
machines each with sub-set of resources > Protection & Isolation using HW+firmware combination
Hypervisor Hardware
Shared CPU, Memory, IO
LDom 2 Solaris 10
App App Zone
LDom 3 Solaris Express

App App Zone 1 App Zone 2
App
App
CPU Mem
CPU Mem
CPU Mem
CPU I/O
Page: 8
Logical Domains Technology

Virtualisation and partitioning of machine resources
> Each domain is a full virtual machine, with a dynamically reconfigurable sub-
set of machine resources, and its own independent OS > Protection & isolation via SPARC hardware and LDoms Hypervisor firmware
LDom A
LDom B
LDom C
LDom D
OS Environment of choice LDoms Hypervisor

CPU CPU CPU CPU CPU Memory I/O
Linux
FreeBSD
CPU CPU Memory
CPU Memory
Platform Hardware
Page: 9
Memory I/O
Coupled with Throughput Computing

Industry trend to many cores & strands per chip
> Most applications don't scale well > Even Solaris doesn't scale perfectly
Ability to dynamically partition system matches throughput computing model

> Can run multiple services and OS instances on same
box / chip > More flexibly be able to utilise available compute power
Page: 10
Components
Page: 11
Components Hardware & Software

Hardware
> UltraSPARC-T1 (Niagara) family of CMT processors > T1000 Netra CP3060 > T2000 Netra T2000
Operating System
> Solaris 10 11/06 (U3) + patches > OpenSolaris (build 57+ recommended)
Firmware
> 6.4.0
LDoms Manager Software

> 1.0 (Solaris package - available as Free download)
Page: 12
Features
Page: 13
Fundamentals
Each virtual machine should appear as an entirely independent machine
> > > > >
own kernel, patches, tuning parameters own user accounts, administrators own disks own network interfaces, MAC & IP addresses Start, stop and reboot independently of each other
Paravirtualised OS
Page: 14
Features
Add and remove cpus while OS is running Protection and isolation via SPARC hardware and LDoms Hypervisor firmware Service domains for I/O
> direct I/O access
Page: 15
Features UltraSPARC-T1 specific

Up to 32 LDoms per box
> CPU threads individually assignable to different LDoms
Memory assignable to with an 8KB granularity Two PCI busses independently assignable to the same or different LDoms. Guest Domains can be configured, started and stopped independently
> Without requiring a power-cycle of machine
Page: 16
Architecture
Page: 17
Architecture
Logical Domains overview Hypervisor LDoms Manager Virtualised I/O Dynamic Reconfiguration (DR)
Page: 18
Hypervisor Support
Hypervisor software is responsible for maintaining separation between domains
> Using extensions built into a sun4v CPU
Also provides Logical Domain Channels (LDCs) so that domains can communicate with each other
> Mechanism by which domains can be virtually networked
with each other, or provide services to each other
Page: 19
LDoms Manager
One Manager per host HV
> Controls Hypervisor and all its LDoms
Exposes control interface via CLI Maps Logical Domains to physical resources
> Constraint engine > Heuristic binding of LDoms to resources > Assists with performance optimisation > Assists in event of failures / blacklisting
Page: 20
Logical Domains
File Server Web Server Mail Server
Application
OS
Server
Page: 21
Direct I/O
Traditional model
> Existing drivers and devices
Logical Domain
App App App
continue to work
Logical Domain owns PCI root and tree
Device Driver /pci@B/qlc@6 Nexus Driver /pci@B
Privileged
Hyper Privileged
Hypervisor I/O Bridge
Virtual Nexus I/F
I/O MMU PCI Root
Hardware
PCI-Express
Page: 22
Virtualised I/O
Logical Domain A
App App App Virtual Device Service Nexus Driver /pci@B App
Service Domain
Device Driver /pci@B/qlc@6
Privileged
Virtual Device Driver
Hyper Privileged
Hypervisor
Virtual Nexus I/F
Domain Channel
I/O MMU
Hardware
PCI Roo t
I/O Bridge
PCI B
Page: 23
Virtual (Block) Disk device

Logical Domain 1
App App App App
Logical Domain 2
App App App App
Logical Domain 3
App App App App
Service Domain
vDisk Server vDisk Server
vDisk client Driver
vDisk client driver
vDisk client driver
DeviceDriver
Hypervisor
Virtual SAN 2 Virtual SAN 1
I/O Bridge
FC-AL I/F
Page: 24
Virtual Ethernet device

Logical Domain 1
App App App App
Logical Domain 2
App App App App
Logical Domain 3
App App App App
Service Domain
V-Ether Switch V-Ether Switch
V-Ether Driver
V-Ether Driver
V-Ether Driver
DeviceDriver
Hypervisor
Virtual LAN 2: 63.24/16 Virtual LAN 1: 192.168.0/24
I/O Bridge
Gb Ether I/F
Page: 25
Dynamic Reconfiguration (DR)

Ability to dynamically grow or shrink compute capacity of an LDom on demand No need to re-boot Solaris Simply add / remove:
> CPUs > Memory (future) > I/O (future)
Improve utilisation by balancing resources between LDoms

Page: 26
LDoms Manager
Logical Domain 1
App App App App
Logical Domain 2
App App App
Logical Domain 3
App App App
Service Domain
LDoms Manager
OS DR Stack
OS DR Stack
OS DR Stack
Hypervisor
HV DR Stack
Page: 27
Example: vCPU reconfiguration

Logical Domain 1
App App
Logical Domain 1
App App
Logical Domain 2
App App
V
Hypervisor
V P
Hypervisor
V P P
V P
Example command line operations: % ldm remove-vcpu 1 Logical_Domain_1 % ldm add-vcpu 1 Logical_Domain_2
Page: 28
Configuration Examples
Page: 29
Configuring a machine to run LDoms

Make resources available (remove from control domain) primary#> ldm set-mau 2 primary primary#> ldm set-vcpu 8 primary primary#> ldm set-mem 8g primary Configure services primary#> ldm add-vcc port-range=5000-5100 primary-vcc0 primary primary#> ldm add-vds primary-vds0 primary primary#> ldm add-vsw net-dev=e1000g0 primary-vsw0 primary
Page: 30
Adding another domain (I)

Add disk device
primary#> ldm add-vdsdev /dev/dsk/c1t0d0s2 vol1@primary-vds0
Adding resources
primary#> ldm create domain1 primary#> ldm add-mem 2g domain1 primary#> ldm add-vcpu 4 domain1 primary#> ldm add-vdisk vdisk1 vol1@primary-vds0 domain1 primary#> ldm add-vnet vnet1 primary-vsw0 domain1
Show domain constraints

primary#> ldm list -l domain1
Page: 31
Adding another domain (II)

Booting the newly created domain primary#> ldm bind domain1 primary#> ldm start domain1 Connect to the console primary#> telnet localhost 5000
Page: 32
More information
BigAdmin Portal > http://www.sun.com/bigadmin/hubs/ldoms/ Sun Blueprint > Beginners Guide to LDoms: Understanding and Deploying Logical Domains > http://www.sun.com/blueprints/0207/820-0832.pdf
Page: 33
Logical Domains
Liam Merwick
[email protected] LDoms Development SPARC Platform Software
34

LDoms Configuration

Uploaded by

Copyright:

Available Formats

LDoms Configuration

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

LDoms Configuration

Uploaded by

Copyright:

Available Formats

Logical Domains

Liam Merwick Sun Microsystems, Inc.

Solutions from Sun

Virtual Machines OS Virtualisation Resource Mgmt.

Multiple OSes Single OS Trend to flexibility Trend to isolation

Solaris Resource Manager (SRM)

Virtualisation for SPARC platforms

Virtual Machine for SPARC

stable interface sun4v

LDom 3 Solaris Express

Logical Domains Technology

OS Environment of choice LDoms Hypervisor

CPU CPU Memory

Coupled with Throughput Computing

Ability to dynamically partition system matches throughput computing model

Components Hardware & Software

LDoms Manager Software

Features UltraSPARC-T1 specific

with each other, or provide services to each other

Logical Domain owns PCI root and tree

Device Driver /pci@B/qlc@6 Nexus Driver /pci@B

Hypervisor I/O Bridge

Virtual Nexus I/F

I/O MMU PCI Root

Virtual Device Driver

Virtual Nexus I/F

Virtual (Block) Disk device

vDisk client Driver

vDisk client driver

vDisk client driver

Virtual SAN 2 Virtual SAN 1

Virtual Ethernet device

Virtual LAN 2: 63.24/16 Virtual LAN 1: 192.168.0/24

Dynamic Reconfiguration (DR)

Improve utilisation by balancing resources between LDoms

Example: vCPU reconfiguration

Configuring a machine to run LDoms

Adding another domain (I)

Show domain constraints

Adding another domain (II)

You might also like