1 | © 2012 Oracle Corporation – Proprietary and Confidential

Safe Harbor Statement

The following is intended to outline our general product
direction. It is intended for information purposes only, and
may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing
decisions.
The development, release, and timing of any features or
functionality described for Oracle’s products remains at the
sole discretion of Oracle.

2 | © 2012 Oracle Corporation – Proprietary and Confidential

 Oracle Training Materials – Usage Agreement
Use of this Site (“Site”) or Materials constitutes agreement with the following terms and conditions:

1. Oracle Corporation (“Oracle”) is pleased to allow its business partner (“Partner”) to download and copy the information,
documents, and the online training courses (collectively, “Materials") found on this Site. The use of the Materials is restricted to
the non-commercial, internal training of the Partner’s employees only. The Materials may not be used for training, promotion, or
sales to customers or other partners or third parties.

2. All the Materials are trademarks of Oracle and are proprietary information of Oracle. Partner or other third party at no time has
any right to resell, redistribute or create derivative works from the Materials.

3. Oracle disclaims any warranties or representations as to the accuracy or completeness of any Materials. Materials are provided
"as is" without warranty of any kind, either express or implied, including without limitation warranties of merchantability, fitness
for a particular purpose, and non-infringement.

4. Under no circumstances shall Oracle or the Oracle Authorized Boot Camp Training Partner be liable for any loss, damage,
liability or expense incurred or suffered which is claimed to have resulted from use of this Site of Materials. As a condition of use
of the Materials, Partner agrees to indemnify Oracle from and against any and all actions, claims, losses, damages, liabilities
and expenses (including reasonable attorneys' fees) arising out of Partner’s use of the Materials.

5. Reference materials including but not limited to those identified in the Boot Camp manifest can not be redistributed in any format
without Oracle written consent.

3 | © 2012 Oracle Corporation – Proprietary and Confidential

 Oracle VM for SPARC I/O Virtualization
Instructors Name
4 | © 2012 Oracle Corporation – Proprietary and Confidential
 Module Objectives
• Introduction of I/O Virtualization Concept
– Goal of I/O Virtualization
– I/O Virtualization Solution
• Platform-specific I/O Virtualization
• PCI SIG I/O Virtualization

• I/O Virtualization in OVM for SPARC

– Root Domain
• Roles of root domain in I/O Virtualization in OVM SPARC
– Direct I/O
– Dynamic SR-IOV

5 | © 2012 Oracle Corporation – Proprietary and Confidential

 I/O Virtualization Concept
• Goal of I/O Virtualization
– Implement virtualization to provision physical I/O devices directly into a
domain
• Importance of I/O Virtualization
– Trends to consolidate more workloads in a single server
– Demand for more I/O bandwidth, ideally near bare-metal performance
– Software-based virtual I/O is complex
• High CPU consumption
• Low performance
– IOV is next wave in hardware-assisted virtualization

6 | © 2012 Oracle Corporation – Proprietary and Confidential

 I/O Virtualization Solution
Platform-specific IOV
• Direct I/O
– Assign PCI devices directly to a guest domain, aka PCI-Passthrough
– Guest Domain can use all resources and capabilities in the device
– Will leverage chipset features
• DMA & Interrupt Remp

• Device-specific IOV
– Some PCI device has special support in virtualization
• Hybrid NIU in T2, T3 platform

7 | © 2012 Oracle Corporation – Proprietary and Confidential

 I/O Virtualization Solution
PCI-SIG IOV
• An IO Virtualization effort driven by PCI-SIG organization
– Published as part of new PCI standard
• Already available in all kinds of PCI-E devices
– Ethernet Card
– InfiniBand
– HBA
– …
– Share resources in PCI device, make one as many
– Benefits
• Across platform
• Across different device types
• Nearly bare-metal performance

8 | © 2012 Oracle Corporation – Proprietary and Confidential

 PCI-SIG IOV standards
• SR-IOV
– A PCI-E device can expose as two sets of functions
• Physical Function (PF)
– Full-featured PCI function that has full control of the device
• Virtual Functions (VF)
– Lightweight PCI function that can only use specific resource in the device
– Can have much more VFs than PF in a PCI device

• MR-IOV
– Based on SR-IOV sharing PCI devices among multiple servers
• Get same benefits common virtualization idea
– Increase device utilization
– Support more I/O devices with limited PCI Slots
– Simplify fabrics cabling

9 | © 2012 Oracle Corporation – Proprietary and Confidential

 I/O Virtualization in OVM for SPARC
Root Domain
• Root Domain is a special I/O Domain
– Owns whole PCI Root Complex
– PCI device attached to the RC will also assign to the Root Domain
– Root Domain has the responsibility to initialize the RC and devices
• Role of Root Domain in OVM for SPARC IOV
– Root Domain can assign PCI devices or functions to other domains
– Support delayed reconfiguration to handle IOV requests
• Only I/O related operations are support in delayed reconfiguration for a Root
Domain
– add-io, set-io, remove-io, create-vf, destroy-vf

10 | © 2012 Oracle Corporation – Proprietary and Confidential

 I/O Virtualization in OVM for SPARC
Prepare Root Domain for IO Virtualization
• Must enable iov for RC that owns the PCI devices
– Enable iov will remove some resources from RC
– IO Performance will be impact a little bit
• How to enable iov of the RC
– # ldm set-io iov=on <pseudo name of RC>
– You can also set “rc_add_policy” property to “iov” before adding RC to
the Root Domain
• # ldm set-domain rc-add-policy=iov <Root Domain>

11 | © 2012 Oracle Corporation – Proprietary and Confidential

 I/O Virtualization in OVM for SPARC
Direct I/O
• Direct I/O will assign a PCI device to other domain
– Can be individual PCI device or whole PCI RC
• A PCI device equals to a PCI slot

• Has to check before PCI device is to be removed from the

Root Domain
– Ensure the device is not used for critical service
• Connecting to boot disk
• As the only network connecting with external world
– Use either following ways to check the device
• In control domain, run # ldm list-io -l
• In root domain, run # prtdiag -v

12 | © 2012 Oracle Corporation – Proprietary and Confidential

 Steps to add a physical PCI device to domain
• Start delayed reconfiguration for Root Domain
– # ldm start-reconf <Root Domain>
• Remove PCI device from Root Domain
– # ldm remove-io <bus|device> <Root Domain>
• Save Configuration to SP
• Reboot Root Domain to release PCI device
• Shutdown the guest domain
– # ldm stop-domain <guest domain>
• Add PCI device to guest domain
– # ldm add-io <bus|device> <guest domain>

13 | © 2012 Oracle Corporation – Proprietary and Confidential

 I/O Virtualization in OVM for SPARC
Dynamic SR-IOV
• OVM for SPARC supports SR-IOV in different PCI devices
– Ethernet Card
– InfiniBand
• How to know which devices support SR-IOV
– # ldm list-io
– In latest OVM SPARC version, supported SR-IOV devices
• Network: IOVNET.PF<n>
• InfiniBand: IOVIB.PF<n>

14 | © 2012 Oracle Corporation – Proprietary and Confidential

 Steps to configure SR-IOV
• Create/Remove VF of PCI device
– # ldm create-vf <pf name>
– # ldm destroy-vf <vf name>
– Can create/remove VF dynamically if following requirements meet
otherwise start delayed reconfiguration
• Root Domain is not using the PCI device
• No VF is assigned to any domain

15 | © 2012 Oracle Corporation – Proprietary and Confidential

 Steps to configure SR-IOV – (Continued)
• Add/Remove VF for I/O domain
– Stop the I/O domain first
• # ldm stop-domain <io domain>
– Add/Remove VF
• # ldm add-io <vf> <domain>
• # ldm remove-io <vf> <domain>

16 | © 2012 Oracle Corporation – Proprietary and Confidential

17 | © 2012 Oracle Corporation – Proprietary and Confidential
18 | © 2012 Oracle Corporation – Proprietary and Confidential
19 | © 2012 Oracle Corporation – Proprietary and Confidential