0% found this document useful (0 votes)

105 views

Oracle Databases On VMware - Best Practices Guide

Oracle_Databases_on_VMware_-_Best_Practices_Guide.pdf

Uploaded by

vbpanchal

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

105 views

Oracle Databases On VMware - Best Practices Guide

Oracle_Databases_on_VMware_-_Best_Practices_Guide.pdf

Uploaded by

vbpanchal

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 26

Oracle Databases on VMware Best Practices Guide

2011 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. This product is covered by one or more patents listed at http://www.vmware.com/download/patents.html. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

Oracle Databases on VMware Best Practices Guide

Contents
1. 2. 3. 4. Introduction ...................................................................................... 5 vSphere ........................................................................................... 5 Production Support for Oracle Databases on vSphere ..................... 6 ESX Host Guidelines........................................................................ 7
4.1 General .......................................................................................................................... 7 4.2 Memory ........................................................................................................................ 10 4.3 Virtual CPU .................................................................................................................. 12

Storage Guidelines ........................................................................ 14

5.1 Storage Virtualization Concepts................................................................................... 15 5.2 Storage Protocol Capabilities....................................................................................... 15 5.3 Database Layout Considerations ................................................................................. 16 5.4 VMFS versus RDM ...................................................................................................... 19 5.5 General Guidelines ...................................................................................................... 20

6. 7. 8. 9. 10.

Networking Guidelines ................................................................... 21 Performance Monitoring on vSphere .............................................. 22 Timekeeping in Virtual Machines ................................................... 23 Summary ....................................................................................... 24 References ................................................................................ 25

Appendix A: Virtual Machine Memory Settings ..................................... 26

2011 VMware, Inc. All rights reserved. Page 3 of 26

Oracle Databases on VMware Best Practices Guide

2011 VMware, Inc. All rights reserved. Page 4 of 26

Oracle Databases on VMware Best Practices Guide

Introduction
This Oracle Databases on VMware Best Practices Guide provides best practice guidelines for deploying Oracle databases on VMware vSphere . The recommendations in this guide are not specific to any particular set of hardware, or size and scope of any particular Oracle database implementation. The examples and considerations provide guidance, but do not represent strict design requirements. The successful deployment of Oracle on VMware vSphere 5 is not significantly different from deploying Oracle on physical servers. DBAs can fully leverage their current skill set while also delivering the benefits associated with virtualization. In addition to this guide, VMware has created separate best practice documents for storage, networking, and performance. This document also includes information from a white paper, Oracle Databases on VMware vSphere 4: Essential Deployment Tips (http://www.vmware.com/files/pdf/Oracle_Databases_on_vSphere_Deployment_Tips.pdf). See Section 10, References, for a list of other documents that can help you successfully deploy Oracle on vSphere.

2. vSphere
VMware virtualization solutions provide numerous benefits to DBA administrators. VMware virtualization creates a layer of abstraction between the resources required by an application and operating system, and the underlying hardware that provides those resources. This abstraction layer provides value for the following: Consolidation VMware technology allows multiple application servers to be consolidated onto one physical server, with little or no decrease in overall performance. Ease of provisioning VMware virtualization encapsulates an application into an image that can be duplicated or moved, greatly reducing the cost of application provisioning and deployment. Manageability Virtual machines can be moved from server to server with no downtime using VMware vSphere vMotion , which simplifies common operations such as hardware maintenance, and reduces planned downtime. Availability If an unplanned hardware failure occurs, VMware vSphere High Availability (HA) restarts affected virtual machines on another host in a VMware cluster. With VMware HA you can reduce unplanned downtime and provide higher service levels to an application. VMware vSphere Fault Tolerance (FT) features zero downtime and zero data loss, providing continuous availability in the face of server hardware failures for any application running in a virtual machine.

2011 VMware, Inc. All rights reserved. Page 5 of 26

Oracle Databases on VMware Best Practices Guide

3. Production Support for Oracle Databases on vSphere

Oracle has a support statement for VMware products that is honored around the world. While there has been much public discussion about Oracles perceived position on support for VM ware virtualization, our experience is that Oracle Support upholds its commitment to customers, including those using VMware virtualization in conjunction with Oracle products. VMware is also an Oracle customer; our E-Business Suite and Siebel instances are virtualized; and VMware routinely submits and receives assistance with issues for Oracle running on VMware virtual infrastructure. The specifics of Oracles support commitment to VMware are provided by the MyOracleSupport MetaLink Document ID 249212.1. Gartner, IDC, and others also have documents for their subscribers that specifically address this policy. The following are some of the key facts about Oracle Support: Oracle RAC support is now included for Database 11.2.0.2 and later. Known issues Oracle Support will accept customer support requests for Oracle products running on VMware virtual infrastructure if the reported problem is already known to Oracle. This is crucialif you are running Database 9i, 10g, or another product with a long history, the odds are in your favor that Oracle has seen your problem before. If they have already seen it, they will accept it. New issues Oracle Support reserves the right to ask customers to prove that new issues attributed to Oracle are not a result of an application being virtualized. This is reasonable, as this is essentially the same policy that other ISVs use to some degree. It is key to look at the history of Oracle Support with regard to new issues. Certification VMware vSphere is a technology that resides under the certified Oracle stack (unlike other virtualization technologies that alter the OS and other elements of the stack). As a result, Oracle cannot certify VMware virtual infrastructure. However, VMware is no different in this regard from an x86 serverOracle doesnt certify Dell, HP, IBM, or Sun x86 servers.

VMware recommends that customers take a logical approach and test Oracles support statement. Begin with pre-production systems, and as issues are encountered and SRs are filed, track Oracles response. Our experience is that customers see no difference in the quality and timeliness of Oracle Supports response.

2011 VMware, Inc. All rights reserved. Page 6 of 26

Oracle Databases on VMware Best Practices Guide

4. ESX Host Guidelines

4.1 General
Comments Create a computing environment optimized for vSphere. The VMware ESX or VMware ESXi host BIOS settings can be specifically adjusted to maximize compute resources (such as disabling unnecessary processes and peripherals) to Oracle databases. Comments Create golden images of optimized operating systems using vSphere cloning technologies. After the operating system has been prepared with the appropriate patches and kernel settings, Oracle can be installed in a virtual machine the same way it is installed on a physical system. This speeds up the installation of a new database. Comments Upgrade to ESXi 5. VMware and database administrators can realize a 1020 percent performance boost after upgrading to the latest vSphere release from prior 3.x versions. Comments Allow vSphere to choose the best virtual machine monitor based on the CPU and guest operating system combination. Confirm that the virtual machine setting has Automatic selected for the CPU/MMU Virtualization option. The following are general best practices for host systems. Item Recommendation Justification

Item Recommendation

Justification

Item Recommendation Justification

2011 VMware, Inc. All rights reserved. Page 7 of 26

Oracle Databases on VMware Best Practices Guide

4.1.1 BIOS Settings

BIOS settings for an x86 server can be set to disable unnecessary processes and peripherals to maximize performance. Table 1 describes the optimized settings. Table 1. BIOS Settings Maximized for Performance
BIOS Setting Virtualization Technology Turbo Mode Node Interleaving VT-x, AMD-V, EPT, RVI C1E Halt State Recommendations Yes Yes No Yes No Description Necessary to run 64-bit guest operating systems. Balanced workload over unused cores. Disables NUMA benefits if disabled. Hardware-based virtualization support. Disable if performance is more important than saving power. Disable if performance is more important than saving power. Disables warning messages when writing to the master boot record. For use with some Intel processors. Hyperthreading is always recommended with Intels newer Core i7 processors such as the Xeon 5500 series. Not necessary for database virtual machine. Required for VMware vSphere Distributed Power Management feature. Required for vMotion and VMware vSphere Distributed Resource Scheduler (DRS) features. Not necessary for database virtual machine. Not necessary for database virtual machine. Not necessary for database virtual machine. Not necessary for database virtual machine. Not necessary for database virtual machine. Not necessary for database virtual machine. Not necessary for database virtual machine. Not necessary for database virtual machine.

Power-Saving

Virus Warning

Hyperthreading

Yes

Video BIOS Cacheable Wake On LAN

No Yes

Execute Disable

Yes

Video BIOS Shadowable Video RAM Cacheable On-Board Audio On-Board Modem On-Board Firewire On-Board Serial Ports On-Board Parallel Ports On-Board Game Port

No No No No No No No No

2011 VMware, Inc. All rights reserved. Page 8 of 26

Oracle Databases on VMware Best Practices Guide

4.1.2 Operating System Host Processes

VMware recommends disabling unnecessary foreground and background processes within the guest operating system. Examples of unnecessary Linux processes are: anacron, apmd, atd, autofs, cups, cupsconfig, gpm, isdn, iptables, kudzu, netfs, and portmap. Examples of unnecessary Windows processes are: alerter, automatic updates, clip book, error reporting, help and support, indexing, messenger, netmeeting, remote desktop, and system restore services. For Linux installs, the database administrator (DBA) should request that the system administrator compile a monolithic kernel to load only the necessary features. Whether you intend to run Windows or Linux as the final optimized operating system, these host installs should be cloned by the VMware administrator for reuse. After the operating system has been prepared, install Oracle the same way as for a physical environment. Use the recommended kernel parameters listed in the Oracle Installation guide. Also, it is a good practice to check with Oracle Support for the latest settings to use prior to beginning the installation process.

4.1.3 Upgrade vSphere

vSphere includes numerous performance and scalability enhancements that provide a 1020 percent performance boost compared to previous 3.x versions. The following table summarizes the improvements to the hypervisor including current metrics for vSphere. Table 2. Performance and Scalability Improvements by ESX/ESXi Version
ESX 2.0 CPU Memory Network IOPS 1 vCPU < 4GB 380Mb/Sec < 10,000 ESX 3.0 2 vCPU 16GB 800Mb/Sec 20,000 ESX 3.5 4 vCPU 64GB 9Gb/Sec 100,000 vSphere ESX 4 8 vCPU 255GB 30Gb/Sec > 350,000 vSphere ESX 5 32 vCPU 1TB > 36Gb/Sec 1,000,000

VMware vSphere supports large capacity virtual machines, so it can support larger sized Oracle databases and SGA footprints. vSphere host and virtual machine specifications are as follows: Each ESXi host supports up to 2TB RAM, 512 virtual machines and 2048 virtual CPUs. Each virtual machine can support up to 32 virtual CPUs and 1TB RAM.

2011 VMware, Inc. All rights reserved. Page 9 of 26

Oracle Databases on VMware Best Practices Guide

4.1.4 Hardware-Assisted Memory Management Unit

For best performance, VMware recommends using servers with the latest chip generations that support a hardware-assisted Memory Management Unit (MMU). Hardware-assisted MMU refers to hardware support for memory management unit virtualization. Features that provide the support are available from Intel and AMD and are called EPT and RVI, respectively. Support consists of an additional level of page tables implemented in hardware. These page tables contain guest physical to machine physical memory address translations. On processors that support it, vSphere by default uses hardware-assisted MMU virtualization for virtual machines. This default behavior is configured in the virtual machine settings using VMware vSphere Client, by setting the CPU/MMU Virtualization parameter to Automatic (which is the default).

4.2

Memory
Comments Set memory reservations equal to the size of the Oracle SGA. The memory reservation should be large enough to avoid kernel swapping between ESX and the guest OS because Oracle databases can be memoryintensive. Comments Use large memory pages. Large page support is enabled by default in ESX 3.5 and later, and is supported from Oracle 9i R2 for Linux operating systems and 10g R2 for Windows. Enable large pages in the guest OS to improve the performance of Oracle databases on vSphere.

The following are memory-related best practices. Item Recommendation Justification

Item Recommendation Justification

Appendix A provides a description of virtual machine memory settings that are discussed in this section. For further background on VMware memory management concepts, refer to vSphere Resource Management (http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenterserver-50-resource-management-guide.pdf). When consolidating Oracle database instances, vSphere presents the opportunity to share memory across virtual machines that are running the same operating systems, applications, or components. In this case, vSphere uses a proprietary transparent page sharing technique to reclaim memory, which allows databases to run with less memory than on a physical machine. Transparent page sharing also allows DBAs to overcommit memory without any performance degradation. In production environments, carefully consider the impact of overcommitting memory and overcommit only after collecting data to determine the amount of overcommitment possible. To determine the effectiveness of memory sharing and the degree of acceptable overcommitment for a given database, run the workload and use resxtop or esxtop to observe the actual savings.

2011 VMware, Inc. All rights reserved. Page 10 of 26

Oracle Databases on VMware Best Practices Guide Because Oracle databases can be memory-intensive, and to account for situations where performance is a key factor (and to avoid kernel swapping between ESX/ESXi and the guest OS in mission critical production environments), VMware recommends the following: Set the memory reservation equal to the size of the Oracle SGA. Where the Oracle database is part of a third-party commercial enterprise application (ERP), follow virtualization guidelines from the ERP vendor. Note that setting reservations can limit vMotion operations. A virtual machine can be live migrated only if the target ESX/ESXi host has free physical memory equal to or greater than the size of the reservation. Do not disable the balloon driver. The guest operating system within the virtual machine still needs its own separate swap/page file. Follow the same swap space guidelines given for physical environments.

Though VMware recommends setting memory reservations equal to the size of the Oracle SGA in production environments, it is acceptable to overcommit more aggressively in non-production environments such as development, test, or QA. In these environments, a DBA can introduce memory overcommitment to take advantage of VMware memory reclamation features and techniques. Even in these environments, the type and number of databases that can be deployed using overcommitment largely depend on their actual workload.

4.2.1 Large Pages

vSphere supports large pages in the guest operating system. The use of large pages results in reduced memory management overhead and can increase hypervisor performance. Oracle supports the use of large memory pages in Oracle 9i R2 for Linux operating systems and in 10g R2 for Windows. The following MetaLink Notes are relevant when setting large pages: Note 361323.1 Huge Pages on Linux: What It Is... and What It Is Not... Note 361468.1 Huge Pages on 64-bit Linux Note 401749.1 Shell Script to Calculate Values Recommended Huge Pages/Huge TLB Configuration Note 46001.1 Oracle Database and the Windows NT memory architecture, Technical Bulletin Note 46053.1 Windows NT Memory Architecture Overview

2011 VMware, Inc. All rights reserved. Page 11 of 26

Oracle Databases on VMware Best Practices Guide

4.3

Virtual CPU
Comments Use as few virtual CPUs (vCPUs) as possible. If monitoring of the actual workload shows that the Oracle database is not benefitting from the increased virtual CPUs, the excess vCPUs impose scheduling constraints and can degrade overall performance of the virtual machine. Comments Enable hyperthreading for Intel Core i7 processors. With the release of Intel Xeon 5500 series processors, enabling hyperthreading is recommended.

The following are virtual CPU-related best practices. Item Recommendation Justification

Item Recommendation Justification

VMware uses the terms virtual CPU (vCPU) and physical CPU to distinguish between the processors within the virtual machine and the underlying physical x86-based processors. Virtual machines with more than one virtual CPU are also called SMP (symmetric multiprocessing) virtual machines. VMware vSphere Virtual Symmetric Multiprocessing (Virtual SMP) enhances virtual machine performance by enabling a single virtual machine to use multiple physical processors simultaneously. vSphere supports up to eight virtual CPUs per virtual machine. The biggest advantage of an SMP system is the ability to use multiple processors to execute multiple tasks concurrently, thereby increasing throughput (for example, the number of transactions per second). Only workloads that support parallelization (including multiple processes or multiple threads that can run in parallel) can really benefit from SMP. The Oracle architecture is multithreaded and includes multiple processes which make it a good candidate to take advantage of Virtual SMP. Though larger virtual machines are possible in vSphere, VMware recommends reducing the number of virtual CPUs if monitoring of the actual workload shows that the Oracle database does not benefit from the increased number of virtual CPUs. Virtual NUMA (vNUMA), a new feature in ESXi 5.0, exposes NUMA topology to the guest operating system, allowing NUMA-aware guest operating systems and applications to make the most efficient use of the underlying hardwares NUMA arch itecture. For additional details, refer to ESXi CPU Considerations in Performance Best Practices for VMware vSphere 5.0 (http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf). Oracle NUMA support is disabled by default for 11g and above (see Oracle MySupport Doc ID: 864633.1). In some circumstances enabling Oracle NUMA support may improve performance and the Oracle doc suggests that it be tested in a test environment before deciding to use it with production system. VMware recommends keeping NUMA enabled in server hardware BIOS and at the guest operating system level which should also be the default settings for NUMA support with most servers and guest operating systems. Hyperthreading enables a single physical processor core to behave like two logical processors, allowing two independent threads to run simultaneously. Unlike having twice as many processor cores which can roughly double performancehyperthreading can provide anywhere from a slight to a significant increase in system performance by keeping the processor pipeline busier. For example, an ESX/ESXi host system enabled for hyperthreading on an 8-core server sees 16 threads that appear as 16 logical processors. With the release of Intel Xeon 5500 series processors, enabling hyperthreading is recommended. Prior to the 5500 series, VMware had no uniform recommendation with respect to hyperthreading because the measured performance results were not consistent across applications, run environments, or database workloads. VMware recommends the following practices for allocating CPU to Oracle database virtual machines: 2011 VMware, Inc. All rights reserved. Page 12 of 26

Oracle Databases on VMware Best Practices Guide Start with a thorough understanding of your workload. Database server utilization varies widely by application. If the application is commercial, follow published guidelines where appropriate. If the application is custom-written, work with the application developers to determine resource requirements. VMware Capacity Planner can analyze your current environment and provide resource utilization metrics that can aid in the sizing process. If the exact workload is not known, start with fewer virtual CPUs and increase the number later if necessary. Allocate multiple vCPUs to a virtual machine only if the anticipated database workload can take advantage of all the vCPUs. When consolidating multiple virtual machines on single ESX/ESXi host, proper hardware sizing is critical for optimal performance. Confirm that cumulative physical CPU resources on a host are adequate to meet the needs of the virtual machines by testing your workload in the planned virtualized environment. CPU overcommitment should be based upon actual performance data to avoid adversely affecting virtual machine performance.

2011 VMware, Inc. All rights reserved. Page 13 of 26

Oracle Databases on VMware Best Practices Guide

5. Storage Guidelines
The following are storage-related best practices. Item Recommendation Justification Comments Enable jumbo frames for IP-based storage using iSCSI and NFS. Jumbo frames enable Ethernet frames to have a larger payload, allowing for improved performance. Comments Create dedicated datastores to service database workloads. The creation of dedicated datastores for I/O-intensive databases is analogous to provisioning dedicated LUNs in the physical world. This is a typical design for a mission-critical enterprise workload. Comments Use VMware vSphere VMFS for single instance Oracle database deployments. To balance performance and manageability in a virtual environment, deploy Oracle using VMFS. Comments Align VMFS properly. Like other disk-based file systems, VMFS suffers a penalty when the partition is unaligned. Use VMware vCenter to create VMFS partitions because it automatically aligns the partitions. Comments Use Oracle automatic storage management. Oracle ASM provides integrated clustered file system and volume management capabilities for managing Oracle database files. ASM simplifies database file creation while delivering near-raw device file system performance. Comments Use your storage vendors best practices documentation when laying out the Oracle database. Oracle ASM cannot determine the optimal data placement or LUN selection with respect to the underlying storage infrastructure. For that reason, Oracle ASM is not a substitute for close communication between the storage administrator and the database administrator. Comments Avoid silos when designing the storage architecture. At a minimum, designing the optimized architecture should involve the database administrator, storage administrator, network administrator, VMware administrator, and application owner.

Item Recommendation Justification

Item Recommendation

Justification

Item Recommendation Justification

2011 VMware, Inc. All rights reserved. Page 14 of 26

Oracle Databases on VMware Best Practices Guide Item Recommendation Justification Comments Use Paravirtualized SCSI adapters for Oracle data files with demanding workloads. The combination of the new Paravirtualized SCSI driver (PVSCSI) and additional ESX/ESXi kernel-level storage stack optimizations dramatically improves storage I/O performance.

Storage configuration is essential for any successful database deployment, especially in virtual environments where you can consolidate many different Oracle database workloads on a single ESX/ESXi host. Your storage subsystem should provide sufficient I/O throughput as well as storage capacity to accommodate the cumulative needs of all virtual machines running on your ESX/ESXi hosts.

5.1

Storage Virtualization Concepts

VMware storage virtualization can be categorized into three layers of storage technology. The storage array is the bottom layer, consisting of physical disks presented as logical disks (storage array volumes or LUNs) to the layer above. The next layer is the virtual environment occupied by vSphere. Storage array LUNs are presented to ESX/ESXi hosts as datastores and are formatted as VMFS volumes. Virtual machines consist of virtual disks that are created in the datastores and presented to the guest operating system as disks that can be partitioned and used in file systems.

5.1.1 Virtual Machine File System (VMFS)

VMFS is a cluster file system that provides storage virtualization optimized for virtual machines. Each virtual machine is encapsulated in a set of files and VMFS is the default storage system for these files on physical SCSI disks and partitions. VMFS allows multiple ESX/ESXi instances to access shared virtual machine storage concurrently. It also enables virtualization-based distributed infrastructure services such as vMotion, DRS, and VMware HA to operate across a cluster of ESX/ESXi hosts.

5.1.2 Raw Device Mapping

VMware also supports Raw Device Mapping (RDM). RDM allows a virtual machine to directly access a volume on the physical storage subsystem, and can be used only with Fibre Channel or iSCSI. RDM can be thought of as providing a symbolic link from a VMFS volume to a raw volume. The mapping makes volumes appear as files in a VMFS volume. The mapping file, not the raw volume, is referenced in the virtual machine configuration.

5.2

Storage Protocol Capabilities

When deploying vSphere, the choice of a networked storage system has little to do with virtualization. As with any physical Oracle deployment, the main considerations are price, performance, and manageability. In addition, the protocols available with vSphereFibre Channel, hardware iSCSI, software iSCSI, and NFS are capable of achieving throughput levels that are limited only by the capabilities of the storage array and its connection to vSphere. During its testing, VMware has found that wire speed is the limiting factor for I/O throughput when comparing the storage protocols. VMware ESX/ESXi can reach the link speeds in a single virtual machine environment, and also maintain the throughput up to 32 concurrent virtual machines for each storage connection option supported. For details, refer to the Comparison of Storage Protocol Performance in VMware vSphere 4 white paper (http://www.vmware.com/files/pdf/perf_vsphere_storage_protocols.pdf). Fibre Channel might provide maximum I/O throughput, but iSCSI and NFS can offer a better price-performance ratio.

2011 VMware, Inc. All rights reserved. Page 15 of 26

Oracle Databases on VMware Best Practices Guide When selecting networked storage systems and protocols, it is critical to understand which vSphere features are supported. The following table describes the capabilities for each of the protocols available in vSphere. Table 3. Storage Protocol Capabilities
Type Boot VM Boot vSphere Yes Yes No Yes vMotion HA/DRS Yes Yes Yes No VMFS RDM SRM

Fibre Channel iSCSI NAS Local storage

Yes Yes Yes Yes

Yes Yes No Yes

Yes Yes No No

Jumbo frames are recommended for IP-based storage using iSCSI and NFS. Jumbo frames must be enabled for each vSwitch through the vSphere CLI. Also, if you use an ESX/ESXi host, you must create a VMkernel network interface with jumbo frames enabled. It is also necessary to enable jumbo frames on the hardware as well, including the network switches and storage arrays.

5.3

Database Layout Considerations

The Oracle Optimized Flexible Architecture (OFA) is a set of naming standards and best practices to be used when installing and configuring Oracle software. It is a generally accepted best practice to follow the OFA standards for Oracle virtual installations as well. Beginning in 10g, Oracle introduced Automated Storage management, which also conforms to the OFA naming conventions.

5.3.1 Automatic Storage Management

Oracle ASM provides integrated clustered file system and volume management capabilities for managing Oracle database files. In addition, ASM simplifies database file creation while delivering near-raw device file system performance. A vSphere datastore is an abstraction of the storage layer. LUNs can be thought of as abstractions of the disks themselves. For this reason, care must be taken before configuring ASM disk groups. When creating ASM disk groups, observe the following: Create ASM disk groups with equal disk types and geometries. An ASM disk group is essentially a grid of disks and the group performance is limited by its slowest member. Create multiple ASM disk groups based on I/O characteristics. At a minimum, create two ASM disk groupsone for log files, which are sequential in nature, and another for datafiles, which are random in nature.

If using networked storage, configure the ASM disk groups with external redundancy. Do not use Oracle ASM failure groups. Oracle failure groups consume additional CPU cycles and can operate unpredictably after suffering a disk failure. When using external redundancy, disk failures are transparent to the database and consume no additional database CPU cycles, because this is offloaded to the storage processors. ASM is not storage-aware. In other words, whatever disks are provisioned to a DBA can be used to create a disk group. Oracle ASM cannot determine the optimal data placement or LUN selection with respect to the underlying storage infrastructure. For that reason, Oracle ASM is not a substitute for close communication between the storage administrator and the database administrator. Refer to your Oracle installation guide to create ASM disk groups.

2011 VMware, Inc. All rights reserved. Page 16 of 26

Oracle Databases on VMware Best Practices Guide

5.3.2 Oracle Clustered File System (OCFS)

The Oracle Clustered File System is a POSIX-compliant shared disk cluster file system for Linux that can be used with Oracle Real Application Clusters. OCFS was the predecessor to Oracle ASM that was introduced in Oracle 10g. (Discussion of Real Application Clusters is beyond the scope of this guide.) ASM is the recommended clustering technology. Also, because ASM can also be used for single instance deployments, it provides an on-ramp to Real Application Clusters.

5.3.3 Consolidated or Dedicated Datastores

It is a generally accepted best practice to create a dedicated datastore if the application has a demanding I/O profile. Databases fall into this category. The creation of dedicated datastores allows DBAs to define individual service level guarantees for different applications and is analogous to provisioning dedicated LUNs in the physical world. A datastore is an abstraction of the storage tier and, therefore, it is a logical representation of the storage tier, not a physical representation of the storage tier. Creating a dedicated datastore to isolate a particular I/O workload (whether log or database files), without isolating the physical storage layer as well, does not have the desired effect on performance. 5.3.3.1. Example of Oracle Database Storage Layout on vSphere For mission-critical databases it is common practice in physical environments to spread the database over multiple LUNs to maximize I/O performance (for example, placing log and datafiles in separate LUNs). Follow similar guidelines when virtualized. An example layout is shown in the following figure.

Oracle Databases on VMware Best Practices Guide Figure 1. Example Storage Layout of Oracle OLTP Database on VMware

Figure 1 represents an example storage design for a virtualized Oracle OLTP database. The design is based on the following principles: At a minimum, an optimized architecture requires joint collaboration among the database, VMware, and storage administrators. Follow storage vendor best practices for database layout on their arrays (as is done in the physical world).

Note that Figure 1 is only an example and actual configurations for customer deployments can differ.

5.3.4 Virtual SCSI Controllers

VMware highly recommends using multiple virtual SCSI controllers for the database virtual machines or virtual machines with high I/O load. The use of multiple virtual SCSI controllers allows the execution of several parallel I/O operations inside the guest operating system. VMware also highly recommends separating the Redo/Log I/O traffic from the data file I/O traffic through separate virtual SCSI controllers. As a best practice, you can use one controller for the operating system and swap, another controller for DB Log, and one or more additional controllers for database data files (depending on the number and 2011 VMware, Inc. All rights reserved. Page 18 of 26

Oracle Databases on VMware Best Practices Guide size of the database data files). Refer to the VMware Administration Guide for information on how to add additional virtual SCSI controllers

5.4

VMFS versus RDM

5.4.1 Performance
VMware is often asked which offers better performance, VMFS or RDM. Both VMFS and RDM volumes can provide similar transaction throughput. For more details, refer to Performance Characterization of VMFS and RDM Using a SAN (http://www.vmware.com/files/pdf/performance_char_vmfs_rdm.pdf).

5.4.2 Functionality
VMware generally recommends VMFS, but there may be situations where RDMs are required. Table 4 summarizes some of the options and trade-offs between VMFS and RDM. For a more complete discussion, see vSphere Storage (http://pubs.vmware.com/vsphere50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-storage-guide.pdf). Table 4. VMFS and Raw Disk Mapping Trade-Offs VMFS Volume can host many virtual machines (or can be dedicated to one virtual machine). Increases storage utilization, provides better flexibility, easier administration and management. Can potentially support clustering software that does not issue SCSI reservations, such as Oracle Clusterware. To configure, follow the procedures given in Disabling simultaneous write protection provided by VMFS using the multi-writer flag (http://kb.vmware.com/kb/1034165). Oracle RAC node Live Migration. RDM Maps a single LUN to one virtual machine, so only one virtual machine is possible per LUN. More LUNs are required, so it is easier to reach the LUN limit of 256 that can be presented to an ESX/ESXi host. RDM might be required to leverage third party storage array-level backup and replication tools. RDM volumes can help facilitate migrating physical Oracle databases to virtual machines. Alternatively, enables quick migration to physical in rare Oracle support cases. Required for MSCS quorum disks.

Oracle Databases on VMware Best Practices Guide

5.5

General Guidelines

5.5.1 Partition Alignment

Aligning file system partitions is a well-known storage best practice for database workloads. Partition alignment on both physical machines and VMware VMFS partitions prevents performance I/O degradation caused by I/O crossing track boundaries. VMware test results show that aligning VMFS partitions to 64KB track boundaries results in reduced latency and increased throughput. VMFS partitions created using vCenter are aligned on 64KB boundaries as recommended by storage and operating system vendors. It is considered a best practice to observe the following: Create VMFS partitions from within vCenter because they are aligned by default. Align the data disk for heavy I/O workloads using diskpart. Consult with the storage vendor for alignment recommendations on their hardware.

For more information about this topic see the white paper Performance Best Practices for VMware vSphere 5.0 (http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf).

5.5.2 Paravirtualized SCSI Adapters

A variety of architectural improvements have been made to the storage subsystem of VMware vSphere 4. The combination of the new Paravirtualized SCSI driver (PVSCSI), and additional ESX/ESXi kernel-level storage stack optimizations dramatically improves storage I/O performance. VMware recommends that you create a primary adapter for use with a disk that will host the system software (boot disk) and a separate PVSCSI adapter for the disk that will store the Oracle data files. Results of tests conclude that PVSCSI is not recommended for virtual machines performing less than 2,000 IOPS and issuing greater than four outstanding I/Os. This issue is fixed in vSphere 4.1, so that the PVSCSI virtual adapter can be used with good performance, even under this condition. Follow guidelines in the following VMware Knowledge Base articles: Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters http://kb.vmware.com/kb/1010398 Do I choose the PVSCSI or LSI Logic virtual adapter on ESX 4.0 for non-IO intensive workloads? http://kb.vmware.com/kb/1017652

Oracle Databases on VMware Best Practices Guide

6. Networking Guidelines
The following are networking-related best practices. Item Recommendation Justification Comments Use the VMXNET family of Paravirtualized network adapters. The Paravirtualized network adapters in the VMXNET family implement an optimized network interface that passes network traffic between the virtual machine and the physical network interface cards with minimal overhead. Comments Separate infrastructure traffic from virtual machine traffic for security and isolation. Virtual machines should not see infrastructure traffic (security violation) and should not be impacted by infrastructure traffic bursts (for example, vMotion operations). Comments Use NIC teaming for availability and load balancing. NIC teams can share the load of traffic among some or all of its members, or provide passive failover in the event of a hardware failure or a network outage. Comments Take advantage of Network I/O Control to converge network and storage traffic onto 10GbE. This can reduce cabling requirements, simplify management and reduce cost.

Item Recommendation Justification

The standard VMware networking best practices apply to running Oracle databases on vSphere. For further details follow vSphere Networking (http://pubs.vmware.com/vsphere50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-networking-guide.pdf). This includes designs to efficiently manage multiple networks and redundancy of network adaptors on ESX/ESXi hosts. The key best practice guidelines are: Separate infrastructure traffic from VM traffic for security and isolation. Use NIC teaming for availability and load balancing. NIC teaming occurs when multiple uplink adapters are associated with a single vSwitch to form a team. Take advantage of Network I/O Control to converge network and storage traffic onto 10GbE. Network I/O Control was released in vSphere 4.1 and enables you to guarantee service levels (bandwidth) for particular vSphere traffic types: VM traffic, FT logging, iSCSI, NFS, management, and vMotion. In vSphere use the VMXNET3 network adapter. This is a Paravirtualized device that works only if VMware Tools is installed on the guest operating system. The VMXNET3 adapter is optimized for virtual environments and designed to provide high performance. For further background on network adaptors and compatibility with the ESX/ESXi release and supported guest operating system, see Choosing a network adapter for your virtual machine (http://kb.vmware.com/kb/1001805). For RAC interconnect use jumbo frames. To enable jumbo frames follow Enabling IOAT and Jumbo frames (http://kb.vmware.com/kb/1003712). 2011 VMware, Inc. All rights reserved. Page 21 of 26

Oracle Databases on VMware Best Practices Guide

7. Performance Monitoring on vSphere

The following is a performance monitoring best practice. Item Recommendation Justification Comments Use vCenter and/or the esxtop/resxtop utility for performance monitoring in the virtual environment. Guest OS counters can be used to get a rough idea of performance within the virtual machine but, for example, CPU and memory usage reported within the guest OS can be different from what ESX/ESXi reports.

Always use the VI Client or vSphere Client, esxtop, or resxtop to measure resource utilization. CPU and memory usage reported within the guest OS can be different from what ESX/ESXi reports. Oracle DBA administrators should pay close attention to the following counters. Refer to VMware Communities: Interpreting esxtop Statistics (http://communities.vmware.com/docs/DOC-9279) for a full list of counters.

Table 5. ESX/ESXi Performance Counters

Subsystem CPU esxtop Counters %RDY %USED Memory %ACTV SWW/s SWR/s Storage ACTV DAVG/cmd KAVG/cmd Network MbRX/s MbTX/s vCenter Counter Ready (milliseconds in a 20,000ms window) Usage Active Swapin Rate Swapout Rate Commands deviceWriteLatency and deviceReadLatency kernelWriteLatency and kernelReadLatency packetsRx packetsTx

Table 5 lists key counters to add to the list of inspection points for Oracle DBA administrators. Of the CPU counters, the total used time indicates system load, and ready time indicates overloaded CPU resources. A significant swap rate in the memory counters is a clear indication of a shortage of ESX/ESXi memory, and high device latencies in the storage section point to an overloaded or misconfigured array. Network traffic is not frequently the cause of most database performance problems except when large amounts of iSCSI storage traffic are using a single network line. Check total throughput on the NICs to see if the network is saturated.

Oracle Databases on VMware Best Practices Guide To determine whether there is any swapping within the guest operating system, use in the in-guest counters in the same manner as in physical environments.

8. Timekeeping in Virtual Machines

The following is a timekeeping best practice for virtual machines. Item Recommendation Justification Comments To minimize time drift in virtual machines follow guidelines in relevant VMware Knowledge Base articles. The impact of high timer-interrupt rates in some operating systems can lead to time synchronization errors.

Most operating systems track the passage of time by configuring the underlying hardware to provide periodic interrupts. The rate at which those interrupts are configured to arrive varies for different operating systems. High timer-interrupt rates can incur overhead that affects a virtual machine's performance. The amount of overhead increases with the number of vCPUs assigned to a virtual machine. The impact of these high timer-interrupt rates can lead to time synchronization errors. To address timekeeping issues when running Oracle databases, follow the guidelines in the following VMware Knowledge Base articles: Timekeeping best practices for Linux guests http://kb.vmware.com/kb/1006427 Timekeeping best practices for Windows, including NTP http://kb.vmware.com/kb/1318

Oracle Databases on VMware Best Practices Guide

9. Summary
The best practices and guidelines discussed in this document are listed in this section. Recommendations Create a computing environment optimized for vSphere. Create golden images of optimized operating systems using vSphere cloning technologies. Upgrade to ESX 4. Allow vSphere to choose the best virtual machine monitor based on the CPU and guest operating system combination. Set memory reservations equal to the size of the Oracle SGA. Use large memory pages. Use as few virtual CPUs (vCPUs) as possible. Enable hyperthreading for Intel Core i7 processors. Enable jumbo frames for IP-based storage using iSCSI and NFS. Create dedicated datastores to service database workloads. Use VMware vSphere VMFS for single instance Oracle database deployments. Align VMFS properly. Use Oracle automatic storage management. Use your storage vendors best practices documentation when laying out the Oracle database. Avoid silos when designing the storage architecture. Use Paravirtualized SCSI adapters for Oracle data files with demanding workloads. Use the VMXNET family of Paravirtualized network adapters. Separate infrastructure traffic from virtual machine traffic for security and isolation. Use NIC teaming for availability and load balancing. Take advantage of Network I/O Control to converge network and storage traffic onto 10GbE. Use vCenter and/or the esxtop/resxtop utility for performance monitoring in the virtual environment. To minimize time drift in virtual machines follow guidelines in relevant VMware Knowledge Base articles. Success stories are available at http://vmware.com/solutions/partners/alliances/oracle-databasecustomers.html.

Oracle Databases on VMware Best Practices Guide

10. References
You can find more information about using VMware and Oracle using the following links. Oracle Databases on VMware vSphere 4: Essential Deployment Tips http://www.vmware.com/files/pdf/Oracle_Databases_on_vSphere_Deployment_Tips.pdf Virtualizing Performance-Critical Database Applications in VMware vSphere http://www.vmware.com/pdf/Perf_ESX40_Oracle-eval.pdf Performance Best Practices for VMware vSphere 5.0 http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf VMware Compatibility Guide http://www.vmware.com/resources/compatibility/search.php vSphere Resource Management http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50resource-management-guide.pdf Understanding Memory Resource Management in VMware vSphere 5.0 http://www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5.pdf VMware Communities: Interpreting esxtop Statistics http://communities.vmware.com/docs/DOC-9279 vSphere Networking http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50networking-guide.pdf Comparison of Storage Protocol Performance in VMware vSphere 4 http://www.vmware.com/files/pdf/perf_vsphere_storage_protocols.pdf Performance Characterization of VMFS and RDM Using a SAN http://www.vmware.com/files/pdf/performance_char_vmfs_rdm.pdf vSphere Storage http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50storage-guide.pdf VMware vSphere vMotion Architecture, Performance, and Best Practices in VMware vSphere 5 http://www.vmware.com/files/pdf/vmotion-perf-vsphere5.pdf Disabling simultaneous write protection provided by VMFS using the multi-writer flag http://kb.vmware.com/kb/1034165

Oracle Databases on VMware Best Practices Guide

Appendix A: Virtual Machine Memory Settings

The following figure illustrates the memory settings used for a virtual machine. Figure 2. Virtual Machine Memory Settings

Definition of terms: Configured memory Memory size of virtual machine assigned at creation. Active memory Memory recently accessed by applications in the virtual machine. Reservation Guaranteed lower bound on the amount of memory that the host reserves for the virtual machine, which cannot be reclaimed by ESX/ESXi for other virtual machines. Swappable Virtual machine memory that can be reclaimed by the balloon driver or, in the worst case, by ESX/ESXi swapping. This is the automatic size of the swap file that is created for each virtual machine on the VMFS file system (.vswp file).

For more information about VMware ESX/ESXi memory management concepts and the balloon driver, see vSphere Resource Management (http://pubs.vmware.com/vsphere50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-resource-management-guide.pdf).